performance, chrome, mozilla and tracemonkey
by Christopher Blizzard
So, V8. Well-hyped. It’s got a cool logo. And many claims are being made about its performance. But it is not the only kid on the block. As we blogged about a couple of weeks ago, Mozilla has been investing over the last couple of months in a super-fast JS engine as well.
In terms of claims some members of the V8 team have been bragging a little bit about how V8 is “many times faster” than TraceMonkey. In fact, some guarantees may have been made.
Based on the data above, we’re running about 20% faster than V8 on SunSpider. While I’m sure there will be changes to each of the engines in the coming months I think that the claim that “many times faster” is ludicrous on its face and should be tempered by actual data. [ Note that the Google test is recursion heavy, something we're adding to TraceMonkey right now. This explains the gap on that one type of test. See Brendan's post above or John's post below for more details. ]
It’s also important to realize another fact. Google has had a small army of people working on the V8 engine for two whole earth years. We’re about 60 days into TraceMonkey and we’re already starting to match the performance characteristics of V8. As Brendan put it:
What spectators have to realize is that this contest is not a playoff where each contending VM is eliminated at any given hype-event point. We believe that Franz&Gal-style tracing has more “headroom” than less aggressively speculative approaches, due to its ability to specialize code, making variables constant and eliminating dead code and conditions at runtime, based on the latent types inherent in almost all JavaScript programs. If we are right, we’ll find out over the next weeks and months, and so will you all.
If you want data across browsers you should look at this post from John Resig that contains some graphics that give relative performance of various browsers including Safari, Firefox 3.0.1, IE, etc. His overview is great and gives a much wider view of relative browser performance.
Also as a side note because I have your attention. There are some bizarre and incorrect claims made in the comic about garbage collection. Brendan puts things right as a comment in John’s post:
@Ben: Chrome has a nice GC: exact rooting, generational with copying. Single-threaded, too (not an option for SpiderMonkey, which is used in AT&T 1-800-555-1212 and 411 AVR massively multi-threaded services built by tellme.com, now owned by Microsoft!). It definitely helps cut down on pauses and keep memory use flatter.
The Chrome comic book did have one piece of misinformation, though: it said other browsers’ engines use conservative GC, and have false positive problems because they can’t distinguish random integers from pointers into the heap. This is not true of Firefox, IE, or Opera.
We do live in interesting times.
Any comment on the stability problems of FF3 (esp. in regard to plugins)? Looks like Chrome won’t have those problems, and apparently same goes for IE8… I wonder what’s the plan for FF there?
While it’s outstanding to hear that my favorite browser, firefox, is no slouch in the javascript speed department, I think it’s unfair to compare the performance at this time. Google Chrome has so far been incredibly fast AND rock solid stable while TraceMonkey crashes early and often on the sites I frequent. Without stability, any performance statistics are meaningless to me.
One interesting point that it is worth mentioning is that V8 currently generates x86 code. It is currently the no1 issue on the 64-bit porting page on Chromium’s website and will also be a problem for porting the browser on other less popular architectures.
I’m not sure about TraceMonkey’s portability, current and future status. Perhaps you should enlighten us?
@Shane: What’s “fair” about any of this? V8 has been going for 2+ years with a much larger team than TraceMonkey, which is at 2+ months and still learning to crawl. Try the latest nightlies, our stability is improving. Obviously we will stabilize before shipping in Firefox 3.1. You won’t have to wait two years for stability and more speed, though — more like a few more months.
@Faidon: The TraceMonkey info at shaver’s, Andreas Gal’s, and my blogs have covered portability. We work on 64-bit x86 now, no rearchitecture required. We work on ARM as well.
/be
What Brendan said.
I would also point out, as an example, that the ARM code in TraceMonkey is less than 2,000 lines of code. It’s 2,000 lines of scary assembly, but it’s not huge and it’s well-contained.
[...] in particular, disputes some of the claims that V8 is “many times faster,” and has run some numbers to back it [...]
[...] de Google no va a superar al futuro motor de Firefox 3.1. Christopher Blizzard ha realizado una comparativa y puede verse como TraceMonkey obtiene tiempos de ejecución menores a [...]
I thought it was wierd to call the other javascript garbage collection “conservative” in contrast with v8’s “incremental”. Incremental does have an opposing design, and but conservative ain’t it. From a performance point of view, the benchmarks can be misleading. Cache hit effects might make a full collection sweep more effective than an incremental approach, to the detriment of latency.
In contrast to all of that, Conservative is for stuff like C and C++ where you have pointers and no language support for garbage collection, and has no bearing on incremental or full sweep.
Looks like Mozilla is worried about Chrome…
I don’t need benchmarks to say which engine is faster. I can say that buy running the two browsers, and Chrome kicks ass.
@jldugger: Yeah, it was an odd thing to bring up since they certainly aren’t doing GC outside of the engine. It’s not like they managed to get all objects across the board (including inside of WebKit) into the GC. We’ve been working on that inside of Mozilla right now.
Brendan, Christopher….
It’s not fair to say: “V8 has been going for 2+ years with a much larger team than TraceMonkey, which is at 2+ months and still learning to crawl.”
… c’mon guys this is simply FUD and you know it, and published in planet Gnome…. Both engines have a past and counting this way is plain tendentious
Let’s be possitive about this, it’s a new player in the game, an open source one, it’s doing fairly well, let’s grow together, no more wars please.
@Luis: I think a new “war” here is inevitable. I’m not precisely sure why Google decided to develop Chrome in secret for 2 years — it’s only stated aim (according to the comic) is to *raise the bar* for browser platforms and contribute its own ideas to the space. You don’t need to develop in secret to do that.
@Luis: V8’s approach took two years to develop – a full method/class JIT that supports one platform (x86 – not even 64 bit!) and does make JS quite a bit faster. TraceMonkey does something similar, but is much smaller and might even be faster, runs on three platorms, and has only been under development for a couple of months. These things are true.
It’s not FUD – it’s an observation of the two approaches and the relative cost and how throwing away code and starting from scratch doesn’t always result in the gains that you think it will. It’s like Google completely failed to learn from the mistakes that we made early in the Mozilla project.
And @voracity is right about the failure of Google to do this work in public. As it was stated in a meeting yesterday, we could have had faster JS a year ago if Google had done this work in public, but they didn’t and got no feedback and cost us a solid year of positive co-development. And all so they could have the big launch. I find the public value message at odds with the action on the ground in this case.
Could the comic perhaps be referring to Tamarin’s MMgc? that’s the only javascript-related GC I can think of that has any sort of conservative collection ability. It’d be pretty strange if they were, though….
Very possible. We’re not using that GC. We only took part of Tamarin, nanojit, and bolted it onto SpiderMonkey’s back. I think of it as like a Turbocharger. So we’re still using the same GC that SpiderMonkey uses.
“Incremental Garbage Collection since 1996! Woo!”
We’ve actually been looking at doing garbage collection across our entire heap. See the XPCOMGC work that’s in the Mozilla wiki. See also, the crazy pork stuff that’s been underway for a long time. It’s hot hot hot. And we’ve been doing it in the open, to boot.
Hi,
My biggest (only?) issue with Firefox is the amount of memory that it uses. It seems to use more and more until I quite literally have to kill it and open it up again.
If the team could sort out that aspect of it – I wouldn’t need to look at another browser.
Automatic downloading of fonts would be a nice-to-have. :)
Just downloaded Chrome, and it looks awesome. Not as smooth as Firefox, but it can be expected as its only in its beta stage. With the release of the Google internet browser I wouldn’t be surprised if Google released their own operating system within twenty years, and become as big as Microsoft.