Ryan’s article contains this quote from Nokia developer Oleg Romaxa:

“Nokia will use the best browser for the job,” he said. “Currently, we cannot make a full-featured and integrated browser with WebKit in mobile. But with Mozilla, we do not need to do anything, we can take existing models and API’s which are available. Also, NPAPI support is already in the Gecko web rendering engine. They are also concerned that WebKit is, to some extent, controlled by Apple, who are in competition to Nokia with their iPhone.”

There are a few important things to note here. First, that Mozilla is the complete package. We’ve got everything that you need to implement a browser. Disk cache, integrated (and well tested!) networking, a super-fast JS implementation, an XML UI markup language (XUL) and a brand that regular humans recognize. Those things mean you can get to market faster as a mobile integrator or developer instead of having to create them yourselves again.

Second, our neutral stance. We believe in the web over any particular platform. From Nokia’s standpoint if you’re building on the same technology that one of your major competitors is leading vs. working with someone who absolutely wants a web browser to succeed across all of Nokia’s platforms – which partner would you choose? I’ve often said “pick your partners carefully” and this has to be an important part of any technology decision making process.

There’s also another interesting flip side to this: who is WebKit’s other major competitor? Apple itself. Just like Microsoft’s push to get Silverlight out in the world, Apple wants people to write apps to their native platform. In this case, the iPhone. Given the strategic value of the native platform as part of Apple’s offerings, their investment in WebKit will (or at least should) always lag behind. We’re investing everything we have in the web and our platform and it’s starting to pay dividends.

And since I have your attention here are two other very interesting checkins: GTK+ and directfb (which people are actually building products on) and worker threads (ala Gears.)

Look at our current (and planned) platform support: win32, windows mobile, win32 + qt, mac OSX, linux + gtk2, linux + qt, qt embedded, linux + gtk2-directfb, x86, ppc, arm. We’re bringing the web to everyone and we’re doing it with a single coherent project with regular releases. That’s what I mean when I say “for everyone everwhere.” The web is bigger than any platform and we’re the embodiment of that mantra.

Mozilla is moving. It’s fast and furious now. And I think we’re just getting started.

[ Update: It was pointed out to me that what I wrote above might be misinterpreted as announcing that Nokia had picked a platform or something similar. Just to be clear that wasn't what I was doing, and as far as I know they haven't. I don't have knowledge about that decision inside of Nokia. Only they know. I was just pointing out what a decision making process might look like and the importance of picking well-aligned partners. And the fact that we're running on more and more platforms these days which is cool as hell. ]

Monty (don’t call him Chris) has put up a post explaining some of the work that he’s been doing with samples of much-improved ogg theora video. Note that these samples actually played very poorly in the Quicktime plugin that I have for ogg theora on my mac but worked great on my Linux box. Figures.

Chris has been calling his work “Thusnelda” because he clearly likes words that are hard to pronounce. Either that or all the good names are now taken, I’m not sure.

Anyway, If you really want to see the difference between them you only need to watch the first 10 seconds of each video. Note the pixelization in the back of the silver chair as it moves across the screen in the old video. Then watch it in the new video. The difference is really something you can see. I’ve uploaded pictures here. I suggest that you click through to both of the below images in different tabs and flip between them. (The links will open in new windows/tabs.) The difference is pretty amazing.

Old and Busted.






New Hotness!

First of all, I think that the GNOME project is in a really strange place right now. Others called it “decadence” but I’ve been thinking of it more as a state of transition. Things are changing in GNOME and I think that it’s largely just a reflection of time. GNOME has been around for more than a decade. Leadership is changing. GNOME is evolving into something else. I don’t think that I really understood this until after I was at the advisory board meeting at this year’s GUADEC.

Second, I’ll state what I said to Havoc. That I think that GNOME has evolved into two different projects, each struggling to share code and be successful. One is the “classic desktop” as we used to call it at Red Hat and the second project is built around servicing the mobile and highly-specialized desktop experience market. These are not the same thing, not by any stretch. Different audiences, different goals, different players and different revenue models. As a result you can feel some measure of friendly, but sometimes misunderstood confusion in the GNOME project because the underlying change isn’t fully understood.

A side effect of this change is also that there’s been a major change of the leadership in GNOME. It’s safe to say that in the past the North American Linux distributors played the largest part in determining the direction of GNOME. But those two companies haven’t experienced much growth over the last 10 years, either in terms of market or financial success and that’s been reflected in the relative investment that you see from the two companies.

Novell has lost most of the former Ximian people to attrition after the Novell merger and Red Hat hasn’t invested in a huge number of people relative to the growth of the rest of the company. And desktop opportunities haven’t really grown so things are at best static. Novell is still invested in apps and Red Hat is still invested in underlying infrastructure. Not much has changed on that front in a long time. It’s still all slow-growth high-touch enterprise work. Not the kind of stuff that shapes an industry, to be sure. (Note that I don’t mention Canonical who intentionally does not invest in upstream GNOME and aside from a somewhat large distribution channel, does not lead inside of GNOME.)

This is in direct contrast to the various consulting companies doing mobile work. There are at least 14 20+ people at OH, 20+ people at Collabora, 11 people at Imendio, people at Fluendo, people at Nokia, Litl, a couple of other random consulting shops in places like Spain, all leading and all contributing. These little shops are alive with energy and impact, they are writing new and interesting things and they are really the heart of where GNOME is heading. This is where the new leadership lies. And they are managing to lead as a group and share in the wealth found in GNOME work.

As a side note this signals a change in the geography of leadership of GNOME. It’s moved from the US to Europe in a very short time frame. If you want to be part of the core of GNOME you have to be in Europe. It used to be Boston in the US. That is no longer the case.

But this also signals a different kind of change. An even more important one. In the past the core of GNOME has been creating great experiences for users. GNOME love. Fixing hard technical problems up and down the stack that really solve user problems. D-Bus, HAL, PackageKit, NetworkManager – all of these low level pieces started with creating great user experiences. This is what has differentiated GNOME from KDE – creating simple and wonderful experiences for users. It was never technology focused.

But in the mobile world, and certainly one that’s driven by the financials of consulting organizations, platform is key. (Actually, this is a lie, but bear with me for a second.) The new leadership is more interested in delivering a working platform than a great experience. The experience in the mobile market is more often determined by the customer. In this case it would be the manufacturer or the mobile operator. And that’s something that GNOME has never had to deal with before, at least not in any real sense.

I will make a statement here that will probably be controversial but I believe to be true. Based on experiences with developers, GNOME’s stack is not as good as others that are on the market when it comes to platform-based decisions and platform experience. In particular, Qt and Apple’s platform are both very developer friendly and make it very easy for developers to create good-looking, modern looking apps. And they have more time and experience in embedded environments. With GTK+ it’s possible to make things look good and work fast, but it’s very difficult. If in a platform-driven market and a platform-driven world you’re not the #1 or #2 player it’s going to be very difficult to make a dent in the market. (This is especially true if Nokia decides to fix the Qt licensing.)

Basically what I’m saying is that if mobile is one of the main roads for GNOME (and it clearly is) then platform and embedding needs to be a core competency of the entire project. It can’t be just about experience. And I don’t think that we’re there today. That’s one of the big risks that we face.

I made a statement earlier, one that’s actually a lie. That platform was the key component in the mobile space. But I would actually argue that it’s not. End user experience, which can be driven by the platform in a bottom up fashion (as opposed to top down) is extremely important. Apple showed us that. The ability to make decisions about user experience in mobile is something that hasn’t historically been up to platform people. But I believe that in order to really be successful that GNOME needs to find a way to do this.

I believe that this is also one of the things that is a big downside of having a lot of small consulting shops all working on the same project. While it’s nice to have that diversity, it’s also very difficult to come to decisions about experience. And no one right now is focused on a complete product and experience. Everyone is building tools to let someone else make a decision.

Products almost always win. They lower costs for buyers and producers alike and increase leverage for the people who build the products. They offer the producer the ability to make decisions. And that’s the most important factor in building great experiences for people. Weak products are the result of poor decision making and a lack of design direction. And I worry that’s where GNOME is going. To a place where we’re not at our core competency and we aren’t building things that actual people will care about. That’s not historically what GNOME has been about, and I worry that’s where it’s going.

That’s also why I’m calling this a duality. Two projects existing as one. One without a great deal of success, but an end-user identity and brand and goals, and another with a chance to succeed but without much identity or end-user goals. It’s going to be a rough ride and I think that in order to find success we have to find a way to merge the two into a single set of goals. The desktop isn’t going anywhere, but the mobile project isn’t going to produce GNOME-visible results either. GNOME will be well-hidden behind someone else’s branding and experience. And maybe that’s fine, but it’s not the way to lead and win in the end.

I think the GTK+ 3.0 discussions are a symptom of this. They are sold as a technology change, a chance to clean up, but without a specific user-driven direction as to why someone would want to make that change. (Or at least that’s how it was presented at the AB meeting.) If someone could describe why that change should be made, that would be great. How will it help users? How will it help us achieve our goals? (Holy crap, what are our goals anyway?) How does it help bridge the gap between desktop and mobile? These are important questions, and I think a lot more discussion needs to take place.

So those are my thoughts on the GNOME project today. Hope it helps bring some understanding to how GNOME can more effectively manage itself.

At the end I gave some demos of various bits of what’s coming down the pipe for Firefox 3.next. There’s a bunch of stuff on the web about it now. Here are some links of interest:

Robert O’Callahan

• SVG Paint Servers For HTML
• Using Arbitrary Elements As Paint Servers

Chris Double

• Video Bling
• Chris’ video test page – my tests came from here

Vladimir Vukićević

• Canvas 3D Extension Update

If you’re interested in the slides you can see them on my mozilla people page.

Good luck, Stormy!

He mentions that he’s working with us at Mozilla – very true. And good fodder for another post on another day.

He also touches on an important topic for me, which is the difference between between being “open” and being “good.” Being open is easy. Being good is not. I’ll write more about that at a later date as well.

Atul Varma talks about how Weave works and some of the wider goals for the project.

Weave is a very interesting part of the Mozilla project. The underlying mechanisms are very simple (just a webdav store and a private and public key system, basically.) But the fact that we’ll be able to use it to share data between users on their terms and extension authors should be able to take advantage of the storage that’s there – well – it could be huge. Lots of great potential here.

He also touches on an important part of weave. That we want user data to be private and we want users to control what they are exposing to the world. Mozilla’s servers can’t read the data that you post and neither can anyone else. That’s an important piece of what Weave represents.

Google drops the Google from Gears and will be adding Firefox 3 support.

It’s clear that Google is trying to build a larger community around the Gears plugin and is willing to give up branding in order to try and make that a reality. Also good that they are porting to Firefox 3 given that a huge chunk of the world is about to update to it.

Jorge Castro talks about his views on Mozilla’s commitment and investment in Linux.

Jorge is a pretty fantastic guy and I’m happy that he’s willing to jump out there and say something about the hard work that our community put in for this release of Firefox on Linux support. Great native integration, memory and performance improvements, a great experience for users, etc. Good to see him focus on the big picture which is how much better Firefox 3 will be vs. Firefox 2.

Anyway, that’s it for Friday. Thanks for tuning in!

There are some patches required to some of the GNOME stack to get things up and running. The patches are up on the page.

Good work, guys!

I’ve always seen this position as both good and bad. It’s good for the distribution end users since they don’t have to go and find a build of Firefox. And it’s good for Mozilla because it means that we don’t have to produce builds for N Linux distributions, which is basically an impossible task. It also means that distributions can make late-breaking fixes that are specific to their distribution that really affect their particular user base.

But it’s bad, too. It means we’re disconnected from users on those various Linux distributions. We’re at the mercy of the distributions to make updates in a timely manner, and very often we find ourselves chasing them to make updates that they clearly should be doing. For those users where we have a direct relationship we have a pretty good track record of making timely updates to almost our entire user base. Linux users are cut off – quite intentionally because that is the classic “value add” for a Linux distributor – from our update train, sometimes leaving them vulnerable for weeks or months. (Note that this does not generally affect the top-tier Linux distributors like Canonical, Red Hat and Novell. They are actually excellent at delivering updates because they have dedicated engineers who only have the job to chase Mozilla and be ready when we’re ready to deliver an update – either a firedrill or planned update release.)

There’s another large downside as well. Distributions often make changes to how Firefox is built – be it compiler optimization changes, linking with system libraries instead of the ones that we ship with, adding their own large patches to add support for some random feature or making changes to the default look and feel of the browser.

Contrary to uninformed hyperbole Mozilla actually does a huge amount of testing on Linux. It’s one of our three top tier platforms and we run it through all the same regression and performance testing that the other platforms get. You can see this by looking how much attention we pay to how to tune the compiler to give us the best performance (hint: more -O doesn’t always make us faster, nor does architecture-specific tuning!) or how much time we’ve spent on the reported fsync issues that have affected quite a few people.

Because of the amount of work that we do on Linux and how closely we work with upstream projects (sqlite, cairo, etc) we’re still the experts on what works and what doesn’t. And because we have a pretty full set of tests that we go through we know what versions of upstream projects work well inside of Mozilla. Note that this doesn’t mean we know which versions don’t work with Mozilla, as I will illustrate later. We can’t be compatible with every single version of every upstream project with every single possible configuration, it just doesn’t work.

I’ll use a specific case in point here to illustrate what I’m talking about with Fedora + Red Hat. (Note that I’m pointing this out because it’s a real situation, not that I think that the Fedora + Red Hat guys are doing a bad job – they actually do a fantastic job given the task as far as I’m concerned. The issue I’m about to describe does not affect Fedora 8 or Fedora 9 users – only those who happen to be using Fedora Rawide – the bleeding edge of the bleeding edge.)

Chris Aillon and Matthias Clasen were reporting an issue to me where Firefox was hanging for long periods of time in Fedora Rawhide while opening the history tab. I figured that it was the same old fsync-related problem but they were reporting that it was happening for long periods of time (30 seconds in a lot of cases) and it was happening on systems that were relatively unloaded in terms of IO. I was near the Red Hat office on a personal errand and I thought I would stop by and try to help diagnose the issue. Looking at the issue in a debugger I found that it was hanging down in sqlite and not returning into Mozilla-specific code at all. I also noticed that they were linked against the system sqlite instead of the one that we ship with. I asked Matthias to try a Mozilla-built Firefox on his machine with his profile and it did not have any problems. When Chris generated a build for Fedora Rawhide that used our internal sqlite version it also didn’t have any problems.

It turns out that the sqlite version that’s included in Rawhide, version 3.5.8, has a bug in a particular type of query that Mozilla uses extensively. When Mozilla updated to that version of sqlite our automated testing picked up on the problem and the change was backed out of our tree. Let’s look at the order of operations that caused this particular issue.

1. Mozilla checks in a patch to upgrade to sqlite version 3.5.8.
2. Fedora Rawhide notices the new requirement in configure and bumps their system sqlite version to 3.5.8.
3. Mozilla’s automated testing picks up failures as a result of the new sqlite version and backs out the changes.
4. That backout is missed by the Fedora folks and they are left linking against an sqlite version that contains the problem.

This isn’t the worst example of what goes on when distributors are making changes to upstream software. The impact here was pretty minor – only a few users were affected and the bug was pretty obvious to a large number of people. It does get worse. Ask Debian and Ubuntu users how happy they are about regenerating keys in light of the OpenSSL issues that were recently found with the downstream patch. (I realize that’s an oversimplification of that particular issue but it has a lot of the same qualities as this example.)

This is a real problem, one that we’ve even successfully predicted.

So how does this relate to the fsync issue? Well it shows the opposite end of the of the patch spectrum. Basically every single Linux distribution is waiting for a good fix to that particular problem. And they will all ship a fix to their users. So sometimes distro-specific patching is a good thing.

The trick has to be finding the balance. Right now we know that there are a lot of instances where bad or ignorant decisions are being made. (Just because an option exists in ./configure doesn’t mean that you should use it!) People clearly aren’t taking advantage of Mozilla’s automated testing facilities – in the Fedora example the problem would have shown up pretty quickly if they were running the same tests Mozilla does. And the flows of information between Mozilla and the various distributions is ad-hoc at best. Very often more effort is spent debugging the blame instead of debugging the actual problem at hand. I’ve been on the receiving end of that recently and it’s certainly soul-crushing.

There’s also no easy answer to the multiple-library-version problem, either. Once again, we’re not going to be compatible with everything everywhere, especially on Linux where the platform is more like quicksand than green grass. Just screaming “you should always link against system libraries!!!!” isn’t going to work when the size and complexity of Linux continues to expand without any contraction of the complexity involved in which Linux you should target. That with blind version updating means that we’re just going to be stuck with multiple versions of libraries – assuming you want a quality product that works as well as it does on Windows and the Mac and does so consistently.

So if I had to wrap up this post with some lessons learned I would put them down as this:

1. Don’t change a default configure option unless there’s a very good and very specific reason for it. No, really, don’t touch the optimization flags – we’ve probably tuned them to an inch of the compiler’s life. But if you actually run tests and find something faster, let us know.
2. Don’t ship a patch unless it’s been vetted by upstream.
3. Ship known good patches if it will help your users. But don’t do it without talking to us first. Remember, it’s possible – and likely – that we’ve known about your issue for quite a while. And while you’re at it if you had to fix something consider adding a test to our test suite so it won’t happen again?
4. If you do want to carry some heavy patches consider following the pattern that the enterprise distributions are using: they work together (at least Ubuntu, Novell and Red Hat) to carry patches for some pretty old Firefox releases. And have been relatively successful doing it.
5. And finally, remember that there’s a fine line between adding real value and making a change for change’s sake. The former is always encouraged (within limits!) but the latter often causes more trouble than it is worth.

In closing I think I would just like to say that there’s a lot of work to be done here. And it’s something that will need constant adjustment – there’s no one set of rules that we can develop and expect them never to change. Both Linux distributors and Mozilla can do better than they have done to date in terms of making things better for users on Linux. Thinking about changes, improving communication, understanding the reason for changes or diversion from upstream, etc.

And all of that discussion has to come from the standpoint of making sure that the user’s experience is improved. If it doesn’t improve their experience in a very tangible way, it’s probably not worth doing.