I’ve always seen this position as both good and bad. It’s good for the distribution end users since they don’t have to go and find a build of Firefox. And it’s good for Mozilla because it means that we don’t have to produce builds for N Linux distributions, which is basically an impossible task. It also means that distributions can make late-breaking fixes that are specific to their distribution that really affect their particular user base.

But it’s bad, too. It means we’re disconnected from users on those various Linux distributions. We’re at the mercy of the distributions to make updates in a timely manner, and very often we find ourselves chasing them to make updates that they clearly should be doing. For those users where we have a direct relationship we have a pretty good track record of making timely updates to almost our entire user base. Linux users are cut off – quite intentionally because that is the classic “value add” for a Linux distributor – from our update train, sometimes leaving them vulnerable for weeks or months. (Note that this does not generally affect the top-tier Linux distributors like Canonical, Red Hat and Novell. They are actually excellent at delivering updates because they have dedicated engineers who only have the job to chase Mozilla and be ready when we’re ready to deliver an update – either a firedrill or planned update release.)

There’s another large downside as well. Distributions often make changes to how Firefox is built – be it compiler optimization changes, linking with system libraries instead of the ones that we ship with, adding their own large patches to add support for some random feature or making changes to the default look and feel of the browser.

Contrary to uninformed hyperbole Mozilla actually does a huge amount of testing on Linux. It’s one of our three top tier platforms and we run it through all the same regression and performance testing that the other platforms get. You can see this by looking how much attention we pay to how to tune the compiler to give us the best performance (hint: more -O doesn’t always make us faster, nor does architecture-specific tuning!) or how much time we’ve spent on the reported fsync issues that have affected quite a few people.

Because of the amount of work that we do on Linux and how closely we work with upstream projects (sqlite, cairo, etc) we’re still the experts on what works and what doesn’t. And because we have a pretty full set of tests that we go through we know what versions of upstream projects work well inside of Mozilla. Note that this doesn’t mean we know which versions don’t work with Mozilla, as I will illustrate later. We can’t be compatible with every single version of every upstream project with every single possible configuration, it just doesn’t work.

I’ll use a specific case in point here to illustrate what I’m talking about with Fedora + Red Hat. (Note that I’m pointing this out because it’s a real situation, not that I think that the Fedora + Red Hat guys are doing a bad job – they actually do a fantastic job given the task as far as I’m concerned. The issue I’m about to describe does not affect Fedora 8 or Fedora 9 users – only those who happen to be using Fedora Rawide – the bleeding edge of the bleeding edge.)

Chris Aillon and Matthias Clasen were reporting an issue to me where Firefox was hanging for long periods of time in Fedora Rawhide while opening the history tab. I figured that it was the same old fsync-related problem but they were reporting that it was happening for long periods of time (30 seconds in a lot of cases) and it was happening on systems that were relatively unloaded in terms of IO. I was near the Red Hat office on a personal errand and I thought I would stop by and try to help diagnose the issue. Looking at the issue in a debugger I found that it was hanging down in sqlite and not returning into Mozilla-specific code at all. I also noticed that they were linked against the system sqlite instead of the one that we ship with. I asked Matthias to try a Mozilla-built Firefox on his machine with his profile and it did not have any problems. When Chris generated a build for Fedora Rawhide that used our internal sqlite version it also didn’t have any problems.

It turns out that the sqlite version that’s included in Rawhide, version 3.5.8, has a bug in a particular type of query that Mozilla uses extensively. When Mozilla updated to that version of sqlite our automated testing picked up on the problem and the change was backed out of our tree. Let’s look at the order of operations that caused this particular issue.

1. Mozilla checks in a patch to upgrade to sqlite version 3.5.8.
2. Fedora Rawhide notices the new requirement in configure and bumps their system sqlite version to 3.5.8.
3. Mozilla’s automated testing picks up failures as a result of the new sqlite version and backs out the changes.
4. That backout is missed by the Fedora folks and they are left linking against an sqlite version that contains the problem.

This isn’t the worst example of what goes on when distributors are making changes to upstream software. The impact here was pretty minor – only a few users were affected and the bug was pretty obvious to a large number of people. It does get worse. Ask Debian and Ubuntu users how happy they are about regenerating keys in light of the OpenSSL issues that were recently found with the downstream patch. (I realize that’s an oversimplification of that particular issue but it has a lot of the same qualities as this example.)

This is a real problem, one that we’ve even successfully predicted.

So how does this relate to the fsync issue? Well it shows the opposite end of the of the patch spectrum. Basically every single Linux distribution is waiting for a good fix to that particular problem. And they will all ship a fix to their users. So sometimes distro-specific patching is a good thing.

The trick has to be finding the balance. Right now we know that there are a lot of instances where bad or ignorant decisions are being made. (Just because an option exists in ./configure doesn’t mean that you should use it!) People clearly aren’t taking advantage of Mozilla’s automated testing facilities – in the Fedora example the problem would have shown up pretty quickly if they were running the same tests Mozilla does. And the flows of information between Mozilla and the various distributions is ad-hoc at best. Very often more effort is spent debugging the blame instead of debugging the actual problem at hand. I’ve been on the receiving end of that recently and it’s certainly soul-crushing.

There’s also no easy answer to the multiple-library-version problem, either. Once again, we’re not going to be compatible with everything everywhere, especially on Linux where the platform is more like quicksand than green grass. Just screaming “you should always link against system libraries!!!!” isn’t going to work when the size and complexity of Linux continues to expand without any contraction of the complexity involved in which Linux you should target. That with blind version updating means that we’re just going to be stuck with multiple versions of libraries – assuming you want a quality product that works as well as it does on Windows and the Mac and does so consistently.

So if I had to wrap up this post with some lessons learned I would put them down as this:

1. Don’t change a default configure option unless there’s a very good and very specific reason for it. No, really, don’t touch the optimization flags – we’ve probably tuned them to an inch of the compiler’s life. But if you actually run tests and find something faster, let us know.
2. Don’t ship a patch unless it’s been vetted by upstream.
3. Ship known good patches if it will help your users. But don’t do it without talking to us first. Remember, it’s possible – and likely – that we’ve known about your issue for quite a while. And while you’re at it if you had to fix something consider adding a test to our test suite so it won’t happen again?
4. If you do want to carry some heavy patches consider following the pattern that the enterprise distributions are using: they work together (at least Ubuntu, Novell and Red Hat) to carry patches for some pretty old Firefox releases. And have been relatively successful doing it.
5. And finally, remember that there’s a fine line between adding real value and making a change for change’s sake. The former is always encouraged (within limits!) but the latter often causes more trouble than it is worth.

In closing I think I would just like to say that there’s a lot of work to be done here. And it’s something that will need constant adjustment – there’s no one set of rules that we can develop and expect them never to change. Both Linux distributors and Mozilla can do better than they have done to date in terms of making things better for users on Linux. Thinking about changes, improving communication, understanding the reason for changes or diversion from upstream, etc.

And all of that discussion has to come from the standpoint of making sure that the user’s experience is improved. If it doesn’t improve their experience in a very tangible way, it’s probably not worth doing.

Above is a video that I took of Chris Double’s awesome whatwg video work running on my N810. Note that I didn’t have to do anything special to get this to work. I took the two patches, applied them to my trunk mozilla build and ran them through the ARM toolchain. The code works out of the box with the gstreamer backend that’s included on the N810.

I thought that it was important to show that we’re not just serious about adding video support to Mozilla, but that it can extend to a wide range of devices. If we want to have a world that includes rich internet applications, we have to be sure that includes phones and lower end devices as well.

The video itself is a speech that Cory Doctorow gave at the 2006 Red Hat Summit about copyright and intellectual property. I thought it was fitting.

If you’re a crazy maemo hacker you can get a copy of the build from my people page that includes gstreamer support. Follow Brad’s instructions to get a browser front end once you’re installed xulrunner somewhere. If you want to play a video you should make sure that you’ve encoded it for the N810 using a decent script to take advantage of the hardware accel on the device. Hack up an html page, embed the video like on Chris’ test page and try it out.

Interview with John Lilly on GigaOM

Interview starts about 4 minutes in.

There’s a large number of people in the Mozilla community (outside of the Corporation staff) that don’t know John very well. This interview starts to give a sense of him. Honest, intense and very thoughtful. It’s a good interview and talks a little bit about what Mozilla is about and how he feels he fits into the bigger picture. Worth watching.

For the Record Weblog Launches

For the Record is something that Asa started to help process, catalog and understand what’s being said about Mozilla and Firefox online. The first post talks about the Firefox release cycle and how we get to the point where we’re ready to release.

This is a good companion to the newly-launched about:mozilla newsletter. If you want some low-traffic, high-quality weekly access to what’s going on in Mozilla this is another good place to get it. Signing up is pretty easy.

Mozilla Mobile Talk in Japan

Christian and Jay will be in Tokyo, Japan to talk about Mozilla’s mobile strategy. If you’re in the area, it’s a pay event, but it’s probably worth attending.

Podcast Enhancements in Firefox 3

Will Guaraldi talks about the work that he did to include better podcast support in Firefox 3. Will spends most of his time working on Miro, a wonderful counterpart to Firefox for video podcasts. (Seriously, why aren’t you using Miro already? It’s a great app. Go try it. Right now.) I ran across this feature today when I was looking at some feeds for a friend’s photo site and it’s pretty nice. It shows the inline enclosures and types and gives you a sense of what you’re about to get. Very nice.

D-Bus Support for Extensions in Firefox

Eric mentions my interest in his D-Bus JS bindings. True – I am interested! He’s apparently trying to get help on the Mozilla Linux Platform mailing list and trying to integrate a bunch of stuff. Assuming that he does, it will be a great way to get some more great integration when you’re using Linux. Being able to connect extensions to other services on the desktop and being able to use them to affect your web experience sounds pretty good to me.

Matt Asay on the Future of the Web

Matt Asay posts about his thoughts about the web, including my talk at SCALE. I don’t know Matt personally, but I do read his weblog. Given how much attention he pays to Red Hat and the other Linux vendors, I always assumed that he knew what was going on with us, our current impact and our future. I was wrong. We’re a fantastically transparent project, but that also means that it’s sometimes hard to find the clear message and the meaning in all the noise. Efforts like For the Record will hopefully make it easy for people to understand the meaning we’re trying to create in the world.

As for what’s next, starting in mid-November I will be joining the Evangelism team at Mozilla Corporation. Working with Shaver, Deb, Eric, John and Mark to help tell the story of the Open Web. My role will be to work with other open source projects that are well aligned with Mozilla’s mission and help them take part in writing that story.

I’m incredibly excited about this opportunity. The people at Mozilla are fantastic and what they have done so far has been great for everyone who uses the Internet. I’m hoping that by joining them and working with them on a day to day basis that I can help accelerate the accomplishment of their mission.

It’s going to be a great time.

OK, to the meat of the story. AMD is making the commitment to do two major things:

• To develop of a fully functional 2D and 3D driver that supports all of their newer radeon chipsets. This will be done in full collaboration with the open source community and will have the direct participation of hackers from companies like Red Hat and Novell.
• To release documentation that anyone can use to build and support drivers for their chips.

In my mind it’s the release of documentation that’s most interesting and telling about the commitment. These guys are clearly doing the right thing and are going even further than Intel in their support of open source. It’s not just about the having drivers, it’s also about having the ability to work independently of the company in your development and decision making. Docs make that possible and are a great symptom of the way that they are thinking about how to interact with the open source community.

In my mind end users turn out to be the biggest winners in this story. The binary driver that AMD/ATI has today will continue to live on and be supported. But that doesn’t matter because the goal wasn’t to stop the evil binary driver makers. The goal was to create a great out of the box experience for people who want to use distributions like Fedora and Red Hat Enterprise Linux. And what this means is that we can finally do this on ATI hardware across the board.

It’s also a huge win for those of us who understand that supporting end users and creating great experiences means having open source up and down the stack. For those of us who have been involved in the Fedora project, not including and supporting binary drivers has been painful. It destroyed our user base versus those who would be so quick to give up that flexibility for the sake of some short term gain. But we stuck to our guns and said “No, we can’t innovate in that model. That doesn’t scale and takes away our ability to move quickly. You will have to do better.” And it meant that in some small way we were able to drive the discussion to a place where open source becomes part of the answer instead of part of the problem. It turns it into an opportunity for growth. For those of you who stuck with us through the hard times in Fedora, we thank you. It was worth it and everyone who uses Linux, Fedora or not, will benefit.

It will take a while for the driver to turn from a framework into something useful. And it will take a while for all of the documentation to get published as well. (I hear the 2D docs will come first, followed after a period of time by the 3D docs.) So people will have to be patient. But they have made the commitment and decision to do the right thing. So, once again, thanks to the people inside of AMD who made this happen. We’re looking forward to working with you.

Update: Have a look at Dave Arlie’s post for some more specific information about how things have been moving to date.

Update: From Daniel Stone:

Matthew Tippett just threw a CD full of AMD/ATI specs to Dave Airlie, approved for public release with no NDA. I’m now holding one of those CDs as well. Thanks, AMD.

In the past five hours, we’ve pushed 72,000 PDFs, for a total of around 1TB. This … turns out to actually push the limits of our new fd.o setup.

This is really going to help our story around OLPC. Right now the OLPC XO, like many machines, takes a long time to start up. Far too long. Part of this is the machine’s performance and the fact that we’re still loading a lot of legacy stuff. (RAID? Do we need to check for RAID partitions in the OLPC startup scripts? I’m guessing no!) But when both David and I started looking at it a year ago I think that we both came to the conclusion that the way that we start the desktop it was fundamentally broken, especially on that little machine. That’s why we (and when I mean we I really mean David with me looking over his shoulder) pursued things like mayflower and looking at what architectural improvements we could make to startup. That service activation patch is part of the eventual solution.

The last piece of the puzzle from where I sit is udev. The current set of Fedora startup scripts along with the way that udev is set up takes a whopping 20 seconds or so to start up on the OLPC XO. Why? Well it’s not udev, exactly. Mayflower embeds udev and it starts up in under a second. The problem appears to be the rules, the programs that are run as a result of the rules, and the way that kernel events are handled.

Right now kernel events are handled by udev. Those events are matched against rules in udev’s config files. Those config files specify certain actions. Some of those actions have terrible performance-destroying side effects. Sometimes the rules specify creating device nodes. The device nodes that we create we have to set the console permissions on those nodes. That’s done by running a program after the device is created. In addition there is a global rule that runs modprobe for every event that comes in no matter if it creates a device node. That also requires running a program. On startup, the kernel sends out anywhere from several dozen to several hundred events that result in device creation. Imagine running a program for each of those items. No do that on a machine that runs at about 466mhz. 20 seconds, indeed.

I honestly believe that we can achieve a 20 second startup time for a desktop machine in Fedora if we take an agressive notion to what startup means and continues down this path to a sane architecture. I also believe that we can get the OLPC XO machine started up in that same amount of time, if for no other reason than the number of things we need to load on startup is so much lower. I think that David has a good sense of the architectural things that need to change in udev to make it scale better. He and Kay had a roadmap at one point but I don’t know what happened to it. But it’s a key to getting to that 20 second number. I think we can do it.

Download this video: [Ogg Theora]

Video by Simple Coat.
Produced by Julie Bryce and Kristin Hondros.

I’m looking forward to the day I can upgrade from the old x31 that I’m still using. Richard, just let me know when everything on that x60 works out of the box and I’ll try to get a celebratory purchase order. (I wonder if anyone in the history of the world has ever used the phrase “celebratory purchase order“. Doesn’t appear so.)