I go out to the store this morning and buy one, bring it home, plug it in and I decide to try and figure out what kinds of loads various things sitting on my desk create. I have a little Verizon router for FIOS – it uses about 5 watts. The external USB hard drive about 8 watts. The mac mini that I have running Fedora 8 on it uses about 7 watts. Not too bad, all things considered.

So I decided to drop the router and the drive and find out how much the mini drew when it was running and how much it drew when it was suspended. So I unplugged them and then suspended the mini. The power went from 7 watts to 10. Huh? I started the mini back up and the power usage dropped back to 7. Suspend, it jumps back up to 10. That’s right. My mini running Fedora 8 uses more power when it’s suspended than when it’s sitting at an idle desktop.

I have a new rule. Richard Hughes isn’t allowed to complain about anything Firefox-related until my computers running Linux use less power suspended than when they are sitting at an idle desktop.

I’ve always seen this position as both good and bad. It’s good for the distribution end users since they don’t have to go and find a build of Firefox. And it’s good for Mozilla because it means that we don’t have to produce builds for N Linux distributions, which is basically an impossible task. It also means that distributions can make late-breaking fixes that are specific to their distribution that really affect their particular user base.

But it’s bad, too. It means we’re disconnected from users on those various Linux distributions. We’re at the mercy of the distributions to make updates in a timely manner, and very often we find ourselves chasing them to make updates that they clearly should be doing. For those users where we have a direct relationship we have a pretty good track record of making timely updates to almost our entire user base. Linux users are cut off – quite intentionally because that is the classic “value add” for a Linux distributor – from our update train, sometimes leaving them vulnerable for weeks or months. (Note that this does not generally affect the top-tier Linux distributors like Canonical, Red Hat and Novell. They are actually excellent at delivering updates because they have dedicated engineers who only have the job to chase Mozilla and be ready when we’re ready to deliver an update – either a firedrill or planned update release.)

There’s another large downside as well. Distributions often make changes to how Firefox is built – be it compiler optimization changes, linking with system libraries instead of the ones that we ship with, adding their own large patches to add support for some random feature or making changes to the default look and feel of the browser.

Contrary to uninformed hyperbole Mozilla actually does a huge amount of testing on Linux. It’s one of our three top tier platforms and we run it through all the same regression and performance testing that the other platforms get. You can see this by looking how much attention we pay to how to tune the compiler to give us the best performance (hint: more -O doesn’t always make us faster, nor does architecture-specific tuning!) or how much time we’ve spent on the reported fsync issues that have affected quite a few people.

Because of the amount of work that we do on Linux and how closely we work with upstream projects (sqlite, cairo, etc) we’re still the experts on what works and what doesn’t. And because we have a pretty full set of tests that we go through we know what versions of upstream projects work well inside of Mozilla. Note that this doesn’t mean we know which versions don’t work with Mozilla, as I will illustrate later. We can’t be compatible with every single version of every upstream project with every single possible configuration, it just doesn’t work.

I’ll use a specific case in point here to illustrate what I’m talking about with Fedora + Red Hat. (Note that I’m pointing this out because it’s a real situation, not that I think that the Fedora + Red Hat guys are doing a bad job – they actually do a fantastic job given the task as far as I’m concerned. The issue I’m about to describe does not affect Fedora 8 or Fedora 9 users – only those who happen to be using Fedora Rawide – the bleeding edge of the bleeding edge.)

Chris Aillon and Matthias Clasen were reporting an issue to me where Firefox was hanging for long periods of time in Fedora Rawhide while opening the history tab. I figured that it was the same old fsync-related problem but they were reporting that it was happening for long periods of time (30 seconds in a lot of cases) and it was happening on systems that were relatively unloaded in terms of IO. I was near the Red Hat office on a personal errand and I thought I would stop by and try to help diagnose the issue. Looking at the issue in a debugger I found that it was hanging down in sqlite and not returning into Mozilla-specific code at all. I also noticed that they were linked against the system sqlite instead of the one that we ship with. I asked Matthias to try a Mozilla-built Firefox on his machine with his profile and it did not have any problems. When Chris generated a build for Fedora Rawhide that used our internal sqlite version it also didn’t have any problems.

It turns out that the sqlite version that’s included in Rawhide, version 3.5.8, has a bug in a particular type of query that Mozilla uses extensively. When Mozilla updated to that version of sqlite our automated testing picked up on the problem and the change was backed out of our tree. Let’s look at the order of operations that caused this particular issue.

1. Mozilla checks in a patch to upgrade to sqlite version 3.5.8.
2. Fedora Rawhide notices the new requirement in configure and bumps their system sqlite version to 3.5.8.
3. Mozilla’s automated testing picks up failures as a result of the new sqlite version and backs out the changes.
4. That backout is missed by the Fedora folks and they are left linking against an sqlite version that contains the problem.

This isn’t the worst example of what goes on when distributors are making changes to upstream software. The impact here was pretty minor – only a few users were affected and the bug was pretty obvious to a large number of people. It does get worse. Ask Debian and Ubuntu users how happy they are about regenerating keys in light of the OpenSSL issues that were recently found with the downstream patch. (I realize that’s an oversimplification of that particular issue but it has a lot of the same qualities as this example.)

This is a real problem, one that we’ve even successfully predicted.

So how does this relate to the fsync issue? Well it shows the opposite end of the of the patch spectrum. Basically every single Linux distribution is waiting for a good fix to that particular problem. And they will all ship a fix to their users. So sometimes distro-specific patching is a good thing.

The trick has to be finding the balance. Right now we know that there are a lot of instances where bad or ignorant decisions are being made. (Just because an option exists in ./configure doesn’t mean that you should use it!) People clearly aren’t taking advantage of Mozilla’s automated testing facilities – in the Fedora example the problem would have shown up pretty quickly if they were running the same tests Mozilla does. And the flows of information between Mozilla and the various distributions is ad-hoc at best. Very often more effort is spent debugging the blame instead of debugging the actual problem at hand. I’ve been on the receiving end of that recently and it’s certainly soul-crushing.

There’s also no easy answer to the multiple-library-version problem, either. Once again, we’re not going to be compatible with everything everywhere, especially on Linux where the platform is more like quicksand than green grass. Just screaming “you should always link against system libraries!!!!” isn’t going to work when the size and complexity of Linux continues to expand without any contraction of the complexity involved in which Linux you should target. That with blind version updating means that we’re just going to be stuck with multiple versions of libraries – assuming you want a quality product that works as well as it does on Windows and the Mac and does so consistently.

So if I had to wrap up this post with some lessons learned I would put them down as this:

1. Don’t change a default configure option unless there’s a very good and very specific reason for it. No, really, don’t touch the optimization flags – we’ve probably tuned them to an inch of the compiler’s life. But if you actually run tests and find something faster, let us know.
2. Don’t ship a patch unless it’s been vetted by upstream.
3. Ship known good patches if it will help your users. But don’t do it without talking to us first. Remember, it’s possible – and likely – that we’ve known about your issue for quite a while. And while you’re at it if you had to fix something consider adding a test to our test suite so it won’t happen again?
4. If you do want to carry some heavy patches consider following the pattern that the enterprise distributions are using: they work together (at least Ubuntu, Novell and Red Hat) to carry patches for some pretty old Firefox releases. And have been relatively successful doing it.
5. And finally, remember that there’s a fine line between adding real value and making a change for change’s sake. The former is always encouraged (within limits!) but the latter often causes more trouble than it is worth.

In closing I think I would just like to say that there’s a lot of work to be done here. And it’s something that will need constant adjustment – there’s no one set of rules that we can develop and expect them never to change. Both Linux distributors and Mozilla can do better than they have done to date in terms of making things better for users on Linux. Thinking about changes, improving communication, understanding the reason for changes or diversion from upstream, etc.

And all of that discussion has to come from the standpoint of making sure that the user’s experience is improved. If it doesn’t improve their experience in a very tangible way, it’s probably not worth doing.

As for what’s next, starting in mid-November I will be joining the Evangelism team at Mozilla Corporation. Working with Shaver, Deb, Eric, John and Mark to help tell the story of the Open Web. My role will be to work with other open source projects that are well aligned with Mozilla’s mission and help them take part in writing that story.

I’m incredibly excited about this opportunity. The people at Mozilla are fantastic and what they have done so far has been great for everyone who uses the Internet. I’m hoping that by joining them and working with them on a day to day basis that I can help accelerate the accomplishment of their mission.

It’s going to be a great time.

OK, to the meat of the story. AMD is making the commitment to do two major things:

• To develop of a fully functional 2D and 3D driver that supports all of their newer radeon chipsets. This will be done in full collaboration with the open source community and will have the direct participation of hackers from companies like Red Hat and Novell.
• To release documentation that anyone can use to build and support drivers for their chips.

In my mind it’s the release of documentation that’s most interesting and telling about the commitment. These guys are clearly doing the right thing and are going even further than Intel in their support of open source. It’s not just about the having drivers, it’s also about having the ability to work independently of the company in your development and decision making. Docs make that possible and are a great symptom of the way that they are thinking about how to interact with the open source community.

In my mind end users turn out to be the biggest winners in this story. The binary driver that AMD/ATI has today will continue to live on and be supported. But that doesn’t matter because the goal wasn’t to stop the evil binary driver makers. The goal was to create a great out of the box experience for people who want to use distributions like Fedora and Red Hat Enterprise Linux. And what this means is that we can finally do this on ATI hardware across the board.

It’s also a huge win for those of us who understand that supporting end users and creating great experiences means having open source up and down the stack. For those of us who have been involved in the Fedora project, not including and supporting binary drivers has been painful. It destroyed our user base versus those who would be so quick to give up that flexibility for the sake of some short term gain. But we stuck to our guns and said “No, we can’t innovate in that model. That doesn’t scale and takes away our ability to move quickly. You will have to do better.” And it meant that in some small way we were able to drive the discussion to a place where open source becomes part of the answer instead of part of the problem. It turns it into an opportunity for growth. For those of you who stuck with us through the hard times in Fedora, we thank you. It was worth it and everyone who uses Linux, Fedora or not, will benefit.

It will take a while for the driver to turn from a framework into something useful. And it will take a while for all of the documentation to get published as well. (I hear the 2D docs will come first, followed after a period of time by the 3D docs.) So people will have to be patient. But they have made the commitment and decision to do the right thing. So, once again, thanks to the people inside of AMD who made this happen. We’re looking forward to working with you.

Update: Have a look at Dave Arlie’s post for some more specific information about how things have been moving to date.

Update: From Daniel Stone:

Matthew Tippett just threw a CD full of AMD/ATI specs to Dave Airlie, approved for public release with no NDA. I’m now holding one of those CDs as well. Thanks, AMD.

In the past five hours, we’ve pushed 72,000 PDFs, for a total of around 1TB. This … turns out to actually push the limits of our new fd.o setup.

But now he’s taken it to the next level with Transifex. Here’s how it works. You’re a small project and you’re hosted somewhere. You want translators. Very often you have to build your own community by attracting people, setting up accounts, etc. This isn’t a problem for the larger projects like GNOME and KDE which have their own communities, but it’s harder for smaller projects. With Transifex what Dimitris has done is set it up so that a Fedora translator can, via the Fedora site, have new translations checked into the upstream project. This is done through a single “fedora” account that the project has set up and checkins through that account will contain account information about who checked it in and how to contact that person. He’s setting it up so that it will support the most common revision control systems (git, cvs, svn, hg) and it should be seamless from the standpoint of the upstream projects and the translators.

While I’m sure that a lot of projects won’t want to use this service, it’s likely to be very useful to a lot of people. And what I love about it is that it fits so well into the Fedora Way: it vastly improves the lives of both translators and upstream projects in a way that grows the entire pie for everyone. Connecting the hundreds of translators who are active in Fedora with the thousands of projects out there with a very low barrier to entry solution.

Great job, Dimitris!

This is really going to help our story around OLPC. Right now the OLPC XO, like many machines, takes a long time to start up. Far too long. Part of this is the machine’s performance and the fact that we’re still loading a lot of legacy stuff. (RAID? Do we need to check for RAID partitions in the OLPC startup scripts? I’m guessing no!) But when both David and I started looking at it a year ago I think that we both came to the conclusion that the way that we start the desktop it was fundamentally broken, especially on that little machine. That’s why we (and when I mean we I really mean David with me looking over his shoulder) pursued things like mayflower and looking at what architectural improvements we could make to startup. That service activation patch is part of the eventual solution.

The last piece of the puzzle from where I sit is udev. The current set of Fedora startup scripts along with the way that udev is set up takes a whopping 20 seconds or so to start up on the OLPC XO. Why? Well it’s not udev, exactly. Mayflower embeds udev and it starts up in under a second. The problem appears to be the rules, the programs that are run as a result of the rules, and the way that kernel events are handled.

Right now kernel events are handled by udev. Those events are matched against rules in udev’s config files. Those config files specify certain actions. Some of those actions have terrible performance-destroying side effects. Sometimes the rules specify creating device nodes. The device nodes that we create we have to set the console permissions on those nodes. That’s done by running a program after the device is created. In addition there is a global rule that runs modprobe for every event that comes in no matter if it creates a device node. That also requires running a program. On startup, the kernel sends out anywhere from several dozen to several hundred events that result in device creation. Imagine running a program for each of those items. No do that on a machine that runs at about 466mhz. 20 seconds, indeed.

I honestly believe that we can achieve a 20 second startup time for a desktop machine in Fedora if we take an agressive notion to what startup means and continues down this path to a sane architecture. I also believe that we can get the OLPC XO machine started up in that same amount of time, if for no other reason than the number of things we need to load on startup is so much lower. I think that David has a good sense of the architectural things that need to change in udev to make it scale better. He and Kay had a roadmap at one point but I don’t know what happened to it. But it’s a key to getting to that 20 second number. I think we can do it.