Pretty neat to see someone ported over flashblock to run on MicroB on the N810. I haven’t looked at the code to see how they do it, but I’ll bet it was a rough ride since that’s written in C and C++, not in XUL. (I also assume that Antonio did this work?)

As we continue to bring up Fennec it will be interesting to see how fast we can start migrating add-ons over to support Fennec. It will be a different experience, but many of the same pieces will be there so having support for Firefox for the desktop and Firefox on your mobile device will likely be possible. (Mark, correct me if I’m wrong.)

He mentions that he’s working with us at Mozilla – very true. And good fodder for another post on another day.

He also touches on an important topic for me, which is the difference between between being “open” and being “good.” Being open is easy. Being good is not. I’ll write more about that at a later date as well.

Atul Varma talks about how Weave works and some of the wider goals for the project.

Weave is a very interesting part of the Mozilla project. The underlying mechanisms are very simple (just a webdav store and a private and public key system, basically.) But the fact that we’ll be able to use it to share data between users on their terms and extension authors should be able to take advantage of the storage that’s there – well – it could be huge. Lots of great potential here.

He also touches on an important part of weave. That we want user data to be private and we want users to control what they are exposing to the world. Mozilla’s servers can’t read the data that you post and neither can anyone else. That’s an important piece of what Weave represents.

Google drops the Google from Gears and will be adding Firefox 3 support.

It’s clear that Google is trying to build a larger community around the Gears plugin and is willing to give up branding in order to try and make that a reality. Also good that they are porting to Firefox 3 given that a huge chunk of the world is about to update to it.

Jorge Castro talks about his views on Mozilla’s commitment and investment in Linux.

Jorge is a pretty fantastic guy and I’m happy that he’s willing to jump out there and say something about the hard work that our community put in for this release of Firefox on Linux support. Great native integration, memory and performance improvements, a great experience for users, etc. Good to see him focus on the big picture which is how much better Firefox 3 will be vs. Firefox 2.

Anyway, that’s it for Friday. Thanks for tuning in!

There are some patches required to some of the GNOME stack to get things up and running. The patches are up on the page.

Good work, guys!

I go out to the store this morning and buy one, bring it home, plug it in and I decide to try and figure out what kinds of loads various things sitting on my desk create. I have a little Verizon router for FIOS – it uses about 5 watts. The external USB hard drive about 8 watts. The mac mini that I have running Fedora 8 on it uses about 7 watts. Not too bad, all things considered.

So I decided to drop the router and the drive and find out how much the mini drew when it was running and how much it drew when it was suspended. So I unplugged them and then suspended the mini. The power went from 7 watts to 10. Huh? I started the mini back up and the power usage dropped back to 7. Suspend, it jumps back up to 10. That’s right. My mini running Fedora 8 uses more power when it’s suspended than when it’s sitting at an idle desktop.

I have a new rule. Richard Hughes isn’t allowed to complain about anything Firefox-related until my computers running Linux use less power suspended than when they are sitting at an idle desktop.

I’ve always seen this position as both good and bad. It’s good for the distribution end users since they don’t have to go and find a build of Firefox. And it’s good for Mozilla because it means that we don’t have to produce builds for N Linux distributions, which is basically an impossible task. It also means that distributions can make late-breaking fixes that are specific to their distribution that really affect their particular user base.

But it’s bad, too. It means we’re disconnected from users on those various Linux distributions. We’re at the mercy of the distributions to make updates in a timely manner, and very often we find ourselves chasing them to make updates that they clearly should be doing. For those users where we have a direct relationship we have a pretty good track record of making timely updates to almost our entire user base. Linux users are cut off – quite intentionally because that is the classic “value add” for a Linux distributor – from our update train, sometimes leaving them vulnerable for weeks or months. (Note that this does not generally affect the top-tier Linux distributors like Canonical, Red Hat and Novell. They are actually excellent at delivering updates because they have dedicated engineers who only have the job to chase Mozilla and be ready when we’re ready to deliver an update – either a firedrill or planned update release.)

There’s another large downside as well. Distributions often make changes to how Firefox is built – be it compiler optimization changes, linking with system libraries instead of the ones that we ship with, adding their own large patches to add support for some random feature or making changes to the default look and feel of the browser.

Contrary to uninformed hyperbole Mozilla actually does a huge amount of testing on Linux. It’s one of our three top tier platforms and we run it through all the same regression and performance testing that the other platforms get. You can see this by looking how much attention we pay to how to tune the compiler to give us the best performance (hint: more -O doesn’t always make us faster, nor does architecture-specific tuning!) or how much time we’ve spent on the reported fsync issues that have affected quite a few people.

Because of the amount of work that we do on Linux and how closely we work with upstream projects (sqlite, cairo, etc) we’re still the experts on what works and what doesn’t. And because we have a pretty full set of tests that we go through we know what versions of upstream projects work well inside of Mozilla. Note that this doesn’t mean we know which versions don’t work with Mozilla, as I will illustrate later. We can’t be compatible with every single version of every upstream project with every single possible configuration, it just doesn’t work.

I’ll use a specific case in point here to illustrate what I’m talking about with Fedora + Red Hat. (Note that I’m pointing this out because it’s a real situation, not that I think that the Fedora + Red Hat guys are doing a bad job – they actually do a fantastic job given the task as far as I’m concerned. The issue I’m about to describe does not affect Fedora 8 or Fedora 9 users – only those who happen to be using Fedora Rawide – the bleeding edge of the bleeding edge.)

Chris Aillon and Matthias Clasen were reporting an issue to me where Firefox was hanging for long periods of time in Fedora Rawhide while opening the history tab. I figured that it was the same old fsync-related problem but they were reporting that it was happening for long periods of time (30 seconds in a lot of cases) and it was happening on systems that were relatively unloaded in terms of IO. I was near the Red Hat office on a personal errand and I thought I would stop by and try to help diagnose the issue. Looking at the issue in a debugger I found that it was hanging down in sqlite and not returning into Mozilla-specific code at all. I also noticed that they were linked against the system sqlite instead of the one that we ship with. I asked Matthias to try a Mozilla-built Firefox on his machine with his profile and it did not have any problems. When Chris generated a build for Fedora Rawhide that used our internal sqlite version it also didn’t have any problems.

It turns out that the sqlite version that’s included in Rawhide, version 3.5.8, has a bug in a particular type of query that Mozilla uses extensively. When Mozilla updated to that version of sqlite our automated testing picked up on the problem and the change was backed out of our tree. Let’s look at the order of operations that caused this particular issue.

1. Mozilla checks in a patch to upgrade to sqlite version 3.5.8.
2. Fedora Rawhide notices the new requirement in configure and bumps their system sqlite version to 3.5.8.
3. Mozilla’s automated testing picks up failures as a result of the new sqlite version and backs out the changes.
4. That backout is missed by the Fedora folks and they are left linking against an sqlite version that contains the problem.

This isn’t the worst example of what goes on when distributors are making changes to upstream software. The impact here was pretty minor – only a few users were affected and the bug was pretty obvious to a large number of people. It does get worse. Ask Debian and Ubuntu users how happy they are about regenerating keys in light of the OpenSSL issues that were recently found with the downstream patch. (I realize that’s an oversimplification of that particular issue but it has a lot of the same qualities as this example.)

This is a real problem, one that we’ve even successfully predicted.

So how does this relate to the fsync issue? Well it shows the opposite end of the of the patch spectrum. Basically every single Linux distribution is waiting for a good fix to that particular problem. And they will all ship a fix to their users. So sometimes distro-specific patching is a good thing.

The trick has to be finding the balance. Right now we know that there are a lot of instances where bad or ignorant decisions are being made. (Just because an option exists in ./configure doesn’t mean that you should use it!) People clearly aren’t taking advantage of Mozilla’s automated testing facilities – in the Fedora example the problem would have shown up pretty quickly if they were running the same tests Mozilla does. And the flows of information between Mozilla and the various distributions is ad-hoc at best. Very often more effort is spent debugging the blame instead of debugging the actual problem at hand. I’ve been on the receiving end of that recently and it’s certainly soul-crushing.

There’s also no easy answer to the multiple-library-version problem, either. Once again, we’re not going to be compatible with everything everywhere, especially on Linux where the platform is more like quicksand than green grass. Just screaming “you should always link against system libraries!!!!” isn’t going to work when the size and complexity of Linux continues to expand without any contraction of the complexity involved in which Linux you should target. That with blind version updating means that we’re just going to be stuck with multiple versions of libraries – assuming you want a quality product that works as well as it does on Windows and the Mac and does so consistently.

So if I had to wrap up this post with some lessons learned I would put them down as this:

1. Don’t change a default configure option unless there’s a very good and very specific reason for it. No, really, don’t touch the optimization flags – we’ve probably tuned them to an inch of the compiler’s life. But if you actually run tests and find something faster, let us know.
2. Don’t ship a patch unless it’s been vetted by upstream.
3. Ship known good patches if it will help your users. But don’t do it without talking to us first. Remember, it’s possible – and likely – that we’ve known about your issue for quite a while. And while you’re at it if you had to fix something consider adding a test to our test suite so it won’t happen again?
4. If you do want to carry some heavy patches consider following the pattern that the enterprise distributions are using: they work together (at least Ubuntu, Novell and Red Hat) to carry patches for some pretty old Firefox releases. And have been relatively successful doing it.
5. And finally, remember that there’s a fine line between adding real value and making a change for change’s sake. The former is always encouraged (within limits!) but the latter often causes more trouble than it is worth.

In closing I think I would just like to say that there’s a lot of work to be done here. And it’s something that will need constant adjustment – there’s no one set of rules that we can develop and expect them never to change. Both Linux distributors and Mozilla can do better than they have done to date in terms of making things better for users on Linux. Thinking about changes, improving communication, understanding the reason for changes or diversion from upstream, etc.

And all of that discussion has to come from the standpoint of making sure that the user’s experience is improved. If it doesn’t improve their experience in a very tangible way, it’s probably not worth doing.

On Friday a pile of people will all be in San Francisco who care about embedding Mozilla. Turns out that me, Behdad, Dave Camp, Vlad and Christian will all be around in the same city at the same time and we’re going to spend Friday afternoon looking at embedding apis. I hope that out of this we’ll at least have some ideas about what’s already out there in terms of interfaces and functionality and we’ll come up with a laundry list of possible resources. The goal here will be to just start getting ideas and resources together, not to make decisions on what we’re doing. The community that we need to build needs to be much larger than just this subset of people and that will be reflected in our strategy for determining what these APIs look like.

If you want to join us on Friday afternoon, drop me a line and let me know. We’ll be hanging out at the Mozilla headquarters in Mountain View, discussing first steps and looking at existing APIs. I’ll write something up and send it out once we’re done to the .embedding and .mobile lists.

Above is a video that I took of Chris Double’s awesome whatwg video work running on my N810. Note that I didn’t have to do anything special to get this to work. I took the two patches, applied them to my trunk mozilla build and ran them through the ARM toolchain. The code works out of the box with the gstreamer backend that’s included on the N810.

I thought that it was important to show that we’re not just serious about adding video support to Mozilla, but that it can extend to a wide range of devices. If we want to have a world that includes rich internet applications, we have to be sure that includes phones and lower end devices as well.

The video itself is a speech that Cory Doctorow gave at the 2006 Red Hat Summit about copyright and intellectual property. I thought it was fitting.

If you’re a crazy maemo hacker you can get a copy of the build from my people page that includes gstreamer support. Follow Brad’s instructions to get a browser front end once you’re installed xulrunner somewhere. If you want to play a video you should make sure that you’ve encoded it for the N810 using a decent script to take advantage of the hardware accel on the device. Hack up an html page, embed the video like on Chris’ test page and try it out.