profiling startup time

One of our internal kernel hackers at Red Hat, Dave Jones, recently did some profiling of what’s called through the kernel during startup. He traced stat()s, open()s, and commands exec()ed. There were some pretty amazing results. Some of the highlights include:

• Executing /sbin/hotplug 317 times
• Over 470 exec() calls during startup
• cups, during startup stats 1212 times and opens 172 files
• cups fires up printconf-backend which does another 1304 stats and 155 opens
• hal does 1980 open calls and 7106 stat calls. Dave says “Wow.”
• Xorg during startup does 2420 stat calls and 370 open calls
• gdm calls stat 1871 times and 314 open calls. And it opens 126 fonts.

There’s a lot more than that, but those are the highlights.

I’m out in california right now visting the Mozilla folks and was joking about these pretty bad performance problems and Brendan overheard me. Paraphrased: “We made this mistake at SGI! Don’t repeat the past!” He pointed me at this awesome post from Tom Davis at SGI that was posted a long time ago in comp.risks. It’s worth the time to read.

The stories that are told in that post sound hauntingly familiar in our open source community, especially in the desktop community. Features over performance, getting ahead instead of getting things right. Not that I don’t think that we shouldn’t have priorities – we certainly should. But I think that sometimes performance and reliability have taken a back seat to other work when it probably shouldn’t.

Going back to my specific cases that Dave outlined at the top of this post, one of the most common complaints we hear in our desktop group from customers is about boot time.  We know that some huge percentage of boot time could be solved by reading in a bunch of common blocks all at once at the beginning of startup (either by using block reordering or some other method, but that’s not important here)  but I suspect that the problem isn’t so much that we have to speed up the I/O as it is we just have to be smarter about when we do need to do I/O.  I suspect that we could make a huge difference in our startup time if we took a two-pronged approach:

1. Clean up a lot of the low-hanging fruit listed above and
2. Make the I/O that we do have to do a lot faster.

These are really two different projects that need to get done.