Christopher Blizzard

I added support for del.icio.us to whoisi this evening. Let me know if you see problems.

Update: There are still server throttling issues so sometimes adding del.icio.us urls doesn’t work. Still working on making the polling stuff smarter about that.

Garrett LaSage put together a user theme for whoisi.com. He’s got a link to the screenshots on his blog. It works with either Stylish or Greasemonkey. So cool!

July is going to be a pretty intense travel month for me. Here’s a rundown of where I will be:

• July 6th-July 10th: Istanbul, Turkey. I’ll be there for GUADEC and I’m speaking on Wednesday. Hopefully to re-iterate how important the web and its users are, talk about some of the cool stuff coming down the pipe and how GNOME might be able to take advantage of that.
• July 15th: Boston, MA: I’ll be at this Web Innovators Group: Boston event. I have no idea what to expect from this. Maybe some innovation or something. On the web. Come and talk to me about Mozilla, Firefox or Whoisi if you want. I’ll be the one standing in the corner looking lonely.
• July 21st-July 26th: Portland, OR: I’ll be out in Portland for OSCON Open Mobile Exchange. I’ll be speaking on Monday about Mozilla + Mobile and the awesome work we’re doing. The wife and very cute baby will be with me on this trip and we intend to take most of the rest of the week off hanging out with friends and seeing that part of the country.
• July 26th-August 1st: Vancouver, BC: Firefox summit plus one day early in Vancouver to explore it. I’ve never been and I hear it’s a nice city.

If anyone wants to meet up to discuss things, let me know. I’ll be around!

net@night 59: Barbecue Anchor with Leo Laporte and Amber MacArthur

They cover lots of stuff - the iPhone plan in Canada and talk with someone who used to be at Yahoo!. But there’s also a nice comment from Leo about whoisi about 19 minutes in. He sounds pretty excited. I’ve never heard Leo before - he’s got a pretty awesome radio voice! (Originally found on twitter.)

It looks like identi.ca is finally public. I’ve been following Luis Villa on it for quite some time on whoisi. I added support to whoisi today to support rendering entries properly. Took about 30 minutes to write the code and test cases.

I’ve been slowly watching people add identi.ca accounts to whoisi so that they don’t have to end up looking at a bunch of different locations to follow people. Kind of neat to see the system work. Whoisi makes it easier to transition from twitter to identi.ca because you can follow people on both and also gives you a solid mapping of identity from one to the other. Very handy, indeed.

I haven’t explicitly added support for del.icio.us yet to whoisi, but that hasn’t stopped people from adding about 150 accounts to various entries. del.icio.us is one of the services out there that’s most agressive at rejecting people if they are polling incorrectly. And sadly, whoisi is doing it wrong at the moment.

They ask you not to poll any particular entry faster than once every 30 minutes and ask you not to connect to the service more than once per second. Whoisi doesn’t have per-host limits in its polling at the moment nor does it have a deterministic polling interval. (It chooses a random poll interval from 1 minute to 60 minutes to spread the load out especially during the initial polling service startup.)

Sadly, this affects adding new del.icio.us accounts as well. So if you’ve tried to add one and it fails, they might be rejecting access for a bit. I’ve seen this reported by a couple of people.

I’ll be writing a bunch of new code to handle per-host and per-RSS throttling and will add support for del.icio.us at the same time. (I need it for a bunch of other reasons, too. If nothing else because I don’t want to die when twitter dies.)

The first version of about:mobile is up on devnews. Enjoy!

(Did you sign up for the email version yet?)

I thought it might be a good idea to make a little post about the first few days of whoisi just to give people a sense of how things are going on the backend here.

First of all, holy crap. Frankly, I had no idea that people would be as interested as they are. I knew I had a kind of quirky and weird idea and that it would be compared to a lot of other services that are out there but I had no idea the kind of response that I would get. It’s still a very very small site by any standards that I’m used to (since I work on such a huge project my sense of scale is way off) but it’s been pretty awesome to see the reactions and the pickup in the techie-web-2.0-bubble-conversation-cloud and watching on summize has been fun. It’s clear that a lot of people are in it for the gold rush without really understanding it but there are a lot of people who are learning how to use the site and, bugs and performance aside, are having a really good experience with it.

I wrote down a couple of numbers just as I was publishing my announcement blog post. When I posted, there were about 473 people on the site, the vast majority of whom I added myself. As of this writing there are about 2253 people now listed on the site. I suspect that’s mostly people adding themselves, but not entirely. There are clearly people using the site to follow large numbers of people and adding their friends. Not a huge number of people, mind you, but quite a few. I should have some numbers to share at some point on that. It will be a laughably small number if you’re used to google or friendfeed or facebook but for a guy who wrote this on weekends and nights I’m pretty happy with it.

Want to have some fun? Just sit there on the site and click on the Random Person link. When I used to do this all I would see was my friends. Now I get all this great random content. So frigging cool. It’s my favorite whoisi feature at the moment. I recommend investing a half and hour and just enjoying whatever people are doing.

Some notable quotes from tech folks:

• Dave Winer really liked it and did a post on it including a great little twitter about it and a followup converation on FriendFeed. (After that post I was really glad that I spent much of last week doing perf work on the web site before announcing it otherwise it would have been crushed.)
• Leo Laporte really seemed to enjoy it and got the sense of it right out of the box.
• On Friday night at 5pm, just I had just stepped out of the house after doing some tuning on the box to try and keep it alive for a few more hours, TechCrunch did a post about it. I was nowhere near a computer. Being on TechCrunch is great, except that spike in traffic meant that every single person who probably tried it from that article would have had a “holy crap, why can’t I add myself and why is it so slow?” experience. (Since then I’ve done even more work and it’s back to acceptable for now. More on that later in the post.) It also means that I have an entry in CrunchBase which is pretty neat in that silly self-serving ego kind of way. It’s neat to have an article up there, though.
• I really liked this post from Christopher Clay because I think he really understood the model and what it might mean down the road. The potential is there for all those grandiose things he mentions, I guess, but mostly I’m focused on trying to make good and fun experiences for people. Anything else is just a side effect of that. (For the record, I’m not happy where whoisi is on that road yet, tons of stuff left to do, but it’s on the right track.)

So what has changed in these last few days?

Since it’s a weekend and night project for me I did a lot of stuff this weekend, but I wanted to call out a few people who really helped me out:

• Mrinal Wadhwa, who basically pointed out what was causing the most serious bug on the site. If you were having problems adding yourself and getting a response saying that your url belonged to someone else you never even heard of, this is the guy who figured out what was causing it.
Shane Bishop who pointed out a problem from the login later/follow page that would cause a random 500. I think that a lot of people saw this, sadly. But he gave me a really good bug report to work from.
• Shawn Lauriat, who found an XSS problem on the site, which I fixed, and then he found another, and I fixed that too. Then I went through every single template on the site with a fine tooth comb and then again and then went through them again. So I’m pretty confident at this point. But he is the guy who pointed them out, which is great, and continues to test. Go Shawn!
• Joe Shaw, the lovely number 6, let me hang out on his couch Saturday afternoon and work on performance issues.
• And my wife, who has to date not cut me with a knife in my face area given all the time that I spent on it this weekend when I should have been hanging out with her.

This isn’t the oscars. What changed?

I actually got a lot of stuff done this weekend, but not a huge amount of it is user-facing (other than the fact that it should actually be able to keep up for at least a while.)

When adding new people you get really weird results of other people who supposedly own that url. This problem was two fold. First, whoisi tries to figure out if there’s a duplicate person on the site already based on the urls that it sees. A person-id is a scarce resource and once you allocate that number it’s hard to change to something else so getting that right up front is important. Given that, one of the things that it does is look at the urls you pass in a look for matches. It also took the preview feed that it downloads and looks at the links in that to see if there are duplicate matches vs. other people. This actually worked really really well until people started adding del.ico.us bookmarks and google reader feeds to the site which include links to other people’s sites, making it look like ownership. So needless to say, I ripped that out at about midnight tonight. Clearly, I didn’t think that one through. Once again, thanks for the excellent reports that helped track this down.

Backend performance enhancements. Right now I’m basically completely bound to CPU performance based on how fast I can poll sites. The algorithm, as I described in a previous post, is pretty braindead and a lot of it still is. But at least it’s faster braindead. That’s what I spent most of the weekend on: buying myself some headroom for the future. We’ll see how long that lasts.

Fixed the XSS problem that was found. As I mentioned above, I spent a lot of time auditing code and I think that I found the problem areas. If someone else finds something, let me know.

Fixing broken HTML that was on the site. There were some missing closing div tags on some of the objects on the site. Frankly I’m amazed that things rendered at all, but those are fixed now. Also added a background color for the page for those crazy people who change that setting.

Added an error message so that if you go to the follow page and the people you’re following don’t have anything posted it lets you know what’s going on. This is that 500 that a lot of people were seeing. Lots of people added one person and went to the follow page. Bam, 500. Whoisi doesn’t show you stuff that is in a person’s timeline when they are first added. (This would cause an explosion on the everyone page, for example.) So you have to wait for them to start adding things before things start showing up on the timeline. So I added a page that explains that.

If you’re into mobile stuff and you want to know what’s going on with the Mozilla Project on the mobile front, take the time to subscribe to our mobile newsletter called ‘about:mobile’. (or follow it on The Mozilla Developer News Weblog. But, really, you should get on the mailing list because it’s cooler that way.)

We intend for it to be monthly, but might post more often than that if we have interesting things to report. So sign up! Only takes a couple of clicks.

Update: I just added twitter.com to /etc/hosts and pointed it at a site that doesn’t have a webserver. Works for now until twitter comes back.

Having some lunch and I thought it might be worth a small post while my burrito cools.

I just had to disable polling on whoisi because twitter is down. Again. Whiosi’s polling system, in case you were wondering is basically as dumb as a wooden post right now. I’m not trying to pretend that I thought it would work forever, nor that it was very good. But it works well as long as the internet is pretty healthy and the number of failures is evenly spread out among sites on the web.

Here’s how the poll system works right now for each site in the database: refresh every site every n minutes where n is a random number between 1 and 30. That’s it. And it does that for everything. No backoff, no per-site limits, etc. It’s easy to plug that kind of thing into the code, but it’s yet another thing on the “not yet done list.” Designed to be smart, but without the brains behind it.

You also have to understand how jobs are run. Jobs come from two sources, the “master service” (which I’ll describe in a later post) and the web site. But they all run through the same job queue. So when you try and add a new person it tries to go out and make a little preview of the site. That job has to compete with site refreshes that are also underway. The limit on the number of jobs that can be run at the same time is also dumb. Right now it’s 50 at once. Not 3/sec or 50 waiting for I/O, just 50 in progress.

So when you have a few hundred twitter accounts you’re polling and they fail by having to time out, the queue gets backed up. Given how many people are adding accounts right now I thought it would be good if the site interacted well instead of having things refresh instantly. It’s a tough choice but it’s how it is until twitter recovers from whatever its latest pain is.

I wish that twitter would fail by giving an immediate 500 or even a connection refused. The slow death of waiting for a response is basically the worst possible thing that can happen. Fail faster. Please.

Not that I should throw stones for even a second, given how dumb my code is. But just a lesson and what happens when a (dare I say important?) service dies.