August 2005

Thanks to everybody that got the new Drupal gear going: Kjartan, Corey Shields, Mike Marineau, Matt Rae and several other folks I’m sure I’m missing here. It’s looking much better and the best is yet to come.

I don’t have any data on the old server other than the load would sometimes reach in excess of 20+. Sometime after that it would melt down …

We’re about half-way through the deployment of the new gear. We have one web head and the Sun v20Z installed as the database server. We’ll be adding another web head and an administration/staging server. This will handle things like SVN, staging and a raft of other services that we don’t want to put on the production website.

Looking at the Dell order status page as of this writing (4:10 PST 8/26/2005) I see the new gear shipping as of 9/7/2005 which by Dell standards usually means it will be here on that date. We’ll get it racked, built out and integrated into the existing infrastructure. We should experience no downtime because of that.

Now I can’t really quantify the “speed” of the equipment but I can give you a few numbers and talk about the current load on the machines.

The main web head is currently operating at a modest load:

16:20:45 up 13 days, 23:53, 2 users, load average: 0.16, 0.38, 0.45

It’s done about 170,000 requests in the first 5 hours of operation. Oddly enough looking over the first few hours of logs there are some particularly bad hosts hammering the forums. More investigation will be needed as the user agent is “Mozilla/4.0 (compatible; Google Desktop)”. Looks like a broken/buggy client.

On the database server things are good. We’re also running at a modest load here:

16:23:03 up 13 days, 23:49, 2 users, load average: 0.31, 0.18, 0.19

The load is deceiving though. Looking at the MySQL ’status’ we’re humming along 250-350 queries a second over the course of the course of the day. You can see the CPU usage and load graphs below (I’m sure these will get more interesting over time):

The load on the site should drop and the responsiveness should only increase when we add the new web head into the mix. We have a student intern that is now working on dialing up all of the things that the Drupal team is asking them to do (the devil is always in the details).

Hope this answers some folks questions … I know its not as thorough as some would like but at least it gives people and idea of what’s going on. We’ll be sure to follow-up with more in-depth statistics, etc over time.

Thanks everybody for your patience while we sorted this out!

Social networking … this isn’t about foocamp v. barcamp … its much bigger than that. Tools are going to shape the future of these networks.

There is something about BarCamp that I can’t quite put my finger on. Its nothing to do with the FooCamp v. BarCamp nonsense … that’s all hogwash to me. What is fascinating is the way this community connected, organized and happened in such a short time. O’Reilly’s FooCamp may have been the inspiration but going over the weekend’s events at BarCamp I see that its just an entirely different beast.

Technology for the sake of technology has always kind of bugged me. Mainly because I’ve been in that business for so long. I’m a dork and I like doing dork things. Building or enabling new technologies that help you build or enable even newer technologies never really rang true for me. Its like building a pyramid that really never goes anywhere or does anything. But looking at how BarCamp was organized and how it happened makes me see social technology in action. Using the tools to organize socially quickly and easily. Again, BarCamp was a bunch of dorks connecting to talk about dork things … I’ll concur with that … however, the way that they organized is the fascinating part of all of this.

In looking through all of the flickr photos for both FooCamp and BarCamp its obvious a good time was had by all … I think this is just the start of something new and exciting … these technologies that we surround ourselves with everyday are just starting to enable some new and insanely great things. I can’t wait to see them as pervasive as the microwave.

It has taken us a few weeks to get things sorted with the Drupal donations but as of Friday 8/19/2005 we are officially on track.

Dries and I have been working for a few weeks on getting the generous donations of the Drupal community moved over to the OSL so we could execute on the plans we laid out earlier. Unfortunately international money transfers aren’t as easy as international file transfers. That said, we got it figured out this week and have started in earnest on finalizing things.

We have ordered the Dell gear that we originally outlined. We’ll end up with 3 new machines (2 web heads and an admin/staging server) in addition to the Sun v20 that was donated to us. That gear should be here in a few weeks.

In the mean time, we have deployed one Dell 1850 + the Sun v20 and are preparing to move the Drupal site over. These machines have been configured/tuned to handle the entire Drupal site for the time being (we had a spare 1850 around and will replenish our backup supply with one from the new gear that is landing). Once that other gear lands, we can start working on things like SVN/CVS and other tools to help the Drupal development.

We have also hired another intern here at the OSL that will be dedicated to Drupal for the next 2 months. This intern will work with our full-time people to help the Drupal community execute on the transition to the new equipment as well as bring up new services to grow the community even further.

The OSL is really excited about this opportunity and we can’t wait to really, really finalize the equipment and infrastructure so Drupal can best focus on its “main thing” … building an amazing CMS …

It’s pretty amazing really. My son, now at the ripe old age of 14 months (as an aside, how long do I have to keep referring to his age in months?!), has this inate ability to wake up every single morning right at 6am. Now this isn’t give or take a few minutes. With eerie atomic-clock-like-accuracy this kid starts stirring when my nightstand clock blinks “6:00″ at me. So I’ve got that going for me. Which is nice.

Wow. If I was an actual user of eBay I might have been fooled by this. I was sent a note by “eBay” saying that my account had been locked out. The email was HTML and said I should respond immediately. They even tease you a bit with “we’ll never ask you for your email, account information, etc.”:

Now, digging a little deeper into the actual HTML we find this where the “Repond Now” button is I find:


<a href=\”http://anytimeforums.org/ebayisapidllsignin.html\”><img alt=\”Respond Now\” src=\”http://pics.ebaystatic.com/aw/pics/VIQnA/respondNowButton_117×21.gif\” border=0/></a>


So they are actually using images off of eBay’s servers. Now, how about this domain “anytimeforums.org” … let’s check it out:


kveton@ack:~$ whois anytimeforums.org
Domain ID:D107060364-LROR
Domain Name:ANYTIMEFORUMS.ORG
Created On:01-Aug-2005 17:30:48 UTC
Expiration Date:01-Aug-2006 17:30:48 UTC
Sponsoring Registrar:eNom, Inc. (R39-LROR)
Status:TRANSFER PROHIBITED
Registrant ID:AC900BDA71766AFE
Registrant Name:Krystle Van Dyke
Registrant Organization:
Registrant Street1:541 Roxanne Dr
Registrant Street2:
Registrant Street3:
Registrant City:Antioch
Registrant State/Province:Tn
Registrant Postal Code:37013
Registrant Country:US
Registrant Phone:+1.6157819794
Registrant Phone Ext.:
Registrant FAX:
Registrant FAX Ext.:
Registrant Email:PHILIPNEOVO@AOL.COM
…


Wow. Looks like it was registered just over a day ago and then the mail starts showing up. Interesting. Alright, how about this little domain, let’s take a look at that:

Which you can look at if its still up at here (if its even still up when you read this).

My guess is the “_private” directory is where things are shoved (the input user names and passwords) which can then be yoinked if you authenticate. I wonder if they even have a nice CGI in there? Who knows. Back to the task at hand.

Now, if you click on the link (or just navigate from the directory listing there) to the ebayisapidllsignin.html file you get this little gem:

All of this is pretty clever and I know that some people are going to get caught by this. I can’t imagine this is the first time this has ever been done but this just seemed very clever. In any case, I have forwarded the note I received onto the abuse group at eBay.

Yesterday at 4pm PST (aka 00:00 GMT) the “Hour of Doom” hit the Mozilla infrastructure as it does at the start of every month. However, this time we were ready.

I worked with Dave this past week to get the application servers for our PHP services behind the LVS cluster here at the OSL. With OSCON and Linux World Expo coming up I knew we would not have many cycles to get it done until after. Getting this built out now would allow us to brace ourselves in part for the deluge of requests to addons.mozilla.org as well as to prepare for what has been deemed the “hour of doom”.

The best way to describe the “hour of doom” is like having to pay down a bill every month on something silly you did while drunk months and months ago (note: the author has never done this, but he had a friend once that did …). The silly thing we did in this case was a bug in Firefox that would cause older clients (it has been fixed now) to continually phone home for the first 7 days of the month starting right on the nose at 00:00 GMT. We discovered this bug in January (somebody correct me if I’m wrong there) and have been having to deal with it ever since.

In the past few months, Dave, Polvi and Corey have been bustin’ their humps getting the new LVS clusters (yes, its plural) up to handle these bursts in traffic as well as have a roadmap for growth over the next few years for the MoFo. Polvi made some great configuration scripts for managing the clusters and Dave has been a maniac with the Red Hat Network configurations (its pretty darn slick now — we can change the role of any server behind the cluster in a matter of minutes). Corey has been a trooper fixing boxes here that were broken in shipping as well as moving earth to help get the cluster into place in a timely fashion.

So yesterday at 00:00 GMT it hit and although there were a few hiccups, the monster held together through an amazing deluge of traffic (upwards of 100 Mbit of just AUS traffic). Dave and Polvi did some final tweaking that brought the load down considerably and should have us ready for next month … here’s to smooth sailin’ and butt kicker SA’s … Good job guys …

Why on earth didn’t I think of this?! Open source beer?! Genius. Not only that, the beer recipe is released under the Creative Commons license.

I got a kick out of that little kid shouting “Wow, it’s a Firefox!” at my t-shirt yesterday … so did Ted Leung who was also there too. Friggin’ hilarious. And yes, Portland is to open source as Mecca is to Islam …

Day 2 of the Summit was even more engaging then the first (even though I had to cut out a little early to tend to my garden/house before the big show). We were lucky to have a few lawyers on hand and got a great (albeit brief) tutorial on trademarks and copyrights. Excellent stuff (although at the end of the first half of the day I found myself wanting to curl up into the fetal position and suck my thumb for fear of litigation).

We had lunch out at Brasserie Montmartre which was great (anything with crab in it can’t be all that bad) and the discussion there was lively as well.

Thanks goes out to Allison Randal for setting up the FLOSS Summit. Just the opportunity to connect with all of these people was fantastic and I can’t wait for OSCON to start! See you all there!!