Scott Kveton

I have truly enjoyed watching Apple grow over the years. As a matter of fact, my first computer was an Apple LC II (16Mhz 68030 processor with 4Mb of RAM and a 80Mb hard drive) and so I’ve always had a special place in my heart for them. I have watched the iMac, the iPod, iLife and others as Apple has grown from a hardware/software company into a marketing machine. You don’t buy a Mac because of the superior technology; you buy it because of its coolness factor.

Don’t get me wrong, I own a 12″ PowerBook and you’ll hear me more often than not say “use the best tool for the job”. To me, Apple laptops are hands down the best laptops going. I bought my 12″ PowerBook for no more than the equivalent Dell and everybody that sees it oohs and ahhs. You can’t put a price on that. The big thing for me is that it does the job; email, ssh and presentations when I’m on the road. Best-tool-for-the-job.

Next up, check out that bitchin’ iPod. Another investment I have made recently is in the iPod mini … how frickin’ cool is this thing?! My wife even loves it and she hates technology … doesn’t get it. My father could even work this thing and more importantly read the screen. That is huge. The coolness factor and the hipness of the iPod design is something that sets it apart from other “devices” … at the end of the day, the iPod is a way of living, everything else is just a gadget.

To me, Apple is like Nike or Volkswagon … they are boutique marketing companies that farm out their manufacturing (well mostly anyways) to foreign factories, etc and are able to focus on design. Heck, even AMD does it; they rent/lease time at fabs now instead of owning/building their own. Genius IMHO. Take Volkswagon; what is the new Bug anyways? Its a frickin’ VW Gulf for crissakes … now put the two next to each other and tell me what has more of a coolness factor? What do I think they will do next? Look out for the next VW Scirocco … and it will be the frickin’ coolest machine on the block. But I digress.

So where will Apple head next? I see something quite obvious for them to move into and its the perfect blend of their strengths of marketing, product design and integration with all of their computing products; why don’t they go into the DVR/PVR market?

Think about it. Everybody in the world would love to own an Apple DVR/PVR … iTV if you will … it would be a sexy design, would integrate with technologies like Apple’s Airport Express and iPod and it could do pictures (a la the color iPod), play music, record your favorite shows and integrate with your other Apple hardware as seamlessly as saying “rendevous”.

I’ll be the first to step up to the plate and slap down my $699 for the iTV. Bring it on.

Thank goodness for Slashdot for providing me with so much blog fodder over the past few days … what would we do without Cmdrtaco?!

I am a PHP developer and have been for quite some time. I’m no frickin’ brainiac but I do okay. I am also an avid user of Apache and more importantly Apache 2. We have been using Apache 2 here with the worker mpm + nptl threads + 2.6 kernel on ftp.osuosl.org for quite sometime with amazing success. I actually can’t justify more hardware because switching to that from Apache 1.3 + 2.4 kernel made such a huge difference. We can handle 1000 req/s at 250Mbit/s without ever getting the load on the machine above 1.0.

We also use Apache 1.3 here for hosting our PHP applications and like most folks we cannot justify moving to Apache 2 on those machines … it just doesn’t make sense for us.

Now I feel for both the Apache and PHP camps. Its a tough debacle to be in and one that is obviously generating some tension between developers. So how do you get folks to make the switch? My answer: features.

Apache 1.3 is great, but it just doesn’t scale like Apache 2 can. Now, I know with Apache 2 + PHP you still have to use prefork and this essentially makes it like Apache 1.3 in terms of scalability. However, some of the new features and updates to old ones coming out in Apache 2.1 I believe will make users make the switch simply for ease of use and ability to keep your environment sane.

Last week we had the chance to interview Paul Querna for a job here at the OSL. Part of the interview was to do a presentation on developing the next-generation infrastructure for the Mozilla Foundation’s applications (such as update|download|developer.mozilla.org). Paul’s presentation was fantastic and shed light on some of the features that I believe will bring Apache 2 into the lime light: mod_proxy, mod_cache and mod_proxy_balancer.

Now I know what you’re saying: “mod_proxy and mod_cache have been around for years … heck they were in Apache 1.3!” Well, I drilled down a little by grabbing the latest alpha via SVN from svn.apache.org and also talked with some of the developers of those modules. mod_proxy and mod_cache have been updated quite a bit and developers are expecting about a 20% performance improvement over the versions in 1.3 and even 2.0. This is great for long-time users of the best-single-threaded-app-ever: Squid. With mod_proxy and the mod_cache memory and disk modules now, you can essentially replace Squid plus get the added benefits of a fully-threaded application that can scale plus with added support for all of the HTTP 1.1 calls (some of which even Squid still has problems with). Not only that, you can leverage all of the other modules in Apache like mod_rewrite, etc that allow you infinite flexibility in managing your site(s).

mod_proxy_balancer is really interesting. Now, tie in mod_proxy and mod_cache … you have a great caching reverse proxy solution. Now, if you can leverage your static content through this plus tie that back to your Apache 2 + PHP in prefork on the backend you can get a lot more mileage out of your hardware. Your caching servers would essentially do most of the heavy lifting and then would hand back queries to the application servers that would than pass them through the caching servers. Sweet ass sweet ™!!!!

The next question is to see how all of this works in practice … I shall be the guinea pig of course … Big thanks goes out to Paul Querna for his excellent presentation and getting us to think about this stuff.

If my initial tests show anything, I can forsee the combination of the above modules bringing Apache 2 together with PHP once and for all …

I just read an interesting blog post by Peter Torr from Microsoft. The posting brings up some questions about the security of downloading Firefox from Mozilla’s download redirector as well as the need for digitally signing releases of software. I’d like to make a rebuttal to some of Peter’s points below.

I helped author the download redirector that Mozilla uses to distribute its software. We originally had 10 mirrors in the main ftp.mozilla.org rotation. This was all well and good when Firefox and the other Mozilla software products were not as popular. With the 1.0 release of Firefox our mirror network hit a wall and was about to meltdown until we were able to install the download redirector. The redirector works as follows: the user clicks on “Get Firefox” on the Mozilla website which redirects them to a download.mozilla.org URL with information like OS, language and product that they want to download. The redirector then queries its database of valid mirrors (more on that in a second) and then redirects the user via a HTTP 302 response. The entire response is a little over 100 bytes and allows us to now leverage 50+ mirrors now instead of just 10. In addition to that, we can weight the mirrors based on bandwidth available. This allows us to send more traffic to the mirrors that can handle it and less to those that have smaller pipes.

Every 15 minutes the download redirector queries each mirror and makes sure that the latest release of each product is on that mirror. If it is not, then it removes it from the queue and tags it as disabled. Bear in mind, this is a v1.0 application. What is v2.0 going to bring? We will actually download the files from each mirror and check their md5 hash. I’ll be the first to admit that leveraging md5 isn’t ideal but until there is an objective way to sign applications, I don’t see another option.

Now, in a perfect world I would love to be able to have all of this come off of one box but at the time of the Firefox 1.0 release we were pushing close to 4.4GBit/s across all of the mirrors (that requires a little math and extrapolating from the redirector statistics). That said, I believe the Mozilla Foundation is accepting donations.

Finally, Peter makes a note about how we must make it so users only trust software digitally signed by the vendor/developer. He even goes so far as to talk about getting a VeriSign code signing certificate for doing so. Now, I don’t think anybody has a problem with signing software; its a great idea. The big problem today, is having to buy that trust from less than savory vendors. This is a great opportunity for the likes of free, open solution to digital signatures and certificates.

It’s obvious to me that the honeymoon is over for the Mozilla Foundation and I’m excited to see honest dialog from the likes of Peter Torr … what doesn’t kill us can only make us stronger.

I always thought that if I were a CIO and had to make a decision about open source it would be a hard one. Afterall, you have to take so many things into consideration. What is your “main thing”? What strengths does your organization possess? What does your “product” run on? These are all very important questions for somebody in charge of the technical direction of an organization.

Most people know me as an advocate of open source; a guy that will always do the right thing at the end of the day. I’ll not try to hide it; I believe open source solutions are good ones in most cases. So its hard for to imagine being in the shoes of a CIO and having to make that sort of decision. Let’s be honest; in 20 years from now, you may be in a position of a CIO but your skill set will be entirely different. What would set you apart from others in seeing open source as a valid choice and then being able to execute on it?

Well, I believe I have a couple of opinions on that matter …

1. Evaluate your options: Look around you. What are you competitors doing? What are your peers doing? This sounds silly but a lot of people don’t do this. In the very rare case you will in fact be blazing some sort of trail with the decisions you make but it won’t happen on accident. If you’re not taking time out to evaluate your options and think about your overall strategy then you aren’t doing your job.
2. Choose people: Your people are your biggest asset. Again, sounds simple but most folks miss it. Where are your people’s strengths? Who are your shining stars? What path do they want to take? Where are your shining stars headed? This can sometimes be an indication of their decision making skills … if they are on the path to professional growth and not just in a holding pattern they may be on to something that you can tap. Look for the people in your organization that analyze the problems at hand. People who make snap judgements without looking at all possibilities are the same people that will end up getting you locked into some vendor/product that will cost you in the long run. The people who take a moment to examine the facts are a huge asset and totally infectious in an organization that promotes transparancy.
3. Know tactical loss vs. strategic win: I was talking with some folks on the Indiana University staff and we were discussing their choice to totally scrap their working course management software in favor of re-building it with a J2EE-based tool chain instead of sticking the its current development track which was Microsoft ASP/.NET. This is a huge decision to make. 95,000 students, faculty and staff that were quite happy with the solution they already had and they decided to scrap it in favor of building it again. This is called a short term tactical loss leading to a strategic win. That was over 2 years ago and the migration is complete. IU is now better leveraging their strengths in J2EE for the CMS development as well as being able to tie that software into their other J2EE applications that drive ERP and FIS. So many institutions think on the short-term when we will actually be here for the long-term and we might as well just do it right the first time.
4. Know your strengths: Where do the strengths of your company lie? I met with the CIO of a small medical software company a few months ago. By small I mean less than $1 billion a year in revenue. He had a hard path to take. As the new CIO at this company he had to make a decision; what development platform should his company choose for their systems. The expertise of his company was Windows, pure and simple. What was his decision? Go with .NET as a platform for the future.
5. Look at the facts, ignore the pundits: How Laura DiDio still has a job is completely beyond me. She is the Yankee Group’s “unbiased” technologist at large often pontificating on why TCO of Windows is way lower than any other solution. Funny that Microsoft are the ones that are paying the tab on these “reports” they put out. Anyhoo … how do you sift through fact and biased opinion? Even your people will do this … the fact is people don’t like the unknown and it forces them to push what they do know even if the technology or path is flawed. One thing you should always keep in your cadre of CIO tools are some technologists that you can depend on. People that you can put hard questions to and get an honest response from. If you don’t have something like this or you don’t have people in your organization that you can depend on for this then you are just flying blind.

These are just some of the things I have seen while looking up through the myriad of organizations that I’ve been involved with. Thanks everybody … I’ll be here all week …

The dust has finally settled on the Firefox 1.0 release and we’re finally getting a picture of what exactly happened last week.

Tuesday was a pretty hectic day but I have to say that all of the Mozilla and Gentoo folks that pitched in to help were absolutely heroic. We figure there were about 100 million hits across the three www.mozilla.org servers for a total of about 7 million unique visitors. Quite amazing considering we had one old and heavily loaded box available as of Monday morning before the release.

As of Monday night, we had 10 mirrors in the main ftp.mozilla.org rotation. The majority of these were machines hosted at AOL that had extra IP’s so we could add them several times into the round-robin. These mirrors did brilliantly until they just about collapsed around 2pm PST on Tuesday.

Plan b) came in the form of a download redirector that would use all of the mirrors at once. At the time of this writing we’re looking at close to 2.7 million downloads of Firefox leveraging our new download redirector (that may be released in the near future). Now we just need to trim up the main archive so that we can then advertise for more mirrors. If we’re going to hit 15% market share, we’re going to need a lot more mirrors.

The other issue we were having was with update.mozilla.org service. This site sees connections from every Firefox client that is on the network. Fortunately the clients connect at random times so the load is pretty steady. The downer is that there are so many clients … David Miller put a Squid cache in front of the update.m.o service and we saw an immediate improvement but it was still a bit sluggish. We added one more squid cache on another machine and that made all of the difference in the world. We’re able to handle a lot more connections now and the application server doing the heavy lifting in terms of version checking, etc no longer has to burden itself with the likes of handing out images over-and-over … its all in the Squid caches. BTW – 8GB of memory on one box with your Squid cache means you can put everything in memory … wow is that fun and impressive to watch in action.

Watching this release take off … watching everybody respond so well … seeing it all in action was one of the most impressive things I have ever seen. Yes, we had some hiccups in service. But we got through it and we’re making plans for the future.

Thanks go out to the boys at CNN that helped tweak our squid configurations. Lance & Stuart from Gentoo for configuring up the Nocona box that just wouldn’t stay up how we’d configured it. Blizzard for putting up another www.mozilla.org mirror. All of the mirrors across the globe; cshields, Peter Losher, Neil Bright, etc. Everybody in #bmo on irc.mozilla.org that pitched in and countless others that I know I’m missing.

After much anticipation, Firefox 1.0 is out now. And because there wasn’t enough said about it on the NET today, I’m going to anty in as well.

This morning around 12am PST Firefox 1.0 hit the streets. The download infrastructure held up pretty well for most of the day but finally crashed around 2pm PST. Fortunately we had a plan b). More on that in a bit.

Just about every site out on the NET had a link to www.mozilla.org and subsequently it was soon to be a goner. We threw in a couple of machines to the www.mozilla.org rotation and fortunately they don’t do anything funny with their website. All straight up HTML/CSS and no PHP or CGI’s … thank goodness. By about 7am PST they we were serving up close to 9000 requests/sec across all of the web servers in the rotation. One of the machines we threw in was a Dell 1850 with the new Intel Xeon’s with 64-bit extensions. It promptly fell over with some sort of memory errors. Switching the machine to nptl threads (from LinuxThreads) and using mpm worker with Apache 2.0 we were able to really rein in the box. By the afternoon it was happily chugging along at 500-800 requests/second with a load average of a little less than 0.20. All of this brought to you by Gentoo Linux. We’ll see how the web infrastructure holds up after the NY Times ad hits.

One of the biggest issues with any of the Mozilla products is getting the bits out to the end-users in a fashion that won’t require them to be clicking around some web tree trying to figure out what they need. In addition, we have about 40+ mirrors across the globe, each with varying available bandwidth, that we need to spread the load across. On top of all of that, we’d love to know how many downloads for each product, OS, etc from each mirror. This of course is a bit of a daunting process.

Mike Morgan and I wrote a mirror administration application that would handle the above issues. By 2:11pm PST we had it in production and it worked flawlessly. We were able to spread the load over the rest of the available mirrors and weight them based on available bandwidth. We would query each mirror every 15 minutes to make sure they were available and had the right files for each product. After the first hour we had processed a little over 50,000 downloads. All available through a pretty web interface all running on a Gentoo Linux machine that wasn’t even creeping above a 0.10 load average. We’ll see how this holds up during the big releases we have to handle in the near future.

I drive a 1973 Toyota Hi-Lux pickup truck. This sucker has well over 100,000 miles on it (odometer got stuck sometime after the first flip) and it keeps on tickin’. Its not fast, I can’t tow much or carry that much in the back but I’ll tell ya what; its perfect for my needs. I drive it once a week or so or during vacation times I use it a lot when I’m doing work on the house. It meets my needs.

Now, I could be like a lot of people out there who want to buy a big huge Ford or Chevy truck, but why? Its not what I need. That said, Ford and Chevy owners would scoff at me for owning my truck; they laugh at it. But why on earth would I want a huge truck payment every month and the pain of fueling a mongo truck like that in this day-and-age? It doesn’t meet my needs. You know what? I paid cash for my truck (a whopping $850), insurance is less than $60 a year and I never put more than $5 of gas at a time in there. Exactly meets my needs.

So, when I hear people scoffing at my choice of Debian, Gentoo or some other not-uber-popular distro it just makes me laugh. Again, why do I want the hassle, the up-keep, the payments, etc of something that doesn’t exactly fit my needs?

Often times, the people that do laugh/question my choice of distribution usually do so totally biased. They use the Ford or Chevy truck of distributions and haven’t ever even taken the Hi-Lux for a spin. You know what? Its different; but different isn’t always bad.

What are some of the factors in choosing your distro you should consider? Well, I can name a few; security, licensing, cost, ease-of-use, community and access to expertise. If you’re a Red Hat or SuSE user and you continually deride the other distros ask yourself this; why? Have you ever tried one of these other distributions? Have you engaged with the communities surrounding them? One of the most powerful aspects of community-based distributions (even the lesser-known ones) is just that; the community.

Don’t just scoff; consider, be reasonable and leave your snap judgements at the door.

I have always thought that Netcraft was a great site and I visit it pretty regularly. I have been reading their articles about Linux usage across the globe and I’m curious that nobody has targeted the Cobalt market.

Cobalt was an early-entry Linux company that was skilled in making appliance-type network boxes for doing all sorts of things like web serving, DNS, DHCP, etc all in one little blue box. They were really great for smaller shops that needed these types of tools in a reliable fashion but without the cost or headaches of having to do it themselves from the ground up. I have worked with these and they are actually quite good pieces of technology.

In 2000, Sun Microsystems acquired them for $2 billion during the height of the dot com craziness. Netcraft recently reported that Sun was discontinuing the Cobalt line. The sad part here is that Sun has not offered up a replacement strategy for the easy-to-use Cobalts.

According to Netcraft as of July 2004 Cobalt still maintains a 20% market share of the Linux server market. That 20% comes in at well over 500,000 machines with only the top 20% of that being by the big name ISP’s. So, somewhere out there are 400,000 Cobalt’s running soon-to-be out-of-date software in need of a replacement. I see two very interesting scenarios here.

1. LiveCD city baby: Most of the Cobalts have CD’s in them. Why couldn’t someone build an easy-to-use LiveCD to either run on or re-install the Cobalts with their OS? Seems like a great opportunity to get some real mindshare as well as machine share for their Linux.

2. Build a better box: With the likes of Pogo Linux and Penquin Computing that specialize in Linux hardware out there, why have they not targeted this market? If you built some custom migration tools and provided pay-for-support for users migrating you just might have a case for getting at some of those 400,000 machines out there. Once you hook them on your products and skills, you can leverage it into more business.

Looking across the Linux landscape even Debian GNU/Linux barely has more machines than there are Cobalts out there. Seems like an excellent opportunity for growth for some lucky Linux distro to me.

I just love my job. I can’t think of a more fun thing to do than to get together with like-minded folks and coordinate on possible projects. Especially when it has to do with open source.

I got a chance to meet with Jay Thompson yesterday from Edge Design about collaborating on a project. Edge has been around for about 20 years doing a mix of graphic design and web work. They have also done some pro-bono work for the likes of the Oregon Food Bank and the Children’s Cancer Association.

Jay has had contacts with several similar groups that are looking for technologies to help enable their organizations to better leverage their volunteers. On a second and third meeting with these different groups, he realized they all really wanted similar variations on the same tool. He immediately thought of open source and set out to find a technology partner to help make that happen.

Many of these non-profit groups have many volunteers and they need to be able to do an assortment of tasks (ideally from the convienience of a web browser). Track time on projects, scheduling of tasks/meetings, portfolio management and reporting. The idea would be to build a modular, open source application that does all of these (essentially an enhanced groupware) that allows users to enable/disable the bits and pieces they want as well as easily develop against the API to quickly extend the application. This is right up our alley.

I am very excited about working on this project. I think it would be an excellent opportunity for the OSL to get some more development work but would also give us a chance to help spearhead the creation of a trully useful open source tool that touches more than just a bunch of geeks (not that I don’t like geeks but they have a lot of fun software already).

Read an excellent article by Brad Wheeler from Indiana University about the possibililities for Open Source in higher education by the year 2007.

This is a great article with excellent talking points about the how’s and where’s of Open Source as well as some thoughts for folks with different perspectives; university administration, foundation, etc. I have been seeing some interesting trends recently and reading Dr. Wheeler’s article really helps put them in perspective.

During my visit to Linux World Expo last week I had a chance to attend a reception held by the OSDL which was given in part to give a chance for the University affiliate members to meet in person. While there I met with Joe Little who is a systems administrator from Stanford. Joe mentioned the fact that a lot of Universities right now are busy trying to capitalize on open source in the sense of each forming labs and trying to create a critical mass at their school. This essentially goes against the nature of open source that it is supposed to be a bit more organic and not so forced. I have to agree completely and unfortunately we’re as much to blame as any other University (see the OSL).

This morning on the univ-linux list Seth Vidal mentioned that it might make sense for the higher education schools to get together:

“Has anyone here ever considered putting together a conference for sysadmins, researchers, etc of linux on college universities and research institutions? Just to get an idea of similar goals, areas of overlap, etc. If something like that were to come about what areas would be of most interest to have talks on? What time of the year would be best?” This is a fantastic idea.

Back to Dr. Wheeler’s article, there is a great quote that comes from a veteren CIO, “With open source software, how do I know that the community cooperation that works today won’t dissipate next year? The Apache Web server has the Apache Software Foundation to ensure its evolution. What is the equivalent in higher education?” Putting this together with the thoughts of Joe Little and Seth Vidal, what do we do?

I am proposing a Higher Education Application Foundation. This foundation would be focused on combining the efforts of higher education across the globe using open source methodologies for sharing and collaboration. I know for one, the OSL would be the first to participate. We need to come together as higher education to help a) learn from each other and b) leverage the tools and experiences we have all learned over the years.

So where do we start?