I just got forwarded an article by Hank Williams from a client who has been watching the ongoing Twitter scalability rumpus that has kept us entertained for literally years now. Blaine's recent point about scalability is one I would mostly agree with, and one which I'm sure he's tired of making. I haven't really been following the Twitter-talk on all of the techie blogs, except through hearsay. "Did you hear that TechCrunch said Twitter is dumping Rails?" "Did you see that post about Twitter scaling?" Yada yada yada. Half of it is inaccurate the moment it is posted, and the other half is fraught with misunderstanding and wild speculation.
One of the reasons I've been loath to follow these discussions is underlined by William's concession at the end of his post that, indeed, Twitter's scaling challenge is a "hard ... problem that requires a very specialized ... architecture." Aren't they all? And so that is why I don't read the pseudo-tech jibba-jabba of TechCrunch and others about the scalability of Rails in general based on the experience of one very unique site. Scalability is unique to each site. Indeed, you are a unique site when you have to worry about scalability. That means you are getting lots of traffic, and probably making money. Most sites don't make it that far.
But I didn't write this post to join the Rails scalability rant. I just wanted to mention one observation that I haven't heard yet in the Rails scalability discussion. Here it is: the web is a moving target, and the current discussion often fails to acknowledge that. The web today is demanding more of frameworks, languages, developers, and system architectures. For instance, on MOG, we knew scaling would be a major problem from the get-go because a central piece of the application involved hundreds of thousands of agents running on users' disks, reporting tens of thousands of songs. What website was doing that in 1997? For what piece of software are those numbers not a scalability issue? It's the same with Twitter (as Hank points out): millions of people tweeting all at once, very frequently, via web, phone, Facebook apps--who knows, maybe there's even a Twitter client for your hair dryer or coffee maker now. This sort of challenge is (relatively) new for the web. And new ones like it are being invented every day. The target is constantly moving and changing shape.
If we nailed down the goalposts of web scalability, no one would bother posting and trolling and debating about Rails scalability. It would be solved by a handful of good engineers in a couple of weeks. For good. But what would be the fun in that? You'd never hear about another site that does something you wouldn't think possible 5 years ago. And, as an engineer, you'd never have the fun of solving new scalability challenges to support those new sites.
Sorry, comments are closed for this article.