October 4, 2007

Platform Dreams?

Over on the Union Square Ventures blog, Albert Wenger penned an excellent rant/request titled I Want a New Platform, partially inspired by the Huey Lews & The News song I Want a New Drug:

One that won’t go away (under load), one that doesn’t keep me up all night (worrying about scaling), one that won’t make me sleep all day (when I should be adding features). If you have tried to build an Internet site or application recently that needs to work for thousands or tens of thousands of concurrent users you may share this desire.

Why is this still so hard? Why do we find ourselves worrying about locating experts in the dark art of database performance tuning? Why are we spending time haggling with Rackspace over the price for another set of servers or racking our own servers at Equinix? Why are we writing our own user management (from scratch)?

He goes on to talk about the inadequacy of the tools we're all using.

Because we are using tools that simply were not made for the job. Relational databases have gotten faster and better, but their fundamental construct of data stored in rows and neatly parceled out across tables (which frequently need to be joined) does not really match up with either rapid development using some notion of objects or with scaling horizontally using commodity hardware. Web servers started out by simply providing static content and were then forced into running applications with the result of not doing either particularly well. Yes, people have been writing new ones to address that, but even those are fundamentally designed to work on a single machine. The second you go beyond one machine you need separate load balancers, reverse proxies, caches and all sorts of other paraphernalia just to make stuff work together. Worse yet, the web server, the application and the database are connected to each other via thin straws that were bolted on after the fact with important information either not passed at all or only painfully (e.g., information about which user is requesting a particular set of data).

The post goes on to talk about a lot of specific problems that everyone runs into sooner or later. It's a really good read, including the comments.

Our question for you, the developers looking to Yahoo! for technology, is this: What should Yahoo's role be in solving these problems? What high- or low-level services would help to change the game in building the next generation of on-line applications?

We clearly believe that Infrastructure like Hadoop is part of the solution. But that's only the tip of the iceberg.

Jeremy Zawodny
Yahoo! Developer Network

Posted at October 4, 2007 11:01 AM

Comments

Posted by: Tim Tripcony at October 4, 2007 12:35 PM

Posted by: Sumit Chachra at October 4, 2007 1:00 PM

Posted by: Jeremy Zawodny at October 4, 2007 1:50 PM

Posted by: Danny at October 7, 2007 12:55 PM