March 7, 2008

An Introduction to ZooKeeper Video

A few weeks ago, I had the chance to capture video of a presentation given by Benjamin Reed from Yahoo! Research. His presentation was an introduction to ZooKeeper, a highly available and reliable coordination system built by Yahoo! Research and released under the Apache License, Version 2.0.

Preparing to post the video, I asked Ben for a a summary of the motivations for building ZooKeeper. Here's what he had to say:

In 2006 we were building distributed applications that needed a master, aka coordinator, aka controller to manage the sub processes of the applications. It was a scenario that we had encountered before and something that we saw repeated over and over again inside and outside of Yahoo!.

For example, we have an application that consists of a bunch of processes. Each process needs be aware of other processes in the system. The processes need to know how requests are partitioned among the processes. They need to be aware of configuration changes and failures. Generally an application specific central control process manages these needs, but generally these control programs are specific to applications and thus represent a recurring development cost for each distributed application. Because each control program is rewritten it doesn't get the investment of development time to become truly robust, making it an unreliable single point of failure.

We developed ZooKeeper to be a generic coordination service that can be used in a variety of applications. The API consists of less than a dozen functions and mimics the familiar file system API. Because it is used by many applications we can spend time making robust and resilient to server failures. We also designed it to have good performance so that it can be used extensively by applications to do fine grained coordination.

We have found ZooKeeper to be applicable to many distributed applications inside of Yahoo! and expect it to be applicable to many more outside of Yahoo! For that reason we released it as open source under the Apache license. If you are writing a distributed application, ZooKeeper can help.

And here's the video...

download (m4v)

A PDF copy of the slides is available too.

While filming his 1 hour presentation, I found myself really wishing that ZooKeeper was available 6 or 7 years ago when I was struggling with how to perform distributed processing of news feeds for Yahoo! Finance. ZooKeeper is clearly a more elegant solution than the hack we put together!

Ben will be speaking about ZooKeeper later this month at the Hadoop Summit.

More videos are available on YDN Theater.

Jeremy Zawodny
Yahoo! Developer Network

Posted at March 7, 2008 2:52 PM

Comments

Posted by: jonathan at June 13, 2009 10:00 AM | Permalink