Hadoop computes the 10^15+1st bit of π (Hadoop and Distributed Computing at Yahoo!)

My Projects
Tools & Services
APIs and Services
- Flickr
- Geo
- Mail
- Search
- Social
- Updates
- YAP
- YQL
Developer Tools
- Hadoop
- YUI
- YSlow
- View all »
Resources
- Blog
- YDN Theater
- Hack U

YDN Hadoop and Distributed Computing at Yahoo! Hadoop computes the 10^15+1st bit of π

Yahoo! Developer Network Blog

May 13, 2009

Hadoop computes the 10^15+1st bit of π

I used Yahoo's Hadoop clusters to compute the 1,000,000,000,000,001st bit of π. The 7 hexadecimal digits of π starting at the 10^15+1 bit are:

6216B06

Although Hadoop is primarily used for data-intensive applications, it can also be used to run CPU-intensive jobs on many machines. Computing a range of bytes in π using a BPP-type formula, requires a lot of arithmetic operations and therefore CPU, but not much storage. When computing the 10^15+1st bit of π, the first 30% of the computation was done in idle slots of our Hadoop clusters spread over 20 days. The remaining 70% was finished over a weekend on the Hammer cluster, which was also used for the petabyte sort benchmark.

This validates the results calculated by PiHex, which took more than 2 years on 1734 computers from 56 different countries.

My program was written entirely in Java and ran on Hadoop 0.20. An earlier version is checked in as a Hadoop example named BaileyBorwinPlouffe. The new code will be uploaded soon.

-- Tsz Wo (Nicholas), Sze

Posted at May 13, 2009 8:00 AM

Bookmark this on Delicious

Comments

This experiment is awesome, Nicholas! Demonstrates that Hadoop can be used to rn CPU intensive jobs as well!

Posted by: dhruba at May 14, 2009 5:52 AM | Permalink

Even taking into account that the first third was done during idle slots, Hadoop did this in under a month compared to PiHex's 2 years? I know even just hardware has gotten much much faster since PiHex's calculations (circa 2000?) but that's a pretty huge difference.

Posted by: HB at July 24, 2009 11:24 AM | Permalink

The huge difference probably is because the Hadoop framework efficiently utilizes the cpu recourse in a cluster. Note that the computation would have been completed within 3 days if it was performed by Hammer alone.

PS: The new codes have been checked in to Hadoop 0.21.

Dhruba and HB, thanks for your comments.

Posted by: Tsz Wo (Nicholas), Sze at August 3, 2009 2:03 PM | Permalink