Hadoop User Group (Hadoop and Distributed Computing at Yahoo!)

Welcome, guest Sign In

My Projects
Tools & Services
APIs and Services
- Flickr
- Geo
- Mail
- Search
- Social
- Updates
- YAP
- YQL
Developer Tools
- Hadoop
- YUI
- YSlow
- View all »
Resources
- Blog
- YDN Theater
- Hack U

YDN Hadoop and Distributed Computing at Yahoo! Hadoop User Group

Yahoo! Developer Network Blog

March 1, 2009

Hadoop User Group

Hadoop User Group meetings have now been held in Beijing, Berlin, London, New York, San Diego and Washington DC, in addition to the Bay Area, with one in the works in Bangalore. In the Bay Area, we typically host them on the third Wednesday of each month at the Yahoo! campus in Santa Clara.

The meeting last week featured Matei Zaharia from UC Berkeley talking about the Fair Scheduler for Hadoop. The need for a scheduler has been a known requirement for quite a while, and Matei got started working on this while he was an intern at Facebook. His talk described their goals of providing fast response time for small jobs and guaranteed SLA’s for production jobs. It then discussed the concept of pools, the scheduling algorithm for assigning resource capacity, as well as installation, configuration and administration of the scheduler.

This was followed by a talk from Aaraon Kimball from Cloudera on Importing Data from MySQL which discussed techniques for loading data from databases into HDFS.

Next month’s user group meeting will feature Yahoo!’s Milind Bhandarkar talking about performance enhancement techniques for Hadoop developers.

Posted at March 1, 2009 8:14 PM

Comments

Your readers, particularly Hadoop developers and users, might be intersted in Aster Data Systems' upcoming webinar on MapReduce for Data Warehousing and Analytics. There is an overlapping interest in both Hadoop and MapReduce, and the two frameworks are sometimes complementary. To register for the webinar, please visit www.asterdata.com/mapreduce_webinar.

Posted by: Ryan at March 17, 2009 11:21 AM | Permalink

Fulltime MySQL DBA position in NYC. Please contact Peter@mapssg.com:

Looking for a hands-on MySQL DBA with over two years of experience in designing and administering MySQL database servers. Day-to-day activities include supporting multiple software teams with their MySQL needs, including schema review, index optimization, general optimizations, data migrations and server management.

As data sets continue to grow we will also need to migrate our backup and recovery strategies to more sustainable solution (LVM, MySQL Master-Master, etc.). The candidate should also possess a desire to work with large statistical data sets and help make sense of them using technologies such as MapReduce/Hadoop/etc.

Additionally, the individual should be creative when designing MySQL and other data related solutions with the ability to "think outside the box"

The ideal candidate will posses a working knowledge of:

- MySQL 5.0, 5.1

- MySQL Replication -- 5.0 and 5.1

- MySQL Management & Administration

- MySQL internals (not code-level, but in principle)

- RedHat/CentOS Linux (required) and Solaris (preferred)

- Linux Kernel optimization/tuning

- File Systems & File System Management

- General UNIX/Linux System Administration skills

- Systems Automation

- FOSS monitoring tools (Nagios, Cacti, Monit, Munin, etc.)