November 6, 2009

Reminder: We are Retiring pre-Rev A OAuth on Monday, November 9th

Many of you may have already seen our forum post from October 14, 2009 talking about how Yahoo! is retiring the use of the pre-Rev A OAuth flow for its APIs and services.

Why are we retiring pre-Rev A OAuth?
There was a security vulnerability found within pre-Rev A OAuth early this year. The new OAuth 1.0 A revision has been put into place to repair these vulnerabilities.

For full details on this, please see the original postings:
- YDN Forum Post: http://developer.yahoo.net/forum/index.php?showtopic=1263
- YDN Blog Post: developer.yahoo.net/blog/archives/2009/04/oauth_update.html

Am I affected? Where do I find more information?
Yahoo! has set up a few sources to find information about this migration. If you have questions about the services that are affected, how this might impact you, or full details about how to migrate to the new version, see the following resources:
- OAuth Security Issue FAQ: developer.yahoo.com/oauth/faq/
- OAuth Quick Start Guide: developer.yahoo.com/oauth/guide/oauth-guide.html

My application has stopped working - Is it related to this change?
If your application uses OAuth, or an API that implements OAuth, and your application can no longer authenticate new users coming into the application, you may be implementing the pre-Rev A version. If you suspect this to be the case, see the OAuth Quick Start Guide to upgrade your version of OAuth.

Where can I ask questions or get developer support
To find a developer to speak to about this migration or to help with your application migration, you may post your questions or concerns to the OAuth forum at http://developer.yahoo.net/forum/index.php?showforum=42

Jonathan LeBlanc
Technology Evangelist
Yahoo! Developer Network
Twitter: @jcleblanc

Comments (0) | Permalink

Yahoo! describes its cloud infrastructure at 2009 Cloud Computing Expo

Yahoo! was a sponsor at the Cloud Computing Expo this week in Santa Clara, which drew more than 2,000 attendees from around the globe.

Shelton Shugar, Yahoo!’s SVP of Cloud Computing, presented a keynote on Yahoo!’s vision for cloud computing, and an overview of Yahoo!'s Internet-scale private cloud services. As part of his talk, Shelton announced the open sourcing of Traffic Server, which has already gained huge traction (Traffic Server team, you rock!).

IT Pro commented: “Yahoo is taking another step into the cloud by confirming it is opening up its Traffic Server to an open source version … This is the second time within five months that Yahoo has shown its dedication to the open source cloud, following its opening up of its Hadoop platform in June this year.”

Shelton also announced a new release of the Yahoo! Distribution of Hadoop, which includes exciting improvements to robustness, security, performance, and operability. The YDN published a summary of Shelton’s talk on its blog. Sys-Con published a transcript of its question and answer session with Shelton on its blog.

Yahoo! cloud technology leaders presented a comprehensive overview of our cloud services:

• Raghu Ramakrishnan provided an exciting general session covering Yahoo’s technology leadership in addressing the challenges of building global and scalable cloud solution.
• Surendra Reddy described Yahoo!’s cloud serving infrastructure, declarative approach, and the Open Cloud Protocol (OpenCap).
• Eric Baldeschwieler talked about the massive usage of Hadoop at Yahoo! with more than 25,000 nodes..
• Chuck Neerdaels described Yahoo!'s global cloud storage and edge solutions, specifically Sherpa, MObStor, and Traffic Server.
• Jonathan Trevor presented an introduction to Yahoo! Query Language (YQL). His talk included discussion of YQL Open Tables, which run on the Sherpa cloud storage service, and a cool live demo.

The presentations were complemented by a Yahoo! booth on the exhibit floor staffed by energetic engineers from our team (thank you!) who spread the word about Yahoo!’s cloud computing leadership.

Sys-Con, the event coordinator, described the event in a press release.

---

Dekel Tankel

Director, product management, cloud computing

Comments (0) | Permalink

November 5, 2009

Recap of PayPal X Innovate '09 Conference

I spent this Tuesday at the PayPal X Innovate 2009 conference. I wanted to share the positive experience my colleagues and I had at a great event focusing on one company's developer product line.

It appeared that there were more than 1,700 registered folks in attendance at the Concourse Exhibition Center in San Francisco over November 3-4, with folks having travelled from around the world (including Africa, Dublin, Germany, Italy, and other locations). Walking into the event, there were dancing violinists and mobs (maybe hundreds) of eBay/PayPal employees greeting attendees.

The conference area was preceded with a large refreshments area (which featured both powdered donuts and fruit cups) followed by a demo space. In many ways, this felt a bit like past MacWorld Expos. Twenty or so PayPal API integration partners had kiosks explaining their products, giving considerable central floorspace to the folks using the APIs. PayPal commanded the center of this pit, with a drove of technical and business employees around to answer questions. (Employees wore different colored shirts to identify their domain expertise to other employees, in case an employee was unable to answer an attendee's question.)

The keynote address featured eBay's CEO, John Donahoe, followed by PayPal president Scott Thompson. The crowd cheered their every announcement and joke. (You'd think they paid everyone in the audience... Oh wait, that's exactly what they do everyday.) They announced new pricing tiers for their products, which went over resoundingly well, and they also announced a slew of new APIs for 3rd-party integration. Donahoe made exceptionally clear, too, that PayPal has a much bigger audience than eBay, and he was betting the company on PayPal's ability to own the payments/transactions space... especially thru its developer program.

The keynote then moved on to Osama Bedier, VP of PayPal Platform and Emerging Technology, who walked thru a number of new API integrations from partners: including a corporate Accounts Payable live demo of eBay paying SAP $88K in seconds, Payvment's in-Facebook-Apps' open-source shopping cart system, and Sun's JavaStore, among others. He also demoed their upcoming Objective-C SDK for in-App payments for iPhone Apps--a crowd hit, considering Apple's 30% cut of any developer payments using Apple's hooks.

Sessions throughout the day were standing room only, with at least 250 people each session. Our very own Cody Simms, Senior Director of Product Management, joined a panel on "The Present and Future of App Stores," discussing the future of application distribution, discovery, and monetization with others from Motorola, Qualcomm, and GetJar.

All in all, the event was incredibly polished. It was clear that PayPal takes this audience (and developer initiatives in general) very seriously. Aside from the scale of the event, the number of simultaneous lesson tracks (6), and the number of employees on-site, every attendee received their very own Asus Eee 1005HA netbook, which, as it turned out when I got home and popped it open, came pre-installed with PayPal documentation, tutorial and greeting videos, bookmarks to their docs and dashboards, and IDEs pre-configured with PayPal hooks. And the battery was charged up. And it can be Hackintoshed, which is a good challenge for any hacker geek. Very nice touches.

PayPal's value proposition is clear to its developer audience: use us to make money. It warranted nearly 2000 folks from around the world to shell out ~$300, fly out, and learn about what PayPal is now rolling out to the world. And with the horde of employees on hand to answer detailed use case questions (some spending 3-4 hours per attendee to answer questions), it also seems very clear that PayPal made it worth their while.

Micah Laaker
Director, Y! Open Strategy UED

Comments (1) | Permalink

Expert Yahoo! Developers on Elance

Elance is an online marketplace that lets anyone hire, manage, and pay independent professionals and contractors online. If you're an independent software developer anywhere in the world, Elance is a website where you can list your skills and offer your services. They've been around for a while, and along with payment mechanisms, Elance has also developed rating mechanisms to vet competence and help protect the quality of interactions between freelancers and hiring organizations. You could think of Elance as an eBay for freelancers.

That's why YDN set up a Yahoo! Developers group on Elance -- to give independent developers with demonstrated expertise in Yahoo platform tools and technologies a place to make themselves available to companies and individuals seeking developers with Yahoo!-specific expertise. Members of this group have experience working with the Yahoo! Application Platform (YAP), Yahoo! User Interface Library (YUI), or another of the major areas within the Yahoo! Developer Network, and are ready to apply that expertise to your project. These developers aren't endorsed by YDN, but they have completed some tests we designed to demonstrate their knowledge.

So, if you want to work with Yahoo!'s awesome developer offering -- create a profile on Elance and join the Yahoo! Developers group. Or, if you want to find a developer, be sure to check out the Yahoo Developers listed on Elance.

Thanks!

Yahoo Developer Network

Comments (0) | Permalink

November 4, 2009

Tech Thursday - words, galleries, gardens, images and wrong shifts

Remember, remember the fifth of November...

Every Thursday is Tech Thursday where we share a random assortment of technical links we found and liked.

• WordNik is a new API that could finally end the search for a good, free, dictionary API that is not NSFW like Urban Dictionary
• There is a pretty sweet Government Data Hackathon going on in the US right now sponsored by Sunlight, Mozilla, Google, Redhat and Fedora.
• Two new galleries to play with and contribute to. Check the YUI3 gallery and also take a stroll in Flickr's app garden.
• Mockingbird is a wireframing tool that makes it easy to sketch apps and link them.
• Mozilla partnered with font designers to define WOFF, a packed font format for web use.
• Did you know that you can use the RAM of graphics cards as swap memory?
• An interesting stroll down memory lane about why we have an IMG element in HTML.
• It seems that the age-old IE bug of keyboard navigation moving the screen but not the focus is also a Safari and Opera issue and here is a fix. Better would be to file bugs though.
• Cartographer is a thematic mapping library for Google maps.
• DebugBar helps you make debugging in MSIE much easier.
• Disk Inventory X shows you where all that free space on your mac harddrive went.
• Mu is a library to make using Facebook connect much easier.

You can propose links to us on Twitter (@YDN) or try bookmarking them on delicious with the tag "forydntt".

Comments (0) | Permalink

A summary of Shelton Shugar's talk: "Accelerating Innovation with Cloud Computing"

Shelton Shugar delivered an excellent keynote address “Accelerating Innovation with Cloud Computing” to open the 4th Cloud Conference and Expo in Santa Clara yesterday.

Shugar, Yahoo!'s SVP of Cloud Computing, started off by clarifying that Yahoo! isn't selling anything at the Expo; the company is not into consulting or selling software. At Yahoo!, cloud computing is not about saving money. Yahoo!’s motivation is rooted in the fact that cloud computing drives innovation. Yahoo! has hundreds of products and platforms all over the world. Many of these products were the result of acquisitions, and so they came with their own infrastructure, down to the metal. Cloud computing at Yahoo! is about streamlining the services these products and platforms require. Yahoo! stores hundreds of petabytes of data all over the world, and handles petabytes of internet traffic daily. Scale is of utmost importance.

Yahoo!’s Cloud Strategy

Yahoo! is building a private cloud, deployed in data centers world-wide. Yahoo!’s cloud strategy focuses on two areas: data processing and serving. Data processing refers to data mining and analysis. Serving refers to application environments for Yahoo!’s products, edge capabilities for fast delivery, and a channel for data to flow into storage. This is a multi-year effort. Open source projects play a “central role” in this strategy; Yahoo! consumes and produces them.

Looking Inside the Yahoo! Cloud

The Yahoo! cloud has five components:

1. Edge services
2. Cloud serving where we host applications
3. Online storage for serving content to consumers
4. A batch-processing data warehouse
5. Data collection services to filter and de-duplicate incoming data, and block abusive requests

Edge serving is based on the Yahoo! Traffic Server. Over half of all Yahoo! traffic flows through YTS.

The application serving layer is based on a tiered architecture. Applications can be cloned. Traffic can be split natively, which allows for bucket testing. Developers are freed from having to worry about versions of web serving software, locations of machines, etc. Capacity can be adjusted easily.

Online storage uses RESTful APIs. It’s deployed worldwide. Global replication is supported natively. Multiple consistency models are provided. MObStor (mass object store) is used to store large objects (1MB-2GB) such as images and video. Objects are immutable. Structured content is provided via a product called Sherpa, a key-value store. Sherpa is intended to support enough of the capabilities developers currently depend on relational databases for.

Batch processing is oriented around Hadoop. Hadoop has been running for a few years, and now operates on tens of thousands of machines. Yahoo!’s Hadoop grid stores over 80PB of data. Yahoo! uses it to optimize advertising, process weblogs, etc. Thousands of Yahoos are trained to run jobs on this grid. The Hadoop File System (HDFS) allows thousands of computers to be treated as a single machine. Pig is a higher-level procedural language that generates MapReduce code. It’s almost as efficient a well-written MapReduce code, though the Pig team jokes that most people don’t write well-written MapReduce code. Yahoo! is building columnar storage.

Examples

The Yahoo! homepage When a user visits the Yahoo.com homepage, he/she is interacting with Yahoo! cloud services. The popular stories to display are selected using a feedback loop involving several cloud components. Hadoop is used to optimize ad-matching and build the search index. Edge services are used to cache and load-balance the page content and normalize the news feeds.

Yahoo! Mail

Yahoo! Mail uses Hadoop to identify and filter spam. Before Hadoop, mail engineers had to spend lots of time maintaining storage and machines to process an enormous amount of data. Hadoop abstracts scale, handles failures, and manages multiple users. This allows the scientists and engineers to focus on their jobs. Yahoo! Mail uses cloud storage’s replication services to help detect abuse.

Yahoo! Sports

People want to find game scores as fast as possible. Edge services provide a proxy service to route requests for dynamic content, allowing Yahoo! Sports to provide the most up-to-date content.

Yahoo! Finance

Yahoo! has the most popular finance page on the Web. Yahoo! Finance uses Hadoop to speed advertising by optimizing resource utilization.

Yahoo! Query Language (YQL)

YQL is an SQL-like language. It allows developers to query, filter, and join data from across the Web. YQL uses Sherpa instead of managing its own database.

Open Source at Yahoo!

Yahoo! contributes the code it produces for Hadoop back to the Apache open source community. External developers benefit and contribute back, which in turn benefits Yahoo!. Pig is open source. Zookeeper, a utility Yahoo! uses to coordinate multiple systems, is also open source. Yahoo! is a member of Open Cirrus, a consortium designed to facilitate research in cloud computing. The consortium is composed of nine member companies. Yahoo!’s contribution is m45, a thousand-core cluster. Yahoo! works with some of the leading universities in the world, including University of California Berkeley and Carnegie Mellon University. Yahoo! has built an enormous community around Hadoop. As a result of it’s investment in open source, Yahoo! can now hire people directly out of university to work in several areas of cloud computing. Open source attracts the best and the brightest.

Shugar’s announcement of Yahoo!’s action to open-source its Traffic Server, now an Apache Incubator project, was a highlight of his keynote. Trafic Server can process up to 35,000 trasnsactions per second on commodity hardware. It’s modular and forms the basis for Yahoo!’s caching, proxying, load balancing, routing, etc. Yahoo! pushes 400TB through it daily. Yahoo! hopes to support a vibrant community around use of the Traffic Server like it did with Hadoop. A recent GigaOm post on OStatic gives more information about the project.

Back in June, Yahoo! announced the Yahoo! distribution of Hadoop. Yahoo! selects only the components it needs and tests them well. It's a solid collection of code that’s been proven to work. Yahoo! will be releasing an update shortly.

Change

At the close, Shugar reminds the audience that Yahoo! is fully committed to cloud computing, but “moving to the cloud requires change.” If your organization is like Yahoo!, with lots of legacy systems, you’ll need to make a large organizational commitment, more like a marriage than a transaction. It takes time and investment to create cloud services and migrate to them. Yahoo! migration is a multi-year effort, but cloud computing is worth it. Developers are able to deploy so much faster than before. It’s changing the company culture.

---

Erik Eldridge
Engineer/Evangelist
Yahoo! Developer Network

Comments (0) | Permalink