June 17, 2009

Using Public Data for Good With the Power of YQL

The problems I work on every day at Yahoo! are mostly interesting intellectual challenges. In one way or another, these challenges benefit Yahoo!'s users; you, our developer audience; or our shareholders. However, sometimes I get the desire to do more, as I suspect many of you do too. Being a developer gives you an amazing power of creation. You are not a sheep. You have the ability to change things, from small incremental changes to, perhaps, the whole world. This project was big one for me: I really hope I am sowing the seeds of something much bigger, and with your help I will be.

So what exactly are we releasing? The first part is a new batch of YQL tables providing data on the U.S. government, earthquake data, and the non-profit micro-lender Kiva. The second part is an incredibly easy way to render YQL queries on websites. After all, what good is data that no one can see? There's a lot to talk about so I'll go through it piece by piece.

I was incredibly excited when data.gov was released. While I'm not a U.S. citizen, I think open democracy is incredibly important. Seeing Americans getting more involved in what their government does and how it represents them is a wonderful thing. In the UK (where I was born), there have been a number of inspiring projects such as They Work For You and Fix My Street that help ordinary citizens keep an eye on their political representatives, ask their local government to fix potholes, and more.

To help you use your awesome creative power to get involved as developers we've been working with the Sunlight Foundation to make U.S. government data easier to access and easier to create mashups with. Today we are releasing YQL tables for the Sunlight Labs API. The Sunlight Labs APIs let you search for information about the government. For example, you can find the details of your congressional or senate representatives by name or district. We are also releasing a YQL table for USASpending.gov which allows you to find out where your tax dollars are being spent.

You might be wondering what YQL adds to this mix. It's the easy power of YQL to access this data and mix it together that makes it special. So why not try this query? You'll see the details about the representatives in Congress from the five districts in California that got the most federal funding in 2006.

Our other project with the Sunlight Foundation is the 50 States Project. This one is a little closer to home for many of you. Sunlight want to make sure that legislative information for every single state is easily available. Many of the state government websites are hard to navigate, and provide information in formats that aren't exactly Web 2.0 friendly (CSV anyone?). The 50 States Project aims to rectify that and make important information easily available for anyone who wants to know what's going on in their state. We've built an example of how you can scrape data by building open tables to get the data from the Louisiana legislature. We hope we can inspire you to write many more tables, one for every state. I'm going to be writing in more detail soon about how I did it and what I learned, and how that can be applied to get data from any web form system. If you live in the Portland, Oregon area and are interested in hacking on the 50 States Project, I'll be up at the Hacker Lounge for OSBridge this week.

In the meantime, my colleagues Erik Eldridge and Jon LeBlanc have written about their experiences building open tables for earthquake data and creating rich widgets using YQL data, which are also launching today.

All in all, YQL is making it possible to do amazing things with data for democracy. You can find all these tables on our YQL GitHub repository.

Update: all these tables are now also available via the "Show Community Tables" link in your YQL console.

Tom Hughes-Croucher (@sh1mmer)
Yahoo! Developer Network

Posted at June 17, 2009 1:00 PM | Permalink

Bookmark this on Delicious

Comments

Posted by: Tom Hughes-Croucher at July 1, 2009 12:02 PM

Posted by: me at August 14, 2009 7:41 AM

Posted by: william eveland at December 10, 2009 1:20 PM