July 2, 2009

The state of mobile browsers - PPK in London

Browser tester and researcher extraordinaire Peter-Paul Koch of Quirksmode visited our office in London recently to give a talk about the state of play in the mobile browser world.

Today PPK (as he prefers to be called to keep things brief) sent through the PDF versions of his talk. Stay tuned for a video of the talk, it will crop up sooner or later on the YDN theater.

• Download the Slides in normal quality (12mb)
• Download the Slides in high quality (35mb)

I interviewed Peter-Paul about some of his findings and his ideas about where professional web development is going, and what constitute good testing practices.

The interview recording is available at the Internet Archive and you can also get the 33MB MP3 directly.

Christian Heilmann
Yahoo Developer Network

Comments (0)

July 1, 2009

GeoMaker - Turning web content into maps made easy

Yahoo! offers some cool tools that let you play with web content and enrich it with geographical data. Placemaker finds geographical information in texts or web addresses and lets you display the new-found information on maps. All of this is pretty easy to use for those who love to deep-dive into code and play the API game on the web.

As preparation for an upcoming tech talk about Placemaker I thought I'd have a go at making this a whole process a bit easier. A day sick at home, a dash of YUI and some PHP later, you can see the result: GeoMaker. In just three steps you use content to create a copy-and-paste map to include in your pages. Check out the following screencast to see GeoMaker in action:

As you can see, adding the website address as a URL parameter can shorten the process: http://icant.co.uk/geomaker/index.php?url=http://news.yahoo.com. Another nice-to-have: getting the locations marked up for you as Geo Microformats.

I'd like to get your feedback and once everybody's happy, the source code will be made available on GitHub so that you can host it yourself.

Chris Heilmann
Yahoo Developer Network

Comments (1)

June 30, 2009

Open Source Bridge 2009

General Notes

The Open Source Bridge conference was held at the Oregon Convention Center in Portland from June 17 to June 19. Sessions covered a range of topics: from building and growing open source businesses to yoga and meditation ... but the focus was decidedly technical, with some great sessions on a number of different OSS projects at varying levels of detail. The first two days featured talks of interest to the Open Source community, while the last day was an unconference. After the day was done, the Yahoo! Developer Network crew hosted a 24x7 hacker lounge with WiFi, to alleviate midnight hacking withdrawal symptoms.

I had the opportunity to attend an eclectic mix of sessions, and a few common threads emerged. As a Product Manager in the cloud computing group at Yahoo!, I'll focus on subjects that relate to the cloud, although there was no shortage of interesting discussion on a wide range of other subjects too.

The Cloud:

People in the Open Source Community are interested in cloud computing, but are also quite skeptical of it. Many of them see the cloud as a trick by hosting providers to make more money or to create buzz around existing technologies. (Quotable quote: "Cloud Computing right now is a 1000 mph bullet train of hype.") Others are excited by its potential, while still skeptical of its existing benefits. A third group, in the minority, is very excited about cloud technology.

Two of the sessions I attended, "Virtualize vs. Containerize: Fight!" and "Bridging the Developer and the Data Center" had the most interesting discussions around the cloud, although the subject came up in other sessions too.

Caching as the key to scalability

Whether applications are built in the cloud, in a private datacenter, or running off a box under your desk, the best way to make an application scale better is to find ways to limit the amount of work needed to serve a request. As dynamic, customized applications become the norm, caching becomes more important - and harder to do.

Caching can happen at multiple tiers: *Caching proxies can be used to minimize the number of requests to hit an application server that is generating dynamic pages. *Portions of a page may be static while others are dynamic. Caching page fragments can yield significant benefits. *Opcode caching optimizes the performance of requests when server-side scripts are used by caching the actual compiled code that is executed to serve the request. (For example, JSP does this by default by compiling pages into classes and then using JIT compilation to optimize and cache opcodes.) *Data and queries can be cached to minimize the number of requests, but this needs to be done very carefully.

Most modern web development frameworks provide built-in opcode caching. Couple this with intelligent application design, and you can make large portions of your website cacheable within the application server, significantly reducing the resources necessary to serve a request. Once you've optimized your application itself, you can add a caching proxy that reduces the number of requests that hit the application stack. This can significantly improve a web application's ability to scale. Being able to cache large portions of your web application, even for a few seconds, can significantly boost performance and scale because it's very rare that significant portions of all but the most dynamic sites change multiple times a second.

Once you've hit that barrier, it's time to look into more advanced technology to scale.

A new approach to data on the Web

Some of the most interesting sessions revolved around data and the web, and how the web changes assumptions about what data stores should and should not do.

Rethinking Web Databases

Brian Aker led an excellent session on Drizzle, a reworked microkernel database derived from MySQL and designed specifically for scale and concurrency - in other words, for the cloud. The code is derived from MySQL, but it's designed to have components that can be extended to suit specific needs. The core is optimized to provide only what is viewed as fundamental database operations, leaving everything else to be built as a plugin. This has meant killing some sacred cows, like triggers, stored procedures / prepared statements - even the query cache.

This approach works because the focus of Drizzle is using new technology - multiple cores and massively distributed, scaled systems, 64-bit computing - to address the problems of tomorrow. They avoid having to play catch-up to the big database engines, solving problems that have been solved before (e.g., ANSI compliance). The focus of Drizzle is the web - a group of customers that is still under-served by today's databases - so the feature set focuses on scalability and performance. The architecture provides for extensibility so that anyone who needs more heavy-weight features can add them via plugins. This approach make sense because many developers want to customize different features to suit them - e.g., many people have written their own replication mechanisms. Drizzle aims to support this kind of customization.

Brian said that Drizzle isn't quite ready for prime time yet, although some people are using it anyway.

It's great to see the open source community challenging the assumptions that have defined databases in the past and leapfrogging their proprietary competitors in serving the needs of the web. Building applications at web scale forces us to re-think traditional means of approaching problems. Frequently, one size does not fit all. Building a robust platform that can be extended as needed to suit specific needs makes a lot of sense.

Do you really need ACID?

While Drizzle started with an ACID-compliant storage engine (InnoDB) as the default, a thought-provoking session led by Bob Ippolito of Mochi Media asked attendees to think about how dropping ACID requirements could yield better scalability and availability - a tradeoff that many of the big Internet companies have cottoned on to already. This is another tenet of the cloud - services like our own Sherpa (our structured key-value store) and MObStor (our unstructured storage cloud) have embraced it, as have Amazon's S3 and SDB services among others. When availability, scalability, and performance are paramount, traditional application development models need to be re-examined.

The answer lies in dropping strong consistency in favor of eventual consistency, where the system eventually converges on a consistent state, but may or may not be consistent at any given time. Bob went on a quick tour of the various ways people have attacked this problem - distributed key-value stores (like Amazon's Dynamo), column-oriented databases (like HBase), memcached, document databases (CouchDB). He described how they've chosen to handle the issue of consistency: ignoring conflicts, resolving conflicts internally (hopefully, yielding the expected answer), and allowing the client application to decide how to resolve conflicts by returning all possible values.

This is an interesting area - there isn't a single one-size-fits-all solution, and there are few (if any) rock-solid open solutions to this problem ... yet. After looking at several open source initiatives in this area, Mochi Media decided to use a proprietary solution.

We've found that on the web, availability is paramount and in many cases strict consistency is significantly less important (unless you're building an online banking system!). Even though it would be wonderful to have a system that's simultaneously highly available, globally replicated and consistent everywhere (oh, and cheap too!), the laws of physics make that impossible. However, through careful design, it's often possible to build web applications that have limited (if any) need for strict consistency. Being willing to make this tradeoff opens up a host of means to achieve higher availability, and when these tools are available in the cloud, it enables developers to build better applications quicker.

Approaches to Computing in the Cloud

One of the new terms I learned at Open Source Bridge was "Containerization" - basically a form of lightweight virtualization, more commonly known as Operating System-level Virtualization. Before Open Source Bridge, I've always thought of containers as living higher up in the stack (e.g., application containers).

Traditional virtualization techniques (e.g., Xen, VMWare, etc.) virtualize the machine itself, splitting the host machine into several virtual machines that can then run guest operating systems which run developer code. This makes it possible to run different Operating Systems on the host and guests, and to have them all behave as though they have a complete machine at their disposal. On the other hand, Containerized systems (e.g., OpenVZ, Solaris Containers) provide Operating System level virtualization. Guest operating systems share the same kernel, but still present a virtualized view to the applications they run.

Both traditional virtualization and containerization offer similar benefits in terms of protection and consolidation. The fundamental difference is that Virtualization enforces stronger isolation between instances and provides more flexibility. In contrast, containerization ekes out higher performance from the hardware (since all code executes natively, there's a lower resource tax on the host). Think of virtualization as kernel-space, and containerization as user-space.

However, as hardware manufacturers add support for virtualization in the hardware, the performance gap is narrowing considerably. Similarly, while containerization doesn't enjoy as much enterprise and tools support, that's slowly changing too. As these two technologies evolve, perhaps they will converge on a system that shares the benefits of both approaches.

So... virtualize or containerize? As always, the answer is, "it depends ...!" Containers only work if you can (and are willing to) share the kernel between them, and they come with less enterprise/tools support and features. Virtualization has much more enterprise support, allows greater isolation and OS flexibility - but exacts a greater performance tax.

All the cloud providers today use virtualization and sell a fixed amount of virtual resources per virtual machine. However, in general, people should not assume that the default virtual machine image is optimized for their application. If they're using a cloud service, they should make sure they tune their system image for their application, or they may be left with a system that doesn't perform nearly as well as it could.

With hardware manufacturers adding support for virtualization, I expect the performance penalty for traditional virtualization to reduce significantly. At the same time, I expect that most application developers would prefer as much isolation from other tenants of virtualized hardware as possible - after all, it's hard enough trying to diagnose application crashes without adding the possibility of a kernel panic or OS crash triggered by an application you aren't even aware of. Given this, and the fact that a single kernel version does not suit everyone's needs (some developers care more about stability and will want an older kernel, while others may care a little less about stability and want enhancements in a newer version), I expect that in the days to come, the cloud will continue to be comprised of traditionally virtualized instances.

When all's said and done ...

The Open Source Bridge conference catered to developers and users of open source technology alike. The Open Source community is a hotbed of innovation, and several important new technologies are brewing there now, which will have broad-ranging impact on how software is written in the years to come. While there was a fair share of skepticism about the cloud in its current state, the Open Source community is already tackling some of the problems that are the hallmarks of the massively distributed systems that will form the cloud.

For other attendees' notes and insights, I recommend the #osbridge tag on Twitter.

Navneet Joneja
Sr Product Mgr, MObStor

Comments (0)

June 26, 2009

Hacking Up North : Winners of the Sunderland Hack Challenge

The format of Yahoo!'s university hack days are different in Europe than in the U.S. In the U.S., we visit for a week and give tech talks every day, after the normal day-to-day proceedings (once they are defined for each university environment). These sessions are followed by a 24-hour mad hacking session, presentations, judging, and prize giving.

In the UK, we arrive for a day, (re)introduce the idea of mashups and hacking, show off some of our technologies, and then ask the students to come up with an idea or problem to solve. After this phase of collaboration and support, much of the rest of the work is done at the university, by the students. Instead of being an add-on to the normal university curriculum, the hack project is part of the course deliverables.

This is why the Sunderland students didn't get 24 hours to hack but - (this varies from university to university) - weeks or months to build a full product, complete with architecture documentation, business plan, security and accessibility documentation, and all the other things a real pitch would involve.

Sunderland University, in the colder part of England (the North) hosted a hack challenge this year. I went up to Sunderland with Jamie Lockwood (who runs the U.S. university hack program) and Rajat Pandit. I was there as the ersatz "pitch jury" for the hack entries.

The overall quality of hacks was very high. We had a great time discussing next steps and where the students could go for support to take their projects to the next level.

After much deliberation and scoring, we decided on three projects that stood out, and crowned them the three hack challenge winners:

BTAC

Project Team:
Jill Brown, Steven Teal, Kenneth Ichenwo, Neil Proudfoot, Lynsey Slack, Philip Hewitt, Graeme Stephenson

Our app is a journey planner that compares routes based on mode of transport, i.e., bus, train, air, and car. Users get information about CO2 emissions, distance, cost, and duration (travel times). Our aim is to provide travelers with information to allow them to make informed decisions about their travel options. In particular we want to highlight the amount of CO2 emissions produced by transport.

MNSSA by T-10 Tornadoes

Project Team:
Daniel Morrison, Mark Stoddart, Emma Brown, Natalie Gibson, Daniel Barrow, Liam Coates, Constantinos Zezos

As this is a mobile app running on University servers there is no URL to play with. Instead we've created screencasts of the app (AVI movies).

MNSSA: Mobile Notification System for Student Announcements, improves the efficiency of communication via Sunspace (a Blackboard Learning System).The system distributes announcements to students (regarding their degree, modules, and general events) directly to mobile phone via an application they download using Yahoo! Go. Students no longer need to navigate Sunspace to remain up to date, and they have access to the desired announcements anytime and anywhere via their mobile's internet connection.

Yahooza

Project Team:
Jonathan Boucher, David Bradburn, Anthony Davison, Paul Robertson, Ilias Mantis, Craig Lee, Spiros Soulitios

URL:
http://my-yahooza-1.bpapps.com/

Our mobile application offers a simple design, yet gives the user a large array of transport information to sift through easily. This gives the user the ability to plan a journey ahead of time. The user can simply flick through our application to view whichever public transport data is uploaded onto the version they have bought and downloaded. Through our application, we have the potential to offer a system that's more comprehensive than anything available today.

The Sunderland University hack challenge was a great experience. We saw students with no previous web interface experience hit the ground running by building on top of YUI and Blueprint. This was inspiring. The most frequent request from students? They asked for detailed feedback about their pitches and presentations. Here's some of my feedback on pitching a hack -- thoughts I collected on the train journey south from Sutherland.

We're very much looking forward to repeating the challenge again next year and with more universities.

Chris Heilmann
Yahoo Developer Network

Comments (3)

June 24, 2009

ConvergeSC web event comes to South Carolina

Editor's note: This is a guest post from Gene Crawford, a web designer from Columbia, South Carolina, and a founding member and organizer of ConvergeSC.

The first-ever ConvergeSC conference takes place on Saturday, June 27, 2009, at Amoco Hall at the University of South Carolina, in Columbia, SC. Web designers, web developers, and marketing and sales professionals whose careers involve the web will converge at this one-day event to gain insight into all aspects of the web industry. We'll be joined some of the best and brightest in our industry, both local and from across the country.

ConvergeSC grew from the need among web-focused groups to build a more connected web community, which doesn't exist formally in South Carolina or the region beyond. We’re spread out, working hard with our heads down. ConvergeSC is an opportunity to educate and inspire each other to do better work. We hope you'll leave ConvergeSC with the mission to build or design something amazing for yourselves and for your clients. Whether you’re a hard core designer or a web or marketing summer intern, ConvergeSC has something for you.

We have some great speakers lined up, including two from the Yahoo! Developer Network: technology evangelist Jonathan LeBlanc and community manager Robyn Tippins. Jonathan will introduce the Yahoo! Query Language (YQL) and provide an overview of YDN; Robyn will show us a thing or two about connecting and engaging with your audience using social applications across the Web.

ConvergeSC is for: Web designers, web developers, business professionals, marketing professionals, content creators, and students.

Where: Amoco Hall, Swearingen Building, Department of Computer Science and Engineering, University of South Carolina, 301 Main Street, Columbia, S.C.

When: Saturday, June 27. Full schedule and background on our speakers.

To register online or for more information on sponsorships, speakers, and the agenda, visit www.convergesc.org.

Gene Crawford
Founding Member
Converge SC Conference

About ConvergeSC: ConvergeSC was founded by RefreshColumbia, Columbia Ruby Brigade (Cola.rb), Columbia Enterprise Developers Guild (CEDG), Columbia Adobe User Group (CAUG) and Columbia Linux Users Group (ColaLUG).
.
Follow us on Twitter @convergesc

Comments (0)

June 23, 2009

Research in the sun - developer evening in Barcelona

Last week the Yahoo! Developer Network went to Barcelona, Spain to host our first local developer evening. Around 40 developers showed up to hear about what Yahoo! offers for developers and to get expert answers to their questions.

Yahoo!'s presence in Barcelona is twofold: there's a sales and marketing office that's a sight to behold. The research office is a tad more cramped -- it shares space with university students and staff, but has an amazing aura of creativity.

Yahoo! Research Barcelona has produced some very cool prototypes -- they can be found in the Yahoo! Sandbox. We spent a great day chatting with the team about their current projects and how YDN could help take some of these solutions to the next level.

In the evening, developers from the community arrived and listened raptly to the presentation (which surprised me considering the gorgeous weather and other distractions). You can download the slides as a PDF (11mb) from S3 or watch them on Slideshare:

For detailed notes of the talk with code examples, please refer to this blog post.

I created two demos exclusively for this event. The first is a website about Barcelona. None of the content is maintained on the website - instead it is pulled from Wikipedia, Upcoming, Flickr, and Yahoo! Weather, with a few lines of PHP and quite a bit of YQL magic.

The other demo I showed uses YQL to do four different web searches in a single HTTP request, thus the page rendering does not slow down. This can be done both in PHP or in plain JavaScript and HTML.

We had a wonderful time in Barcelona. The movies about the city don't do it justice - even when they add Scarlett Johansson and Penélope Cruz. If you get a chance, come and see Barcelona - maybe when we're back for another event. ( Until then, you can check out our photos as proof.).

Chris Heilmann
Yahoo Developer Network

Comments (0)