April 30, 2009

Using Yahoo! Pipes for Online Monitoring

Editor's note: This is a guest post from Dawn M. Foster, a consultant, community manager, event organizer, blogger, podcaster, technology enthusiast, and business professional in Portland, OR. Dawn provides consulting services for companies wanting to engage with online communities and has more than 13 years of experience in business and technology.

I am a bit of a fanatic when it comes to monitoring what people are saying and filtering it to find the most relevant information. My tool of choice for this activity is Yahoo Pipes.

I use Yahoo Pipes for my many projects and build Pipes for my clients to help find what people are saying about us, our industry, our competitors, and more, through smart filtering of blogs, news sources, Twitter, and other online sites. This approach has a number of uses both for businesses and individuals:

• Become more responsive by knowing when and where people are talking about you, your company and your products -- on blogs and Twitter.
• Use what people are saying about your company and your products to improve your products, marketing messages, web content, documentation, and other communications.
• Get insight into your competitors and keep up with important information about your industry.
• Tailor your online research to your specific needs and interest areas.

One technique that I use for monitoring and filtering takes a series of keywords from a CSV file and uses Yahoo Pipes to run each keyword through various searches. This technique can be very powerful when you search multiple services. I have a complex Yahoo Pipe that searches a dozen sources (Twitter, blogs, video, images, etc.) in a single pipe with one keyword list to maintain. In the example for this post, I've simplified the process to make it easier to understand the fundamentals of the technique by running the list of keywords only through Twitter search.

You can watch the screencast to see exactly how I created the demo pipe, and below the screencast, you can find links to the demo pipe and more information about the modules used in this example.

2 Minute Screencast Demo:

You can watch the embedded video above, but I recommend downloading the higher resolution Quicktime file (22 MB) or clicking the full screen toggle icon in the player to watch. The quality will be much better than the flash version embedded above.

Technical Details and More Information:

• The Demo Pipe. A copy of the Keyword CSV demo pipe - click "View Source" to see the modules.
• Fetch CSV Module. Enter the URL of the CSV file (make sure that it already exists in a publicly accessible location) along with information about column names and separation characters as needed.
• Loop module with URL Builder. Loops through each element in the CSV file and builds a search URL formatted for RSS output. In this case, we used Twitter search.
• Loop module with Fetch Feed. Loops through each URL built in the previous step and fetches the feed associated with the item.
• Sort Module. Sort by date in descending order to make sure things are sorted in a logical manner.
• Pipe Output. The final module in every Yahoo Pipe.

I have an entire series of free 2-minute Yahoo Pipes videos. I'm teaching a 2-hour Introduction to Yahoo Pipes course in Portland, OR, on May 7th if you want to learn more about Yahoo Pipes. You can also find me this weekend at Barcamp Portland, if you want to learn more about Fast Wonder and Yahoo Pipes.

Dawn Foster
Fast Wonder

Posted at April 30, 2009 7:22 PM | Permalink

Bookmark this on Delicious

Comments

Posted by: Accura Cast at May 1, 2009 12:00 PM