« New! User Product Review for Yahoo! Shopping | Main | Announcing New Maps APIs »
October 5, 2005
Several people have asked us about compressing the XML we return from Yahoo! Search Web Services. We'd been reluctant to do it because some browsers and platforms claim to support compression without actually doing so. However, we can now return compressed data to clients that support compression while working well with systems that don't support it so well.
If you want the data we return to be compressed, do the following:
1) Use HTTP 1.1.
2) Pass in an 'Accept-Encoding: gzip' header.
3) Spoof your user-agent to be Mozilla/5.0 (or higher).
Don't forget to do the expansion on the other end!
Will this save you time? Quite possibly. XML compresses nicely, and savings of 50% on the data size are not uncommon. Depending on how busy your processor is, less data over the network may be fast enough to offset the cost of decompression.
Posted at October 5, 2005 4:33 PM
Why do we need to spoof the user agent? I know that compression is broken in some versions of IE, but why not just not serve gzipped content for the known-bad agents?
The best documentation I'm aware of for which browser works with gzip compression is in the mod_gzip docs: http://www.schroepl.net/projekte/mod_gzip/browser.htm
As you can see it isn't just Mozilla/5.0 that works, and that document ignores the user-agent strings passed by popular libraries (eg, Java, Perl, Python etc) which is what I'd expect most of the traffic for this service to be.
I realize you are trying not to break libraries, but as an implementor I hate to have to put hacks like this in.
(BTW, when you say "Spoof your user-agent to be Mozilla/5.0", can "Mozilla/5.0" go anywhere in the user-agent string?"
Posted by: Nick Lothian at October 6, 2005 9:19 PM
Hi,
Is it possible to provide the Yahoo Job Search Engine(Hotjob) API in the future?
Posted by: Tien-Chih Wang at November 3, 2005 1:00 PM
Copyright © 2008 Yahoo! Inc. All rights reserved.
Privacy Policy - Terms of Service - Copyright Policy - Job Openings