Technology Monitor

Although the new method will not be directly visible to consumers, it means that the company now updates its entire web index constantly and incrementally, rather than trawling the web over periods of about two weeks to produce an entire new index. Tom Stocky, Google’s Director of Product Management, said that it paved the way for a host of new innovative services.

When web users enter an internet search into Google, the company compares their queries against its index of the entire web; previously the index had a series of layers that were updated more or less regularly; now it differentiates more effectively between different types of content, and is updated more often. This new method provides “50 per cent fresher results”, Google claims. Mr Stocky called it a “fundamental change” that would allow Google to cope with real-time information from new web phenomena such as Twitter and Facebook.

The company blog said that “every second Caffeine processes hundreds of thousands of pages in parallel. If this were a pile of paper it would grow three miles taller every second. Caffeine takes up nearly 100 million gigabytes of storage in one database and adds new information at a rate of hundreds of thousands of gigabytes per day.”