The Web In 1996

Wired Magazine

I was looking through some old stacks of Wired Magazine when I came across the May 1996 issue. It featured Jerry Yang and David Filo (Yahoo!) on the cover with a great article about search engines - Seek and Ye Shall Find (Maybe). I was struck by how much the web has changed in the last 14 years.

Remember when Lycos, Excite, and Alta Vista were some of the mainstream search engines? Consider that in 1996, Wired reported that:

  • Yahoo! listed more than 200,000 websites with 20,000 different categories
  • 800,000 people a day used Yahoo! to search
  • there were approximately 500,00 searchable websites with 30-50 million web pages
  • the entire web contained 200-330 gigabytes of text

Today’s laptops could easily store a text version of the web from 1996 but the size of the web was growing by 20 percent a month. There was concern that the search engines wouldn’t be able to keep up with the growth:

In two years, as the Web surpasses the roughly 29 terabytes in the current Library of Congress, will the inverted index become too large to feasibly store? Will it simply take too long to compute? Or will attempts at indexing the Web break down in some other, unexpected way?

The Web in 2010

In 2010, Jerry Yang and David Filo are dumping Yahoo! shares after Yang botched a deal with Microsoft who wanted to buy Yahoo! Today, Goggle owns search and Yahoo! struggles to survive but far more interesting is the enormous size of the web:

  • The first Google index had 26 million pages in 1998
  • By 2000 the Google index reached a billion pages
  • On July 25, 2008, Google announced that it had discovered one trillion unique URLs on the web
  • The web contains at least 19.59 billion pages (as of Feb 15, 2010)

At what point does indexing the web break down and fail? Is this even possible with the insane number of data centers that Google owns and maintains?

Posted in Web at 5:03 PM