How the search engine killed the web directory

In order to understand why links are so important, it may help to understand the problem thatsearch engines use links to solve. It's hard now for many of us to imagine the world without a World Wide Web, but by the end of 1993 there were only around 600 websites and most of those 600 sites were pretty thin; a handful of pages each. In August of that year, when there were just a couple of hundred websites, a well-known computer book publisher O'Reilly and Associates launched what was probably the first commercial directory of the Internet, GNN, the Global Network Navigator.

Much of the directory was based on the whole Internet catalog. In effect, it was like a paper directory of websites posted on the web. In September, another directory appeared on the scene, W3 Catalog. And in January of 1994, when there were around 600 websites, David and Jerry's Guide to the World Wide Web launched. You'll know that directory better by its later name Yahoo! Yahoo! eventually became the world's top search directory and the world's most popular site. These were all web directories.

In other words, they were list of websites with a little information explaining what each list of site contained. That was fine in the early days when the web simply didn't contain much information. But as the web grew, they became unwieldy. If you're trying to find a particular Shakespeare sonnet, a directory will tell you which sites might contain the information, but they don't let you see what's in each site. They simply tell you what each site is about and it's up to you to go to the site to see if it contains what you're looking for.

So the next step was the search engine, a system that created an index of pages within sites.There were various simple search engines early on, but perhaps the first true web searchengine, a system that would allow use of the search through the text contained in web pageswithin index websites, was WebCrawler launched in April 1994. A directory provides minimal information about a site. A search engine, though, lets user search pages within sites, a far more useful service.

During 1995 and over the next few years, all sorts of other search engines appeared on the scene; Magellan, Excite, Infoseek, HotBot, Northern Light, and of course AltaVista whichbecame hugely popular when it launched late in 1995. Finally, in 1998, Google appeared on the scene. By the end of the decade, the writing was on the wall, search engines were the future and over time search engines would more or less kill off the directories. Even Yahoo! had to switch.

In the year 2000, Yahoo! began using Google's index to provide search results to Yahoo!searches, then gradually pushed the directory further down the page until they removed itfrom their homepage entirely. Yahoo! Directory still exists today of course, though most users have no idea where. But the search engines have problems of their own. As the web grew to around two to three million websites and hundreds of millions of web pages by the time Google appeared on the scene, the problem of sorting through the starry amount of data was becoming overwhelming.

Google was based on a revolutionary idea that you could figure out what a web page was about and whether it was a good match for someone's search query, not just by looking at the page itself but also by looking at links pointing to that page. Those links could be inside the website within which the page was found, but could even be pointing to the page from other websites; sites that the owner of the reference web page might not even know existed. And that's what we'll look at in the next video.