Information retrieval on the web

Analysis

Why This Paper Matters

Published in ACM Computing Surveys in 2000, "Information retrieval on the web" by Mei Kobayashi and Koichi Takeda arrived at a pivotal moment: the Web was expanding exponentially, and search engines were becoming the primary gateway to online information. The paper matters because it systematically connects two narratives that were often discussed separately—the explosive growth of the Internet and the inadequacy of the era's retrieval tools. By presenting data from multiple sources on users, hosts, and Web sites, the authors establish that the scale problem was not speculative but already measurable, with trends pointing to continued exponential growth in the coming decade.

The survey also captures a critical user-facing tension. Although roughly 85% of surveyed Internet users reported using search engines and search services to find specific information, those same surveys showed widespread dissatisfaction. Slow retrieval speed, communication delays, and poor result quality—including noise and broken links—were commonly cited. This framing helped define the research agenda for Web information retrieval: not merely indexing more pages, but improving relevance, freshness, and responsiveness under severe scale and latency constraints.

For AI practitioners today, the paper is a useful historical baseline. It documents the problem space before modern ranking, crawling, and distributed retrieval architectures matured, and it highlights concerns—result quality, link integrity, and retrieval latency—that remain relevant in contemporary search and retrieval-augmented systems.

Technical Contributions

• Consolidated Internet growth data: The authors compile figures from several different sources on current and projected numbers of users, hosts, and Web sites, showing that despite variation in specific numbers, overall trends are consistent and point to exponential growth.
• User behavior and satisfaction evidence: The paper reports that about 85% of surveyed Internet users claim to use search engines and search services to find specific information, while also documenting dissatisfaction with current search engine performance.
• Problem taxonomy for Web IR: It identifies commonly cited problems including slow retrieval speed, communication delays, and poor quality of retrieved results such as noise and broken links.
• Review of emerging techniques: The survey discusses the development of new techniques targeted at resolving some of the problems associated with Web-based information retrieval.
• Future trends speculation: The authors offer speculation on future trends, providing direction for subsequent research and development.

Results

The paper's concrete findings are primarily survey-based rather than experimental. It reports that numerical figures on Internet growth vary across sources, but the overall trends cited are consistent and indicate exponential growth in the past and in the coming decade. It also reports that about 85% of Internet users surveyed claim using search engines and search services to find specific information. However, the same surveys show users are not satisfied with the performance of the current generation of search engines, with slow retrieval speed, communication delays, and poor quality of retrieved results—such as noise and broken links—commonly cited as problems. The paper does not present benchmark comparisons or system-level metrics of its own.

Significance

The broader impact of this survey lies in its clear articulation of the Web IR problem at scale. By linking exponential Internet growth to user reliance on search and to documented dissatisfaction, the paper helped justify research investment in scalable crawling, indexing, ranking, and result-quality improvement. Its emphasis on noise, broken links, and latency anticipated enduring challenges in information retrieval. For the AI field, the paper serves as an early reference point for the shift from traditional information retrieval to Web-scale search, influencing how subsequent work framed evaluation and system design. With 733 citations, it remains a widely referenced survey for understanding the foundations and challenges of Web information retrieval at the turn of the millennium.