Powered By Dell Servers, World's Biggest Internet Search Engine Goes Online

EDP Weekly's IT Monitor, August 9, 1999

New FAST Search Index Has More Than 200 Million Web Pages

Fast Search & Transfer, a developer of search technology for the Internet, has just launched FAST Search, the world's biggest search engine (www.alltheweb.com).FAST claims to be the first company to create a search engine with the ultimate goal of searching "All The Web, All The Time."The search engine is the result of more than a decade of research into optimizing search algorithms and architectures, a project originally initiated at the Norwegian Institute of Technology in Trondheim.

FAST Search already includes more than 200 million unique URLs in its database, making its catalog almost twice the size of search engine reseller Inktomi and three to four times bigger than several popular search engines.FAST intends to resell FAST Search to major portals, search engines, ISPs, and content sites.

According to a study published in the July issue of Nature magazine, search engine coverage of the Web has decreased substantially since December 1997, with no engine indexing more than 16 percent of the Web (estimated at 800 million URLs as of July).

At 200 million URLs, FAST Search indexes more of the Web than any other search engine, in one year's time, says FAST, the engine is expected to catalog the entire Internet -- and then to keep up with growth from that point forward.

The FAST Search architecture is based on high-performance Dell PowerEdge servers and Dell PowerVault storage subsystems.The systems operate in parallel with each other to distribute user queries, search the document catalog, and spider the Web through dedicated computing and storage nodes.

A key design objective of the FAST Search architecture is to scale linearly in both query volume (number of searches) and catalog size (number of documents), enabling search to become an "organic application" that effectively keeps up with the growth of the Web.The "parallel server" approach differs markedly from the architectures of other major search engines, says FAST, which typically use a small number of very large, expensive multiprocessor computers.

The parallel FAST Search architecture also delivers fast search speeds -- a typical query on FAST Search races through all 200 million documents in less than one second.And by using many systems working in parallel, FAST Search builds its document index in only 12 hours, a task that takes many search engines several days or weeks to perform.

"The announcement by FAST sets a new benchmark in search engine size and coverage.Those looking for the ability to comprehensively search across the web should definitely consider trying All The Web," says Danny Sullivan, editor of Internet.com's SearchEngineWatch.com, a site devoted to search engines.