AllTheWeb: THE SEARCH ENGINE THAT CAN - AND MORE [Review] - Evaluation

Online Newsletter, March, 2002

[Review]

AllTheWeb: THE SEARCH ENGINE THAT CAN - AND MORE

Do professional online users utilize the many search engines that abound on the Internet? - Yes, of course they do.

But the sad fact of the matter is that only a few search engines are worthy of note because they are either limited in their search function capabilities or are woefully out-of-date in updating links to Web sites. In addition, as we have also previously stated in our newsletters, no previous single search engine covers the entire Web - only about 17%.

More frustration with search engines emerged last month when Northern Light was acquired by divine, inc. and went from "free" to "fee". Northern Light admitted, as so many already know, that advertising alone cannot support a free public search engine. Rather, most successful search engines are made available to the public as a showcase of the search engine itself which is available for sale or lease by the owner.

However, a bright star for online users has emerged with AllTheWeb (http://www.alltheweb.com), which was upgraded and enhanced last November (2001). This search engine, in this reviewer's opinion, surpasses any other public search engine available on the Internet today.

Background

AllTheWeb was developed beginning in 1998, and is owned by Fast Search & Transfer ASA (FAST), a Norwegian (Oslo) Internet software developer, with a Boston-based subsidiary FAST, Inc. - The company began in 1997 as Fast Internet Transfer, founded by scientists from Norway's leading technical university, The Norwegian Institute of Technology, and is a spin-off of Opticom ASA. Both Dell and Lycos have made strategic investments in FAST. - Initially, AllTheWeb, was dubbed under a variety of names, but the current name, AllTheWeb, has stuck.

In 1999, the company predicted that the AllTheWeb search engine was expected to cover -all- of the Internet's more than 200 million URLs. (More than any other single search engine.)

A key design objective behind the AllTheWeb search engine is to scale linearly in both query volume (number of searches) and catalog size (number of documents), enabling [a] search to become an "organic application" that effectively keeps up with the growth of the Web. This "parallel server" approach differs markedly from the architecture of other search engines, which typically use a small number of very large expensive multiprocessor computers. These large systems do not cost-effectively scale with the growth of the Web, and as catalog sizes increase, they are further slowed by increased processing overhead and inefficiencies.

AllTheWeb architecture is based exclusively on high-performance Dell (r) PowerEdge (r) servers and Dell PowerVault (r) storage subsystems. These systems operate in parallel with each other to distribute user queries, search the document catalog, and spider the Web through dedicated computing and storage nodes.

The parallel search engine architecture also delivers exceptional search speed - a typical query races through all 200 million documents in less than 1 second. In addition, by using many systems working in parallel, it builds its document index in only 12 hours - a task that takes many search engines several days or even weeks to perform.

Features

* News stories from more than 3,000 online sources, indexing up to 800 news stories per minute.

* The entire AllTheWeb catalog is updated every 9-11 days as compared to 28 days for Google, and 45-60 days for AltaVista.

* Some search engines only index the title of stories from one or two news feeds. AllTheWeb constantly scours the Web for fresh news articles from thousands of sites.

* AllTheWeb currently contains an index of over 600 million Web pages, 115 million multimedia files, 100 million FTP files, 2 million MP3 songs, and support for 48 languages.

* The AllTheWeb search engine spiders over 1.5 billion Web pages with over 575 million full-text indexed pages.

Conclusion

What a pleasure to use a -real- search engine and get up-to-date results!

With a minimum of advertising, and relatively few cookies, this is the #1 search engine of choice!

The main page allows users to quickly select searches from the Web, News, Pictures, Videos, MP3 Songs, and FTP Files.

Both basic and advanced search options are available.

This reviewer tested several Web sites (including our own and several others created by this reviewer). Unlike other search engines we have used, AllTheWeb returned links to virtually every Web page, except those that were not indexed in a directory on the Web site. - Furthermore, the pages were up-to-date. (Some search engines have not updated our Web site in over 6 months.)

Next we asked AllTheWeb to search several news stories. - One of these at the time was the FallBrook, California fire. This resulted in many very relevant news stories, some of them having been indexed only minutes earlier. -- Ditto on a breaking news story on Domino's Pizza. We had only a hint that the story was breaking and even local news sources such as newspaper and radio and TV Web sites had nothing posted about the story. AllTheWeb posted the news story from 'The Detroit Free Press' which was less than 6 hours after it had been indexed by the search engine. -- Now that's fast! (The next day's local newspaper only published a small paragraph of the original story, making it rather meaningless.)