Diving Into the Deep Web - Company Business and Marketing

Industry Standard, The, Sept 11, 2000 by Elinor Abreu

Companies are hunting for buried treasure in online databases inaccessible to conventional search engines.

CHRIS SHERMAN NO LONGER CALLS the airline to see if his flight is on time before he heads to the airport for business trips. Instead he checks TheTrip.com's online Flight Tracker, which lets him search a constantly updated Federal Aviation Administration database.

"The database maintains real-time information about all planes in flight in the U.S.," says Sherman, Websearch expert for About.com and president of Los Angeles-based consultancy Searchwise.net. "It's information from the cockpit."

Although TheTrip and Flight Tracker are indexed by major search engines, the data inside the tracker database isn't. That information is inaccessible unless you know how to find it. "AltaVista and Google, no matter how good they are, they don't have a prayer of accessing real-time information like that," Sherman says.

The FAA database is part of the invisible Web, sometimes called the "deep Web" -- a vast repository of information hidden in databases that general-purpose search engines don't reach. While much of the information is obscure and useful to very few people, there's plenty of data on the deep Web that people would want, and pay for, if they knew it was available. That includes Securities and Exchange Commission filings, yellow pages, IBM's patent database, the Merriam-Webster Dictionary and Kelly Blue Book information on automobiles.

Not only is the data in the deep Web potentially valuable, it's also multiplying faster than data found on the surface Web. A study released in July by search company BrightPlanet estimates that the deep Web may contain almost 550 billion documents -- far more than the 1.2 billion pages that search engine Google has identified on the surface Web, not to mention the 600 million pages or so that Google is capable of searching.

For companies sitting on potentially valuable information, tapping the deep Web is a market opportunity.

"This is Web traffic that could be worth thousands, if not hundreds of thousands, of dollars over the course of a year," says Danny Sullivan, editor of Search Engine Watch.

Though traditional search engines point to the front doors of databases, they can't easily index information inside them. Many of the databases offer specialized searches. The surface Web, Sullivan points out, is akin to a bookstore or a library in which the search engine knows all the books on the shelves. But if you want to find a book that isn't in stock, or to mine data that's in the books, you need a more powerful tool.

Privately held BrightPlanet is among a number of firms that have sprung up in the past two years to provide search capability for the deep Web. BrightPlanet general manager Thane Paulsen thinks companies will be willing to pay to have their database information mined. Founded in 1999 in Sioux Falls, S.D., BrightPlanet builds search sites for vertical portals and b-to-b sites and licenses its software for firms to use on their intranets.

BrightPlanet also sells a desktop tool called LexiBot, which simultaneously searches multiple databases and other sources. lexiBot competes with IntelliSeek's free Bulls Eye software. Meanwhile, database aggregator Lexis-Nexis and search engine Northern Light offer access to specialized information for a fee, while a growing number of sites offer lists of databases by topic for individual searches.

Big search companies like AltaVista and Google don't see a future in trolling the deep Web. "There is a lot of information in databases that's not that useful, like really large databases of data from radio telescopes," says Google CEO Larry Page. "And it costs us money to index things."

The cost of indexing, however, is falling. And companies like Inktomi, which recently signed a major content distribution deal with partners including AOL, see dollars in the deep water.

"This is a focus of ours at the moment," says Matthew Hall, VP of engineering for Inktomi. "This is where we believe search is going."