Old MacDonald didn't have IT: web farming in the info age

Information Outlook, Dec, 1998 by Eleanor C. Fye

Catching the Winds of Change

As we sail into the twenty-first century, "Integration" should be engraved on all our mastheads. Of all the concepts proffered as key trends of our electro-life, this one is perhaps most vital for special librarians. Efficient e-commerce depends on coordination and integration of the information systems in place throughout organizations. As special librarians, we must decide where we belong - out of the loop or swingin' the hoop.

If it is not our sails that catch the winds of our destinies, it will be our capes. The documented competencies of special librarians indeed suggest red and blue spandex suits. Our positions in business are unique and uniquely diverse. We enable visions for organizational change as well as attend to typographical details. Our experience in evaluating information sources, our specialized subject knowledge, our ability to select and network relevant information products, and our adeptness at communication and instruction provide a platform, as many have discovered, to interact across company divisions in creative ways. This ability to build bridges, to team, to "look for partnerships and alliances," makes us hot properties as paradigms and millenia shift.

"IT' People

Arguably, the most important place to seek partnership and alliance is the friendly neighborhood Information Technology (IT) department. Already, the special librarian functions, in the words of the SLA Competencies document, as "a technology application leader who works with other members of the information management team to design and evaluate systems for information access that meet user needs." Many already are diving headlong into Intranet development and are learning about data warehousing, data mining, and related concepts.

Another niche in IT just waiting for special librarians is web farming. One of the latest, hottest technological issues related to data warehousing, web farming provides a perfect opportunity for special librarians to team with IT and upper management, to maximize the value of current skills, and to add value to current library services.

Data Warehousing and Mining

Before plowing on, it will be helpful to put a couple of terms on the table.

Data warehousing (DW) grows out of the need to view corporate operations data as assets to be mined for information relevant to business growth, especially in developing increasingly customized (or "customer-ized") services. DW involves systematizing the collection and integration of enterprise-wide transaction data, unifying its disparate formats, and making it the basis for knowledge management and decision-support system (DSS) development.

For example, a retailer collects customer zip codes at point of sale. This information is then transferred to a data warehouse; there it is queried and reports are generated that support planning for new store locations. Because data warehouses are generally vast in coverage, a recent trend in this field is the use of artificial intelligence (or intelligent software agents) to alleviate size and time burdens of extracting useful information from data warehouses.

Data mining is the process of uncovering relationships in this data structure to reveal new information relevant to decision-making.

I Never Meta Data I Didn't Like

Web farming is based on the idea that information external to the organization is as valuable to strategic decisions and planning as information collected from internal transaction-based systems. Web farming ideally augments an existing data warehouse, but it is also valuable in contexts that do not include DW. In fact, much of this process is already part of our job descriptions. As stated in the SLA Competencies document, the special librarian "identifies, retrieves, organizes, repackages, and presents information in an actionable form."

In "Reaping the Web," IT Consultant Richard Hackathorn states that web farming "has the objective of refining web content in a systematic manner - refining this content involves the processes of discovering, acquiring, structuring, and disseminating" information. You've done this for years, right?

Goals of Web Farming

According to Hackathorn, the goals of web farming are:

* To discover web content that is highly relevant to the business

* To acquire that content so it is properly validated within a historical context

* To structure the content into a useful form that's compatible with the data warehouse

* To disseminate the content to the proper people so it has direct and positive impacts on specific business processes

* To manage the previous steps in a systematic manner as part of the production operations of a data center environment (see "Reaping the Web").

What makes web farming substantively different from the special librarian's function is that it takes place within a data environment with the goal of developing a production system, usually within the context of data warehousing, to process web queries automatically and systematically.

However, according to Hackathorn (in an article titled Farming the Web), at the first level, web farming simply requires a "highly skilled business analyst" who is able to evaluate and prioritize outside information sources according to their relevancy to business.