Business intelligence integration: extending the information net - Storage Networking

Computer Technology Review, Dec, 2002 by Jim Kanzler

Business intelligence may mean different things to different people. Consequently, it is useful to have a common working definition of this sometimes-misunderstood concept. Business intelligence (BI) includes software applications, technologies and analytical methodologies that perform data analysis. BI (also known as decision support systems) includes data mining, Web mining, text mining, reporting and querying, OLAP, and data visualization.

Taking BI to the next level is the new concept of business intelligence integration (BII). Perhaps the best way to illustrate BII is to equate it to commercial fishing. Now, think of a huge fishing net as your company's data warehouse. It has a wealthy catch of information related to your business. But there are also other fish that escape your net's grasp, bits and pieces of valuable information residing outside of the warehouse that are critical to your company's success. These adhoc data sources include information contained in ASCII files, Excel spreadsheets, Access databases, legacy systems (that didn't quite make the data warehouse list), departmental SQL servers and dozens of other disparate data sources including "finished" reports.

Five Classes of Ad-Hoc Data Sources

One-Time Data: Information that is too small or "out of place" for the data warehouse -- it is a minute amount of data, such as a list, that is used for a specific application, typically an Excel spreadsheet.

Report Results: Quite often the result of an application--a report, for instance--is an ad-hoc data source. The report is interesting, but there is a need to incorporate more information into it. With these reports it is not always possible or desirable to change the sourcing application. The report itself then becomes an ad-hoc data source. We sometimes refer to this as the "morphing report," and it typically occurs when the analysis is iterative, dynamic and complicated, or when passing intermediate results from one analyst to another.

Pending Warehouse Data: Information that will be part of the data warehouse, but the planning for its inclusion has not been completed, or the financial resources are not immediately available.

Legacy System Data: Information that resides in a legacy system and is serving a useful purpose but will never be part of the data warehouse, because the cost or effort to put the information into the warehouse is simply not justifiable based on its perceived value.

External Data: The tremendous amount of data that we are exchanging with our customers and vendors are representative of the data explosion that we have heard so much about. These ad-hoc data sources rarely get inserted into the data warehouse.

Certainly, there are business intelligence tools that--with varying degrees of difficulty, time, expense and resources--are capable of accessing these ad-hoc data sources and potentially integrating them into the information residing in the data warehouse. The question is how quickly, how effectively and at what cost these tools can perform this function. The answer is, frankly, not very quickly, not very effectively and often at significant cost.

This problem becomes magnified when you consider the rate at which these ad-hoc data sources are appearing, an outgrowth of the data explosion. When we typically think of the data explosion we tend to think about our databases growing by the number of rows or tables being added. But in reality the data explosion really means the ad-hoc data explosion. We need to pay more attention to this "new" kind of information age and to have tools available to deal with this reality.

As data warehouses were created, companies integrated their pertinent data into these warehouses. However, the number of data sources began to increase, and it became impractical, as well as expensive, to integrate all of the data from these ad-hoc sources into the data warehouse. Perhaps the information from disparate sources will eventually present itself to the point where it is practical to enter it into the data warehouse. The problem is that the cost and effort required to enter every piece of information into the warehouse begins to show diminishing returns. It's clear that companies will take the time and should enter the majority of their data into their warehouse. It's another matter to expect that every data source will be incorporated in a timely manner as new data emerges.

Enter BII

When you have access to and can incorporate and combine all of your data, then you have truly arrived at a comprehensive solution--BII. The theory behind BII is to recognize that there is a problem and address it with a new class of tools, focusing the technology of business intelligence to incorporate ad-hoc data source as seamlessly and effectively as possible.

If we look historically at the development of computer systems, we can draw correlations between the past and BII. We see that at first a concept or methodology was developed. That was shortly followed by a complicated coding mechanism that only engineers could understand or use. From there we generated tools to help with the coding and finally we developed visual technologies to solve the problem.