RDBMS support helps Blobs find their niche - use of binary large objects in relational database management systems

Software Magazine, Oct, 1993 by Barbara Francett

Vendors expanding large object support; storage, space allocation remain issues

At the European Centre for Medium-range Weather Forecasts (ECMWF), large object technology is helping scientists there determine whether global warming is fact or fiction.

The ECMWF in Reading, England, is conducting a long-range research project to study global atmospheric changes. The project requires reanalyzing 15 years of meterological data -- including pressure, temperature and cloud data, as well as ground, weather balloon and satellite observations.

The project's goal is to develop a data set representing the global atmosphere from 1979 to 1993. Then scientists will subject the data to a consistent set of analysis techniques. This will help meteorological and oceanographic researchers around the world perform many studies. For instance, the data will help determine whether global warming is a reality, or the result of applying inconsistent analysis techniques over the last 15 years.

By the project's completion, scheduled for year-end 1995, the ECMWF will have produced nearly two terabytes of data, according to Rex Gibson, project manager. "Management of the data is critical," he said.

To manage the data, the ECMWF is using the Empress relational database management system (RDBMS) from Empress Software, Inc., Greenbelt, Md. "We use standard database tables to hold descriptions that identify the observational data. The binary forms are accommodated using the database's Blob [binary large object] feature," he said. Using a single SQL command, scientists can then extract the observations stored in Blobs for analysis. "In this way, we can handle the data efficiently," Gibson said.

The ECMWF stores the binary data in a standard form for observational data, which the World Meteorological Organization (WMO) defined. In this form, the Centre can exchange data with any meteorological group in the world. ECMWF maintains the bulk data on servers from Silicon Graphics, Inc. (SGI), Mountain View, Calif., while the analyses run on SGI workstations.

Large object technology is probably one of the least-understood and least-used RDBMS features. Yet all the major RDBMS vendors can now or will soon support large objects.

Developers are putting large objects to use in a variety of niche applications, including imaging, multimedia, geographical and scientific research. Today's pioneer implementations are paving the way for more widespread applications over the next 10 years.

"Large objects" or "Blobs" (which, despite the acronym, are not always binary in nature) are catchall terms for unstructured data types without a fixed length. These data types can be images, video, sound and even very large pieces of text. Large objects can vary enormously in size, from several kilobytes to a gigabyte or more.

The Informix relational DBMS from Informix Software, Inc., Menlo Park, Calif., was an early user of Blob technology. The company began research and development work in the mid-1980s and unveiled the system incorporating Blob technology in 1989.

"Blobs are becoming commonplace," acknowledged Ram Srinivasan, product line manager for servers and connectivity at Informix. However, he contended that the Informix development over the years has allowed the technology to overcome "inherent problems" involving logging and security issues.

Several trends drove the need for large objects in relational databases. First, organizations' determination to reduce paper in the early 1980s led to the electronic file cabinet; that is, the replacement of paper documents with electronic images of those documents. The next step was to combine the massive amounts of unstructured data with the databases that would access them, rather than maintain two separate systems. As a result, RDBMS vendors added large object support to their products.

This, in turn, has led to the challenge of handling unstructured data types in the same way as traditional alphanumeric data types -- with the same integrity, consistency, logging and security constraints. Most relational databases treat large objects as huge "bit buckets." The databases ignore what is in the "buckets" and deal with them as they would anything else in the database.

PRIMITIVE STAGE

Large objects are in many ways inconsistent with RDBMS philosophy, however. As a result, Blob support from most RDBMS vendors tends to be limited. The relational model is based on tables made of rows and columns. This model is perfectly suited to structured, alphanumeric data, but not geared for the unstructured data found in large objects.

"Relational data types are optimized for small data," said Malcolm Colton, director of server product marketing at Sybase, Inc., Emeryville, Calif. "With Blobs, a single element could be as much as 2Gb in size. Most databases are not that big."

Consequently, most RDBMSs can only store and retrieve large objects. Any more than that is usually left up to the application to handle, which requires a great deal of coding, or a dedicated subsystem, as many im-aging implementations use.