IDL 6.3.1--great for interactive analysis

NASA Tech Briefs, May 2000 by Ross, Steven S

Does your organization have piles of data that need to be crunched? If the data set is static, there are some great purely statistical packages to do the job - Statistica, SAS, and SPSS among them. If you don't need the slats, but do need nice graphics, Excel or a similar spreadsheet might fill the bill - especially if augmented by an add-on program such as Harvard Chart XL.

But what if your data set is constantly updated, or you want to send data out to many project team members to play with? What if you need to do the slats and the graphics as well? Then IDL (Interactive Data Language) from Research Systems, Boulder, CO, might be just the thing for you. There's an added bonus. The basic package comes with everything you need to create standalone modules for exploring data, even within other applications such as Word or Excel. IDL is an OLE/COM server and comes with an interface builder's kit. The latest version, 5.3.1, even has ActiveX controls for viewing IDL-generated analyses over a network.

This is programming, though. Is the learning curve worth it? Many will think it is. It is fairly easy to come up to speed with IDL, at least for straightforward tasks. But the extra niceties do take some getting used to. For example, to process really huge data sets - upwards of a million cells - faster, you can go to wavelet processing or you might consider tweaking the system, perhaps to do everything in integer math instead of 64-- bit floating point.

Like MATLAB from The MathWorks, IDL is particularly good at processing arrays - tables and tables of data. The language is compact (a few lines of IDL code can replace hundreds of lines of C or FORTRAN). Users of these languages tend to rely on huge subroutine libraries, but even when such libraries are available, the writing and debugging process can be painful. With IDL, signal processing and image processing can be done almost painlessly, once you learn the basic syntax.

Here's an example: Applying a lowpass filter to the noisy data allows low-frequency components to remain unchanged while high-frequencies are smoothed or attenuated. You can construct a filter function by entering step-by-step commands.

To filter data in the frequency domain, we multiply the Fourier transform of the data by the frequency response of a filter and then apply an inverse Fourier transform to return the data to the spatial domain.

The results can be output graphically, either as static images or as animations. There's good OpenGL graphics support, assuring portability of the programs you write in IDL among various OpenGL platforms.

Existing FORTRAN and C routines can be linked into dynamically. Alternatively, you can write C and FORTRAN programs to call IDL routines as a subroutine library or display "engine."

IDL has been an evolutionary product, with frequent upgrades. Version 5.3 offers faster operation and many enhanced features, including some new commands. Some of the biggest improvements came in speed and in 3D visualization. Often, this has come about because Research Systems has been rewriting subroutines that were written in other languages, into IDL itself as new commands become available.

IDL can read CDF (Common Data Format), HDF (Hierarchical Data Format), HDF-EOS (Earth Observing System extensions to HDF) and netCDF (Network Common Data Format). It can also handle numerous graphics formats including BMP, PNG, GEO TIFF, PPM, SRF, Interfile, JPEG, GIF, and PICT. It can even read DXF (a CAD format) and WAV audio files.

There are versions for Windows, UNDO, VMS, and Macintosh. We reviewed in the Windows NT 4.0 version. The trick to producing portable IDL applications - ones that run on all platforms - is to avoid platform-specific features. Research Systems lists all the trouble spots. If you have, say, the Windows 95/98 version, you can move files across to a Macintosh and use the Mac version to compile the program. Pricing ranges from $1,895 for a singleuser PC or PowerMac license, to $3,895 for a single-user, floating network workstation license.

A full install requires roughly 140 MB of hard drive space in Windows. The software ran well on a 200-MHz Pentium Pro with 64 MB of RAM. You can download a trial version (it won't save or print files, and it runs for only seven minutes at a time) from www.researchsystems.com. The download is about 80 MB. If you lack a fast connection to the Internet, ask for a demo CD.

Steven S. Ross teaches journalism at Columbia University in New York City. He has authored three commercial software pack ages, including a units conversion program and an engineering calculations program.