use of sound for data exploration, The

Bulletin of the American Society for Information Science, Jun/Jul 2000 by Gluck, Myke

When presented with the feasibility of talking movies Jack Warner, early head of Warner Brothers Studios, is quoted as saying, "Who the hell wants to hear actors talk`?" Unfortunately today, many data analysts might be overheard to make similar remarks regarding the use of sound for data analysis. Our work at Florida State University has begun to exploit sound in conjunction with other multimedia support, such as visualization and cartography, to aid data analysts in performing spatial data mining, exploratory data analysis and pattern detection.

Exploratory Data Analysis

Data mining has many descriptions that are useful for our purposes. These include the reuse of data, multiple uses of data and analysis of patterns in large datasets. Data is expensive to collect and often is used once and discarded or archived without review. Data mining suggests that any given dataset may contain valuable information that can be reused for reoccurring analyses or the study of effects over time. Also, the same data can occasionally be reused for purposes for which it was not initially collected yielding new insights at low cost. Data mining may also suggest methods for exploration of data for new insights to the original data related to the original purpose for which the data was collected. Examples include weather data, satellite observation data, aerial photography, U.S. transportation data, U.S. census and county data as well as sampled files of audio and video.

Data mining is actually a subset of a broader range of techniques called Exploratory Data Analysis (EDA) used to seek possibilities of patterns and effects. Often EDA techniques are used to generate hypotheses that more traditional methods may then be used to confirm. EDA often purposely overlooks the known patterns and explores for novel relationships; for example, EDA might ignore the standard results of linear regression analysis examining more carefully the residuals from such an analysis. Said more accurately, EDA and data mining often care more to avoid Type II statistical errors in which useful patterns are missed than be worried about selecting or describing a pattern that on further study is not really there.

A major research arena has been the use of visualization to aid in the analysis of large datasets including virtual reality simulations for large datasets. The eye can often observe minute variations or broad patterns that are difficult to detect when presented with merely large columns of numerical data. Visualization is a natural addition to EDA since both permit more serendipitous examinations of data. Recent innovations in technology, such as larger storage devices, improved display devices and much faster digital processors, have made such software visualizations much more feasible. Such visualization methods allow analysts to manipulate data by rotating, zooming, panning, slicing and dicing datasets to extract meaningful relationships and patterns not easily discerned by traditional statistical techniques. Such visualization techniques also allow analysts with certain disabilities to better examine datasets.

Visualization has a range of marvelous techniques and has led to many interesting insights in large datasets. Work in research for improving visualization techniques has stressed the analysis of various visual variables such as size, shape, orientation, color, texture and position as well as dynamic variables such as duration of image, image shift, image change and image sequencing. Research in visualization has studied how each of these and other such variables affect the ability of analysts to do their work effectively and efficiently.

Sonification: Enhancing EDA with Sound

Researchers, including my research group, have begun to show that sound alone and in combination with visualization techniques can further enhance data mining and EDA methods employed by data analysts to seek relationships and patterns in large datasets. The use of sound in this regard is named sonification. Sonification studies the effects of sound variables such as pitch and relative pitch, volume, duration of tones, timbre (or sound quality, say, cello vs. piano middle C), rate of change of tones, articulation or order of sounds and augmentation (additional tones surrounding the main tone). Sounds are easily used as an alerting tool to quickly gain an analyst's attention to a strange detail or emergency condition. Tools of this sort have been called earcons, serving the purpose of icons to indicate important events. Sound can also be used to locate an object much as we hear a train coming and going. One minor but far from insignificant use of sonification is for those with visual impairments. Some researchers are exploring such use of sonification but our efforts are directed to sonification and sonfication with visualization for general populations.

Several schemes have been devised to link data to sound:

Range depiction. The range of a variable is categorized into several classes and each class assigned a tone. For example, a higher value assigned a higher pitch, louder volume, different instrument, etc.