A machine that thinks and listens - MIT researchers find a way of making a wearable computer aware of auditory cues

Electronics Times, Nov 30, 1998

A group working at the Massachusetts Institute of Technology (MIT) Media Lab has found a way to use auditory cues to help a wearable computer determine where it is as the user moves around.

The goal is to build a wearable computer to react to real-world events so that it does not attempt to give the user information at inappropriate times, interrupt conversations with alarms and events or react to spoken phrases not meant for the computer.

The system works by trying to classify the sounds around the machine into two distinct layers. One contains sound objects such as speech, ringing telephones and passing cars. The other builds up sound scenes, such as street, supermarket or office environments.

The MIT system uses techniques that have also been applied to speech recognition but in a more general way. Filter banks are used to discriminate between speech and other sounds with other feature extraction techniques, such as cepstral analysis and linear predictive co-efficients used to further classify sounds.

Once extracted, the individual features form a time series that is modelled using hidden Markov models (HMMs), another technique used for speech recognition to join individual phonemes into phrases.

Because the system is event-driven, this modelling phase makes it much easier to compile examples that are used to train the system. New sounds are compared with the HMMs to determine which is the most likely scenario for the user's current environment.

Once it has decided on a particular environment, the system determines which messages should make it through to the user. For example, if it detects a conversation of more than several speakers over a period of time, that can be taken as an indication that an audible alarm is not appropriate.

The system could be trained further to recognise when the user is speaking and when the surrounding conversation is taking place further away.