Image Retrieval: Benchmarking Visual Information Indexing and Retrieval Systems

Bulletin of the American Society for Information Science & Technology, Feb/Mar 2007 by Rorissa, Abebe

What do cameras, Hollywood, flickr, YouTube Broadcast Yourself, magnetic resonance imaging (MRI) and computed tomography (CT) scans have in common? Among other things, they are tools or services or places for the creation, production, organization, management and sharing of images and/or videos. Information sources are becoming increasingly multimedia in nature. For the sake of brevity and delimiting the scope, this article will focus only on visual information, more specifically indexing and retrieval of image and video. As you may be aware from current technology events, news and controversies, Flickr (www.flickr.com) is a popular digital photo storage and sharing website and service, while YouTube (www.youtube.com), acquired recently (November 13, 2006) by Google, Inc, is perhaps the most popular free video sharing website.

Even though the exact amount of visual information is difficult to determine, there are an estimated 10 to 20 billion images and videos on the World Wide Web alone. This estimate does not include those in individual and private stock photo/video collections. However, only a small fraction of the large number of images and videos is indexed for effective and efficient retrieval.

What is it about anything visual that attracts people of all ages? Some researchers tell us that it is because our visual sense is the most powerful and well developed among all the human senses. Here in the United States, the day after the Thanksgiving holiday (a great holiday where families get together and give thanks for their blessings, not to mention feast on some great food) is the unofficial opening day of the holiday shopping season. Shopping centers and stores open as early as midnight and usually offer sizable discounts on their merchandise. People form lines in front of stores and, as I joined them, I could not help but notice that the longest lines were at stores such as Best Buy and Circuit City where the hottest items were photo and video cameras, camera phones, plasma/high definition TVs, DVD players/recorders and video game equipment - all appliances for image or video creation, sharing or viewing.

There is no doubt about the value of visual information for human communication. Images, such as paintings, were created and shared long before humans invented written language. Even in this electronic age where we can email or send a text message via mobile phones, words do not quite describe our vacation in Tahiti the way pictures and video clips do. Leonardo da Vinci's Mona Lisa is indescribable, showing why a picture is worth at least a thousand words. It also illustrates why, even if you can use a thousand words, it is still difficult to generate the ones that adequately annotate, describe or index many images. Which leads to the question: What is the current status of image and video indexing and retrieval?

Image Indexing and Retrieval: The Issues

Three brief examples will highlight persistent obstacles to overcome in indexing and retrieving images. First, there have been efforts to automate the process. Recently, I came across a BBC headline that read "System 'spots multimedia content'" in which the author describes a system that can automatically recognize the content in images and videos and assign labels. This is great news. However, a few paragraphs later, the author informs us that the system is not only a prototype but is also limited to certain scenes, such as a beach or a tennis match. While great strides have been made by designers of image indexing and retrieval systems, what the BBC article described is typical of current visual information retrieval systems.

Second, human ability to create images is exploding. Before we had digital cameras, as soon as we came back from a vacation trip, we rushed to a photo printing store and then stuck the resulting pictures in a photo album. Now that digital cameras are cheaper, we take hundreds of photos and dump them on our computer hard drives before we take hundreds more. Some of us are guilty of not organizing our digital photos. It is not entirely our fault. Intuitive and easy to use personal photo collection management tools available commercially or otherwise are scarce.

Third, there are the unique challenges of analysis in various domains, such as medical image indexing and retrieval, which is a very important application receiving considerable attention. Hospitals produce and maintain not only text-based records but also an enormous amount of visual data in the form of MRI and CT scan images. Medical images are becoming integral parts of medical record, which contain multimedia information. Medical researchers and practitioners rely on these multimedia records to make critical and, literally, life and death decisions. What is more, medical imaging is a multi-billion-dollar-a-year industry. While medical image indexing and retrieval are not much different from that of nonmedical digital images, they nonetheless pose some distinct challenges. For example, there are often only subtle variations between two images, especially if they are within the same domain category or anatomical region, making it difficult to identify the difference. Other challenges include the modality, which is whether the medical image was generated using MRI, CT scan or X-ray, anatomical region (e.g., head, lung/chest or leg) and the fact that medical images have lower signal to noise ratio than non-medical images.