Pictures posing questions: the next steps in photography could blur reality

Science News, April 7, 2007 by Patrick L. Barry

When a celebrity appears in a fan-magazine photo, there's no telling whether the person ever wore the clothes depicted visited that locale. The picture may have been "photoshopped," we say, using a word coined from the name of the popular image-editing software Adobe Photoshop.

But today's image processing is just a prelude. Imagine photographs in which the lighting in the room, the position of the camera, the point of focus, and even the expressions on people's faces were all chosen after the picture was taken. The moment that the picture beautifully captures never actually happened. Welcome to the world of computational photography, arguably the biggest step in photography since the move away from film.

Digital photography replaced the film in traditional cameras with a tiny wafer of silicon. While that switch swapped the darkroom for far more-powerful image-enhancement software, the camera itself changed little. Its aperture, shutter, flash, and other components remained essentially the same.

Computational photography, however, transforms the act of capturing the image. Some researchers use curved mirrors to distort their camera's field of view. Others replace the cameralens with an array of thousands of microlenses or with a virtual lens that exists only in software. Some use what they call smart flashes to illuminate a scene with complex patterns of light, or set up domes containing hundreds of flashes to light a subject from many angles. The list goes on: three-dimensional apertures, multiple exposures, cameras stacked in arrays, and more.

In the hands of professional photographers and filmmakers, the creative potential of these technologies is tremendous. "I expect it to lead to new art forms," says Marc Levoy, a professor of computer science at Stanford University.

Medicine and science could also benefit from imaging techniques that transcend the limitations of conventional microscopes and telescopes. The military is interested as well. The Defense Advanced Research Projects Agency, for example, has funded research on camera arrays that can see through dense foliage.

For consumers, some of these new technologies could improve family snapshots. Imagine fixing the focus of a blurry shot after the fact, or creating group shots of your friends and family in which no one is blinking or making a silly face. Or posing your children in front of a sunset and seeing details of their faces instead of just silhouettes.

Since the late 1990s, inexpensive computing power and improvements in digital camera technology have fueled research in all these areas of computational photography. Levoy says that scientists "look around and see more and more everyday people using digital cameras, and they begin to think, 'Well, this is getting interesting.'"

ROBOTS TO SUPERHEROES Computational photography has roots in robotics, astronomy, and animation technology. "It's almost a convergence of computer vision and computer graphics," says Shree Nayar, professor of computer science at Columbia University.

Attaching a video camera to a robot is easy, but it's difficult to get the robot to distinguish objects, faces, and walls and to compute its position in a room. "The recovery of 3-D information from [2-D] images is kind of the backbone of computer vision itself," Nayar says.

Other important optics and digital-imaging advances have come from astronomy. In that field, researchers have been pushing boundaries to view ever-fainter and more-distant objects in the sky. In one technique, for example, the telescope's primary mirror continuously adjusts its shape to compensate for the twinkling effect created by Earth's atmosphere (SN: 3/4/00, p. 156).

Rapid progress in computer animation during the 1980s and 1990s provided another cornerstone of the new photography. The stunning visual realism of modern animated movies such as Shrek and The Incredibles comes from accurately computing how light bounces around a 3-D scene and ultimately reaches a viewer's eye (SN: 1/26/02,p. 56). Those calculations can be run in reverse--starting from the light that entered the lens of a camera and tracing it back--to deduce something about the real scene.

Such calculations make it possible to decode the often-distorted images taken by these unconventional cameras. "What the computational camera does is it captures an optically coded image that's not ready for human consumption," Nayar explains. By unscrambling the raw images, scientists can extract extra information about a scene, such as the shapes of the photographed objects or the unique way in which those objects reflect and absorb light.

PHOTO FUSION One powerful way to do computational photography is to take multiple shots of a scene and mathematically combine those images. For example, even the best digital cameras have difficulty capturing extreme brightness and darkness at the same time. Just look at an amateur snapshot of a person standing in front of a sunlit window.

Compared with a single photo, a sequence of shots taken with different exposures can capture a scene with a wide range of brightness, called the dynamic range. Both a bright outdoor scene and the person in front of it can have good color and detail when the set of images is merged. The method was described by Nayar and others at a conference in 1999.