Van Dam project hopes to marry 3-D graphics with interactive electronic books that one day may train surgeons using virtual reality

See additional articles:

• Trio collaborates on modeling brain cell behavior
  
• Charniak and others push into new areas of speech recognition
  
• Upfal project explores dynamic behavior of networks
  
• Van Hentenryck in the hunt for algorithm that takes uncertainty into account
  

You are observing an accomplished surgeon perform a delicate operation, standing right beside him at the operating table. To get a better look, you peer over the surgeon’s right shoulder. Then you move to the other side of the table, where you view the surgery from yet another perspective. Despite the difficult work at hand, the surgeon patiently explains every problem he encounters, every technique he employs. When you’d like to see one of those techniques again, the surgeon obliges willingly — at no risk to his patient.

In an ITR project led by Brown and UNC, "Teleimmersion" technology will be developed to train surgeons. Trainees will be able to move naturally within a life-sized high-fidelity three-dimensional graphical reconstruction of the surgery. Image courtesy Andrei State, UNC Chapel Hill.

Welcome to the world of virtual reality. This operation may have taken place months ago, possibly thousands of miles away, but if the dream of Computer Science Professor Andries van Dam and his collaborators is realized, medical students and others will be able to witness a past surgical procedure as if they were right in the room.

The National Science Foundation recently awarded van Dam and his team $1,750,000 to pursue this dream. The proposed project blends two of van Dam’s long-term passions — interactive electronic books (with dynamically changing, responsive content) and immersive three-dimensional graphics

Titled "Electronic Books for the Tele-immersion Age," the project brings together experts in several different disciplines, including medicine, and builds on the previous work of the computer scientists on the team. Serving as co-principal investigator with van Dam is Gregory Welch, a computer scientist at the University of North Carolina at Chapel Hill.

For years, medical students have watched surgical procedures on videotape, a medium surgeons find "marginally effective at best," according to van Dam. A video, he notes, has a fixed point of view and lacks three-dimensionality. "In short," he says, "watching a video is not even close to actually seeing the procedure."

Van Dam’s vision calls for a three-dimensional time machine, allowing surgeons and students to walk — virtually — around a life-sized high-fidelity reconstruction of the original event, and to move backward and forward in time. Existing tools for time-varying "scene capture" are crude, says van Dam. The authoring tools and cinematic techniques for this new medium are virtually unknown.

Van Dam and his team are focusing on trauma surgery, although the technology they plan to develop has dozens of possible applications. Teaching trauma surgery, they note, is especially difficult because some situations occur so infrequently that many surgeons encounter them only a few times in their careers. The most effective way to learn the appropriate response to a particular trauma is to observe the surgery firsthand, and then to actively participate in the procedure, but these opportunities are rare. Immersive "electronic books" would make the experience widely available.

None of this will happen soon, van Dam concedes. At the expiration of the NSF grant in 2004, "we’ll be at Kitty Hawk, hoping to fly for half a minute 20 feet off the ground," he says. By 2010, however, he envisions a much longer flight.

That flight, van Dam wrote in his NSF proposal, might consist of the following scenario: Dr. Smith, a surgeon and a leading expert in her field, employs an innovative technique to repair a young girl’s heart. As she performs the procedure, a sea of cameras captures a three-dimensional model of the operating room and all its occupants. Every movement and detail are recorded.

After surgery and back in her office, Dr. Smith dons her three-dimensional glasses, as do several colleagues around the country. All are now "inside" a video of the procedure, free to discuss aspects of the procedure without compromising the safety of the patient. A media specialist then creates an annotated version of the three-dimensional "electronic book," which will be immediately available — via the Internet — to students and surgeons around the world. Now they, too, can step inside Dr. Smith’s operating room, hear her explanation of the surgery, watch her methods from any vantage point, and even move back in time to review an earlier step. Eventually van Dam wants them to feel and manipulate the surgical tools, but simulating the sense of touch, he says, is particularly challenging.

Van Dam’s project is part of a multi-institutional collaboration funded by the NSF. Work on a complementary component of tele-immersion, known as "real-time scene acquisition and reconstruction," is under way at the University of North Carolina, the University of Pennsylvania and the Pittsburgh Supercomputer Center.

"What we’re talking about is the 21^st century idea of a telephone, one in which people feel they are together in a shared space," says van Dam. "We just want to take that idea of telepresence three levels of realism further."