Brown and Microsoft Pair Up to Advance Pen-Based Computing

by Wendy Y. Lawton

In most cases, a computer keyboard makes for easy expression. Tap the keys to string together letters, words, sentences, ideas; hit "save" or press "send."

But what if your language isn't based on letters? Mathematicians and chemists, choreographers and composers express themselves through symbols. Architects and designers sketch. How do they communicate and create using computers?

Not naturally, if at all. But pen-based computing holds promise. Chucking keyboard and mouse for stylus and screen, these systems handle like pen on paper, potentially making entry and manipulation of symbolic notations and sketches natural and easy. The Brown University Computer Graphics Group, led by Professor of Computer Science and Vice President for Research Andries van Dam, has been getting computers to recognize and interpret hand-drawn input for over ten years. Now the researchers have a new partner - the world's largest software company.

Microsoft Research, the research-and-development arm of the software giant, is investing up to $1.2 million over three years in Brown computer science creativity under a joint research alliance. The funding creates the Microsoft Center for Research on Pen-Based Computing, the first academic program in the nation dedicated to pen-centric computing innovation.

The center was launched at a March press conference at the Watson Center for Information Technology. At the event, President Ruth J. Simmons hailed the center as an idea engine critical in a global economy. Rick Rashid, senior vice president of Microsoft Research, predicted that the partnership will drive future innovation.

"What you're seeing now," Rashid said, "is Brown and Microsoft stepping up to the next level."

Although it debuted in the 1960s, pen-based computing has only in recent years gained a major presence in the computer marketplace.

Palm Pilots, Pocket PCs, and other personal digital assistants operate with a stylus. Slim, light, portable computers called tablets allow users to write directly on the screen using a stylus. Electronic whiteboards, meanwhile, are popping up in boardrooms and classrooms. The devices, which look like conventional whiteboards, have touch screen capability and drawing flexibility, allowing users to scroll through computer files, navigate the Internet, and make notes on material on the screen.

"In some cases, the pen is mightier than the keyboard," said van Dam.

A founding member and first chair of the Department of Computer Science, van Dam will serve as director of the new Microsoft center. Faculty, staff, and students working under the alliance will create new software for pen-based computers such as tablets. It's not an easy assignment. Researchers must make systems that recognize a slew of gestural commands and handwriting styles.

But van Dam said the potential payoff is significant. "You've got the ease of a pen with the power of a computer," he said. "We want to push the technology as hard as we can. And if we make it natural enough, people will find it better than keyboarding and mousing."

The design of pen-based software began at Brown in the early 1990s, when Bob Zeleznik, now director of research for the Graphics Group, began work on a gesture-based 3D modeling program called SKETCH. Members went on to create an electronic notebook and prototype tablet programs such as Music Notepad, which allows composers to enter and edit music notation, and MathPad², which recognizes handwritten mathematical notations and uses them to animate simple 2D sketches.

ChemPad is another Graphics Group creation. Started with a seed grant from President Simmons and spearheaded by doctoral student Dana Tenneson, the program generates interactive 3D models from 2D molecule drawings.

Professor of Chemistry Matt Zimmt uses ChemPad during his organic chemistry lectures as well as in separate student tutorials, where more than 200 undergraduates have used the program since spring 2005. Zimmt said the program does something invaluable. It takes the symbolic language of chemistry and transforms its letters and lines into 3D models that show a molecule's structure, size, and energy level.

"When I use ChemPad, an audible gasp goes up in the room," Zimmt said. "The molecule has come to life. ChemPad is a terrific tool. And that's what makes this center exciting. We may get to make more of these tools at Brown."