Harnessing Supercomputers Helps Karniadakis Get to Heart of the Matter

A massive, interconnected system helps the research team build an accurate three-dimensional computer model of arterial blood flow that ultimately will aid doctors who treat arteriosclerosis, coronary artery disease, and other circulatory diseases.

by Faith Singer-Villalobos and Merry Maisel

How will medicine leap to the next stage? The answers will depend not only on advances made in the laboratory, but also on new ways of modeling and simulating medical conditions on large-scale, interconnected computers.

If doctors could watch how blood circulates through our arteries in a kind of 3-D real-time movie, they could see exactly where high blood pressure affects individual arterial systems. They could tailor medication to reduce the pressure exactly. They could also see potential blockages in important arteries and design ways to remove or reduce them without major surgery.

Exact blood flow determination is the challenge taken up by George Em Karniadakis, who is not a physician but a professor of computational fluid mechanics at Brown. "Computational" is the important word here. Karniadakis and his team (which includes Professor Martin Maxey and Assistant Professor (research) Suchuan Dong of applied mathematics, engineering Professor Peter Richardson, and other researchers from Ben-Gurion University in Israel, Imperial College in London, and Brigham and Women's Hospital in Boston) are well on their way to building an accurate three-dimensional computer model of arterial blood flow that will ultimately aid doctors who want to save us from arteriosclerosis, coronary artery disease, and other circulatory diseases.

Their computer model tackles the human arterial "tree" as a series of bifurcating (splitting from one branch into two) pipes extending from the heart. As the computer "heart" pumps, the simulation calculates the speed of the viscous flow as it branches and re-branches.

"Many arterial diseases begin in the places where arteries branch," Karniadakis says, "where there is backflow or erosion of the arterial walls."

How does his computer model work, and where does it get its power?

Computer modeling, simulation and data analysis long ago joined laboratory experimentation and fieldwork in shaping and directing the course of the sciences. The Karniadakis computer model, for example, has to solve thousands of simultaneous equations to capture blood flow dynamics accurately. The first advance for Karniadakis and his group was to make the model run on a very large supercomputer, composed of many processors like the one in your laptop or desktop machine.

The difference between an ordinary computer like a laptop and today's largest supercomputers is very much like the difference between a horse-drawn carriage and a jet plane, only more so. With thousands of processors, supercomputers can make trillions of calculations per second "in parallel" and then sum up the answers. Today a supercomputer can perform all the arithmetic ever done by Galileo in his lifetime in a billionth of a second.

Karniadakis needs more power than a single powerful supercomputer can deliver.

"I need to use multiple supercomputers in parallel," he says. Only such massive, interconnected systems can support real-time, three-dimensional modeling.

Fortunately for Karniadakis and his team, and for thousands of other scientists and researchers, scientific computation is entering a new era in which the supercomputers themselves are interconnected, through projects like the TeraGrid, an effort funded by the National Science Foundation (NSF).

Think of supercomputers as great looms on which all of the sciences taught in separate university departments for so many years are interwoven through the methodologies of modeling and simulation. Chemistry, physics, computer science, and geosciences are woven together on the computational looms, and the TeraGrid enables the looms themselves to communicate with each other.

The TeraGrid has grown into a national-scale initiative to deploy and operate an interconnected system of computers that scientists and engineers can use to solve some of their most challenging problems. NSF recently awarded a $150-million grant to eight advanced computing centers across the nation to expand the TeraGrid. All of the centers are tightly linked over high-bandwidth networks to carry out the TeraGrid project. The system is the nation's largest and most comprehensive distributed computational infrastructure for open scientific research.

"It is a fundamental experiment in building what we call a national cyberinfrastructure," says TeraGrid director Charlie Catlett of the University of Chicago/Argonne National Laboratory. Participants like Karniadakis' group "are figuring out how to do it: how to make all the resources work together while scientists are actually using them - a little like building the boat after you've set sail, but the only way to do it right."

"The TeraGrid makes collaboration across disciplines and institutions much easier, and it helps to make sure that, whatever the problem is, the best minds are working on it," says Juan Sanchez, vice president for research at The University of Texas at Austin, site of one of the eight advanced computing centers across the nation that are involved in expanding the TeraGrid.

The extraordinarily accurate model of blood flow in the human arterial system is so large that it depends on the resources at four TeraGrid sites simultaneously: the center at UT-Austin, the Pittsburgh Supercomputing Center, the National Center for Supercomputing Applications in Illinois, and the San Diego Supercomputer Center. The total capacity of all the computers used is nearly 35 "teraflops" - trillions of calculations per second.

"We take advantage of all of that," says Karniadakis, "and the fast network connections make it possible to synchronize and re-synchronize the calculations as necessary. Now we can investigate what happens when arterial flow is blocked at any point, as may happen in various disease processes, and we can design strategies to prevent or overcome the effects."