Volumetric video is an emerging way of both capturing and viewing video. Actions are recorded using multiple synchronized cameras that encircle a scene. Computer algorithms then rebuild that physical space in 3D, making a video reproduction that can be viewed from any perspective within the space. Directors could use it to show a scene from a perspective where a camera would have been impossible to place. With an added user interface, people can navigate their way through a scene — watching, for example, sporting events from on the field or a concert from the stage.
For all the promise of volumetric video, there are still challenges to overcome, and this new research tackles several. Chief among them, the work introduces a new way of compressing volumetric video, which makes storing and streaming feasible with current media infrastructure.
“Volumetric video is incredibly hard to store and stream,” Rai said. “A 30-minute clip can balloon to terabytes of data, and the formats it comes in are completely alien to the infrastructure the internet already runs on — your computer, your streaming service, your video codec.”
Their solution was to start with the state-of-the-art method of rendering 3D scenes, known as 3D Gaussian splatting. The technique renders 3D images using “Gaussians,” fuzzy blobs that encode the color, opacity and shape of points in space. The quality of the images is high, but file sizes are huge. The innovation in this new work is a way of mapping the 3D scene and its millions of Gaussians into a more manageable 2D image in a way that’s similar to projecting a globe onto a flat map.
The result is “a structured, multi-scale image that encodes the entire dynamic 3D scene,” Rai said. Stack those 3D-encoded images together, and it makes a video with a reasonable file size that is compatible with stalwart video codecs that run Netflix, YouTube and most of the rest of the internet.
Creating a digital twin of the real world
There’s another key challenge the work addresses. Other gaussian splatting approaches to volumetric video work well for short videos, the researchers say, but often break down over longer sequences. To work properly, rendering approaches to volumetric video must keep track of moving objects in a scene. But current tracking techniques often lose objects that temporarily disappear from sight — for example, when a ball temporarily disappears behind a person. They also have trouble dealing with novel movement — a person entering a room in the middle of a sequence of events. This new work introduces a new approach to the problem.