The entertainment industry is already well-practiced at building virtual worlds for film, television, and live events, but how high is the ceiling? In the last three decades, audiences have come to accept (and even expect) photoreal settings and fully animated characters in their content. For the most part, a storyteller’s general goal while using new technologies remains the same: to keep reality intact despite the often-fantastic nature of the things on screen. This requires hard-working teams of talented animators and motion capture experts who toil—sometimes for years—to build assets and craft final performances. It is painstaking work, and quite understandably the most analyzed and criticized part of a production. For artists who work in the immediacy of the on-set space, could there be a better way to capture a true performance for use in all sorts of technical film wizardry?
As we’ve discussed in this ongoing series, storytellers are always seeking new and more engaging ways to spin their yarns, using things like virtual reality technology to break down the wall between performers and audiences. More recently, another technology has taken center stage, redefining the ways in which stories are told. Welcome to the world of volumetric capture (also known as “volumetric video” or “VC”). Referring to the recording of a physical place, person, event, or object in a manner that makes it available in a three-dimensional space, volumetric capture gives storytellers a more efficient way of truly capturing a moment in time. An actor’s emotional performance, a running back’s thrilling forward drive, a musician’s explosive solo—this technology records these elements in a way that lets them be viewed, edited, and repurposed from any angle or perspective. This isn't just the promise of immersion, it’s an entirely new way to create.
If you’ve never seen the process in action, it’s like something out of an Arthur C. Clarke story, using a 10,000-square-foot dome riddled with multiple (sometimes even hundreds of) cameras to capture a live moment by shooting a thing from a bunch of angles simultaneously, recording inward to document the full moment in time as completely as can be done. Then, those extraordinarily talented technical craftspeople stitch all the pieces together to create something in which viewers can truly lose themselves. The final stitched video can act as a fully formed environment with which the creator can interact without inhibition, editing, manipulating, and viewing the element however they choose. The viewer’s encounter is equally engaging, with a host of new experiences that range from the aesthetic (think of how The Matrix’s beautiful “bullet time” sequences showed audiences a new kind of action snapshot) to the enveloping (imagine navigating a recreated 3-D environment in real time). For actors, it must feel a bit more like doing theater than film—recording a full, all-perspective performance in real time. Of course, this leaves a huge amount of heavy lifting to post-production teams, who are tasked with rendering extremely rich and complex video environments and piecing them all together. Still, this creates an increasing number of opportunities for true immersion.
To understand more clearly what this means and how it’s done, look at it from a different angle (pun intended). If you’ve ever viewed a house or apartment on sites like Redfin and Zillow, you’ve used 360º video browsing technology that allows you to virtually tour a property from your device, utilizing LiDAR (light detection and ranging) scanning and photogrammetry to complete a high-resolution spatial scan of the location. This ingenious technique is useful but limited. By moving a 360º camera from room to room, these sites let you traverse a house on a set track, viewing each new room from a fixed position. While this technology is certainly helpful, it has limits. With volumetric capture, your tour would be completely unrestrained. You could look anywhere you want—under the beds, into the closets—and even check how far the counters stick out. VC is what provides the depth (and volume … hence the name) to make it seem like you’re not just watching on a flat screen.
It’s hard to overstate the effect this can have on the filmmaking process. Imagine capturing an actor’s entire performance in one take and using it to create a dynamically edited scene. Want to move the thespian’s head or arm a bit to the left? Not a problem! Feel like the scene should take place at the beach instead of in the jungle? Just adjust it! Creators can tweak just about any piece of this completed element in post without the need to reshoot or change lighting. Think of it as the next step up the ladder from more traditional motion capture technology (like the process that Andy Serkis and company used to create “Gollum” in The Lord of the Rings series and “Caesar” in the more recent Planet of the Apes films). With motion capture, an actor’s physicality is recorded, and that data is used to animate a character with computer generated imagery. Volumetric capture is more akin to bringing an authentic person, place, or thing into a truly virtual space—much as James Cameron did in his 2009 film, Avatar. Even before that, early pioneers of this technique honed the technology on films like 2007’s Beowulf. Released in 3-D stereoscopic vision, the Robert Zemeckis-helmed movie contained intricate performances built on complex motion capture technology that recorded performers on a giant 7.5m x 7.5m x 4.3m stage (dubbed ‘The Volume’ by production). Using this method, filmmakers eschewed the traditional on-set practice of calling “cut” and “action” to start and end each take. In this fashion, performers are not hindered by things like camera reloads, lighting setups, and scene changes.
Or go one step further and imagine VC in association with extended and virtual reality experiences. While classic VR traditionally employs 360º video, volumetric capture has the potential to add considerable depth to an experience—both physically and emotionally. 360º videos allow users to watch the experience from a fixed depth, while volumetric video could give audiences the power to play “director,” exploring the space however they feel compelled. Practically speaking, this technology has an uncalculated number of applications across the mediasphere, and it’s highly likely that we’ll begin to better understand its lasting impact the more it’s adopted across the industry. And if the numbers are any indication, that adoption will continue to ramp up significantly. VC’s benefits have the potential to stretch beyond applications in scripted entertainment. What would its utilization look like in sports, live performances, or even stage work?
Tune in to part two to find out...