What is virtual production: the comprehensive guide

Instead of flying actors and full production teams on location to execute a feature film, Hollywood is turning to virtual production to set the scene. After the global pandemic demonstrated that teams can accomplish more in pre-production remotely, saving time and money, the media and entertainment (M&E) industry further embraced virtual production for larger-scale projects.

Virtual production has appeared on media sets since the late 2010s, with Hollywood’s post-apocalyptic sci-fi adventure film “Oblivion” (2013) being one of the first major uses of the technology and “The Mandalorian” drawing more mainstream attention to the technology. In this blog we’ll cover this emerging trend and how it offers directors and producers new avenues to create captivating content. We’ll also share an expert’s views on the impact of virtual production – Monesha Madison Lever, a freelance producer, who has collaborated with big names like Final Pixel, a leading global virtual production house.

What is virtual production?

Essentially, virtual production merges real-time visual effects with classic filmmaking. Rather than waiting for post-production to begin visual effects, virtual production brings them into the pre-production phase. It's an entertainment technology used in video production where LED panels serve as a dynamic backdrop. On these panels, video or computer-generated images (CGI) can be showcased in real-time.

Virtual production empowers VFX teams to flex their creative muscles to create stunning virtual environments while giving the overall filmmaking process the adaptability to be shot anywhere.

The benefits of virtual production

When much of the world pivoted to remote work, the M&E industry had to quickly adapt. This transition led to a range of opportunities in the field of virtual production for filmmakers. Let’s explore the benefits below:

• Virtual set creation: Virtual production allows filmmakers to craft intricate and expansive digital sets without physical construction or to fly actors and crews to remote or extreme locations.

• Real-time previsualization: With virtual production, creatives can visualize complex scenes and make adjustments on the fly before final shooting.

• Dynamic backdrops: Unlike static green screens, virtual production uses LED walls to display lifelike backgrounds, enhancing actors’ performances and directorial decisions.

• Cost-effective location shooting: Virtual production offers an alternative to expensive location shoots, replicating diverse environments in a controlled studio setting. Portable virtual production sets have also been built-into trailers offering further flexibility.

• Enhanced post-production workflow: Virtual production streamlines the post-production process by integrating real-time VFX, reducing the need for extensive post-edits.

• Interactive lighting: Reflect real-world lighting conditions on actors and sets using virtual production’s dynamic lighting capabilities.

• Real-time animation integration: Blend live-action and animated elements seamlessly with virtual production, making it easier for actors to interact with CGI characters.

“Actors love seeing the world they interact with instead of looking into a green void.  Automobile companies benefit from the sexy reflections cast on their latest models by the LED wall. Post-production benefits from a streamlined process of resolving problems before cameras start rolling. As the technology gets into the hands of more people with diverse creative ideas, the list continues to grow,” shares Monesha.

What teams benefit from virtual production?

Virtual production technology is not limited to the film and television industry. It’s beneficial to content creators and video teams working in other creative capacities as well. Here are some examples of how teams use virtual production to improve process efficiency and expand creative potential that budgets or other logistics may have previously limited.

• Indie filmmakers: Aiming to shoot in various locations. With virtual production, they can emulate international locations without ever boarding a plane, enabling them to craft a global story on a local budget.

• Advertising agencies: Creating commercials that need to showcase products in multiple settings. They can easily switch between a serene beach at sunset to a bustling city square, adding versatility to their campaigns.

• Corporate marketing: Shooting events, keynotes, training videos, product announcements and more. Virtual production elevates the level of video content and allows corporate marketing teams to swiftly respond to customer needs and competitive dynamics. 

• Music video producers: Looking to shoot an artist’s performance in imaginative and ever-changing landscapes. Virtual production lets them blend reality with fantasy, enhancing the narrative.

• Gaming studios: Creating trailers or cinematic sequences for their games. Blending game engine visuals with live actors can achieve a seamless look that resonates with the game’s aesthetic.

• Animation teams: Blending live-action with animated elements most often for children’s shows or mixed-medium storytelling. The characters come to life alongside real actors.

There are so many benefits! It depends on who you ask. With virtual production, directors can film in impossible locations – distant planets or remote locations inaccessible to large crews.

Monesha Madison Lever, Freelance Producer

How does virtual production work?

Virtual production is enabled by the fusion of different technologies. This encompasses motion capture, CGI, facial recognition, Augmented Reality (AR), laser scanning, robotics, and several others, all seamlessly integrated into a unified platform.

Monesha gives us the scoop:

“Three technologies are needed to achieve a “Mandalorian-style” virtual production. Camera tracking, 3D game engine, and LED walls. It’s the integration of these three that allow filmmakers to blend the real world with the virtual world to create realistic environments. The physical camera is tracked into the game engine, allowing the 3D environment to respond appropriately to the DP’s movements and achieve the parallax effect. Practical set pieces are not required but highly recommended to add more parallax and complete the illusion.” 

Directors use software and tools like Unreal, Perforce, and Pixotope, among others, to manipulate and edit their production in real-time in the pre-production stage. Additionally, real-time tracking of live camera footage feeds precise movements into rendering platforms, bringing scenes to life instantaneously. This moves most of the CG work out of the post-production stage and allows teams to have an earlier idea of what the finished project will look like. 

What is the term “volume” in virtual production?

Originally designed for advertising spaces, entertainment venues, and broadcasting, LED panels have since found a new purpose on film sets, serving as dynamic backdrops, known as volumes, for motion picture shoots.

Volume is a physical modular space surrounded by thousands of LED screens or walls that display CGI in real-time. These LED walls can depict landscapes, cities, or any desired background, allowing actors and crew to interact with their environment more intuitively than they would with a traditional green screen.

What is needed for virtual production

One of the major mindset shifts when setting up virtual production is the emphasis on pre-production. In order to replace the green screen setup with an immersive virtual environment on set, many of the creative decisions usually associated with post-production are moved to the pre-production stage. This means production-ready environments, 3D assets, textures and effects need to be built upfront. 

In a virtual production setup, visual development is handled by the Virtual Art department. Their work is explained in detail in a guide by Epic Games but essentially they create camera-ready assets and act as a link between creatives and artists to ensure all feedback is taken into account. Building camera-ready visuals before shooting brings an additional emphasis on collaboration in the pre-production stage where tools like LucidLink can help.

The second critical step is determining the technical requirements of the production. This includes making decisions about the: 

• Camera setup

• Scale of the LED volumes 

• GPU power required for the setup

• Collaborative software tools

• Camera tracking systems 

• Pixel density of the LED volumes in order to avoid moiré issues. 

What is a virtual production workflow?

A virtual production workflow encompasses the processes, technologies, and techniques used to combine live-action shooting with virtual environments and real-time visual effects. While workflows can vary depending on the specific requirements of a project, here’s a general overview of filmmakers’ virtual production workflow:

1. Pre-Visualization (Previs):

• Before actual shooting begins, filmmakers use computer graphics to create a rough representation of sequences, scenes, or entire movies.

• This step helps directors and producers understand and plan camera angles, movements, pacing, and more.

2. Technical Visualizations (Techvis):

• A technical extension of previs. Here, the feasibility of shots is determined, considering the physical attributes of the shooting location, camera rigs, and other technical aspects.

3. Asset Creation:

• Digital assets like characters, environments, and props are modeled, textured, rigged, and animated if necessary.

• These assets are optimized for the real-time engines used in virtual production.

4. Setting Up the Volume:

• The physical space (typically surrounded by LED screens or walls) is set up.

• The screens display the virtual environments where the scenes will take place.

5. Calibration & Tracking:

• Cameras are calibrated to ensure they align with the virtual environment.

• Real-time camera tracking systems are set up, ensuring that as the camera moves, the perspective in the virtual environment adjusts accordingly (parallax effect).

6. Shooting:

• Scenes are filmed with actors on the stage, with the virtual environment providing real-time backgrounds and lighting.

• Depending on the setup, filmmakers might see finalized (or near-finalized) shots right on set.

7. Real-Time Review:

Directors and cinematographers can review shots in real-time together from a shared space like LucidLink to adjust lighting, camera movements, or even virtual assets as needed.

8. Post-Production:

• Traditional post-production processes like editing, color grading, and sound design.

• Additional visual effects, not achieved in real-time, are added.

• Virtual and real elements are further refined and integrated seamlessly.

9. Final Rendering:

• For sequences or assets not rendered in real-time during the shoot, a final rendering process brings everything to its highest quality.

10. Final Review & Distribution:

• After all edits, visual effects, sound designs, and other elements are finalized, the project is reviewed, approved, and prepared for distribution.

While this workflow is a general structure, it’s important to understand that virtual production can take a versatile approach. The steps and technologies can be modified, rearranged, or even skipped depending on the needs and scale of a particular project and team.

Is virtual production expensive?

Monesha fills us in: 

How can LucidLink help with virtual production?

Significant data and large media files need to be accessible and editable in real-time by teams that are potentially distributed around the world. LucidLink can facilitate this by allowing editors, VFX artists, and other professionals to access media assets in real-time, regardless of their location. 

Incorporating a cloud storage and collaboration tool like LucidLink into virtual production workflows can help studios stay nimble, reduce costs, and facilitate seamless collaboration. As the film and media production landscape continues to evolve, and with the rise of remote work, tools that bridge the gap between local and cloud storage will play an important role in helping teams stay connected, organized, and in sync. 

“Footage shot in a virtual production volume works like footage shot on location in a traditional production. Once shot, the files can be saved to LucidLink and handed over to the editorial team,” Monesha shares.

Want to discover how LucidLink can maximize efficiency in your virtual production workflow? Try for free. 

Image source: Monesha Madison Lever