What is virtual production: an in-depth guide

Why haul production teams around the world when you can create stunning footage straight from a local studio? 

That’s the pitch for virtual production (VP), a faster, cheaper and unconventionally creative alternative to a traditional shoot. Virtual production can be used for all kinds of content — whether you’re crafting a Hollywood blockbuster, making branded content or shooting an ad.

In this blog, we’ll cover the need-to-know virtual production information, right from what it is and how it works, through to its benefits, use cases and workflows.

We’ll also share tips and hot takes along the way from freelance producer, Monesha Madison Lever, who has worked with leading global virtual production names like Final Pixel.

What is virtual production?

Virtual production combines smart technologies including LED panels, computer-generated images (CGI), augmented reality (AR) and motion capture to create dynamic backdrops for video production. 

The beauty of virtual production is that the LED screens can depict anything. The only limit is your imagination. By removing everyday production snags like time, weather and physical location, virtual production frees teams to flex their creative muscles. 

Want to shoot a 10-hour dawn? Or maybe flip from a futuristic cityscape in the morning to a barren desert in the afternoon? VP lets you do it all from the same studio.

By blending real and virtual spaces, VP also enables a more interactive environment than a green screen, helping both actors and crew to create more intuitively. 

But, as we’ll explore in more detail later, the nature of virtual production does require a shift in execution, with visual effects (VFX) moving from post to pre-production. This makes preparation and planning extremely important in the run up to any shoot.

How does virtual production work?

Virtual production takes a long list of the latest advances in tech –– including motion capture, CGI, facial recognition, AR, laser scanning, robotics and many more –– and seamlessly integrates them into one platform.

We asked Monesha what this combination of tech looks like in practice on a high-end production. 

“Three technologies are needed to achieve a “Mandalorian-style” virtual production. Camera tracking, 3D game engine and LED walls. It’s the integration of these three that allow filmmakers to blend the real world with the virtual world to create realistic environments.”

“The physical camera is tracked into the game engine, allowing the 3D environment to respond appropriately to the DP’s movements and achieve the parallax effect. Practical set pieces are not required but highly recommended to add more parallax and complete the illusion.” 

Directors use software and tools like Unreal Engine, Perforce and Pixotope to manipulate and edit their production in real-time during pre-production. 

Meanwhile, real-time tracking of live camera footage feeds precise movements into rendering platforms, bringing scenes to life instantaneously. This moves most of the CG work out of the post-production stage and allows teams to get an idea of what the finished project will look like much sooner.

Virtual production in action

Virtual production has been used in film and TV since the late 2000s and early 2010s. James Cameron was an early adopter, using VP in Avatar in 2009. 

Since then the technique has been used in numerous blockbusters. This includes post-apocalyptic sci-fi adventure film Oblivion (2013) which was one of the first major uses of the technology and The Mandalorian, which drew more attention to VP with groundbreaking new techniques.

Beyond movies and film, VP studios like our friends at Cube Studio are empowering teams to use virtual production for all kinds of creative — including ads, customer content and training material.

On top of this, cool companies like Magicbox are making VP more accessible with mobile studios, helping teams to capture virtual content in minutes, rather than months.

We’ll take a closer look at the types of teams that can put virtual production to use shortly, but first let’s check out the broader benefits. 

What are the benefits of virtual production?

With budgets and schedules squeezed ever tighter, virtual production offers a faster and cheaper alternative to traditional shoots. But that’s not all it has to offer. 

Let’s run through some key reasons VP is growing in popularity. 

• Virtual set creation: Filmmakers craft intricate and expansive digital sets, without any physical construction. Actors and crews can stay onsite, with no need to travel to remote or extreme locations.

• Real-time previsualization: Using virtual production, creatives can visualize complex scenes and make instinctive adjustments before final shooting.

• Dynamic backdrops: Unlike static green screens, virtual production uses LED walls to display lifelike backgrounds, improving both actors’ performances and directorial decisions.

• Cheaper location shooting: VP offers an alternative to expensive location shoots, replicating diverse environments in a controlled studio setting. Portable virtual production sets, built into trailers, offer further flexibility.

• Smoother post-production workflow: Virtual production streamlines the post-production process by integrating real-time VFX, reducing the need for extensive post-edits.

• Interactive lighting: Reflect real-world lighting conditions on actors and sets using virtual production’s dynamic lighting capabilities. Time of day also becomes a non-issue, with light dictated purely by your needs. 

• Real-time animation integration: VP can blend live-action and animated elements seamlessly, making it easier for actors to interact with CGI characters.

As Monesha explains, VP can be more than just a low-cost alternative. 

“Actors love seeing the world they interact with instead of looking into a green void. Car companies benefit from the sexy reflections cast on their latest models by the LED wall. Post-production benefits from a streamlined process of resolving problems before cameras start rolling.”

She goes on to share that we’re only just scratching the surface of virtual production’s benefits. “As the technology gets into the hands of more people with diverse creative ideas, the list continues to grow.”

Virtual production and sustainability

Another virtual production benefit that’s growing in importance is sustainability.

Screen New Deal’s report revealed the average big budget film production produces 2,840 tonnes of CO2 equivalent. Half of this comes from fuel, 30% from energy utilities and 16% from air travel. To put that into perspective:

• The air travel on an average tentpole film production is equivalent to 75 return flights between London and New York.

• The average fuel used making a blockbuster is equivalent to 3.4 million miles driven by an average passenger vehicle.

Virtual production provides an opportunity for Hollywood — or any production — to walk the walk on sustainability by upping the use of VP to cut fuel and travel emissions dramatically. 

What kind of creative teams can use virtual production?

The benefits of virtual production aren’t limited to the film and television industries.

Here are a few examples of how creative teams in various industries can use virtual production. to sidestep budget constraints and logistical hurdles, or take their creativity up a notch. 

• Indie filmmakers: Filmmakers can craft a global story on a local budget by emulating international locations without ever boarding a plane.

• Advertising agencies: With VP, ad agencies can quickly create commercials that need to showcase products in multiple settings. By easily switching between a beach at sunset to a bustling city square, they can add an extra dimension to campaigns.

• Corporate marketers: Virtual production can elevate video content to help marketing teams stand out and swiftly respond to customer needs or competitors. VP can be used by marketers shooting events, keynotes, training videos, product announcements and more.

• Music video producers: Producers can get super creative and shoot an artist’s performance in any way that fits with the lyrics or performer. Virtual production lets them blend reality with fantasy and satisfy even the most outlandish requests.

• Gaming studios: VP is brilliant for creating trailers or cinematic sequences for games. Teams can blend game engine visuals with live actors to achieve a seamless look that resonates with the game’s aesthetic.

• Animation teams: Blending live-action with animated elements most often for children’s shows or mixed-medium storytelling. The characters come to life alongside real actors. 

Virtual production can be a great tool for any team with an idea and the resources to shoot content. But what kind of resources do you really need? We asked Monesha for her take. 

Is virtual production expensive?

VP is potentially useful for all kinds of teams. But is it prohibitively expensive? Monesha fills us in: 

“No one wants to hear it, but the answer is: ‘It depends.’ The budget depends on the creative. It depends on how large of an LED volume you need. It depends on how many environments you want to film in. It depends on how wide the camera angles are. It depends on so many factors.” 

“The good news is there are many ways to accomplish virtual production in different budget ranges. The best place to start is with a creative idea. That can be in the form of a script, storyboard or creative brief. Then, consult with virtual production pros and let them know how much you can spend (or can’t spend). They’ll help you determine if it’s the right technique and how to achieve it within your budget.”

Preparing a virtual production setup

As we’ve noted, pre-production and planning become even more important during a virtual production.  

Creating an immersive virtual environment on set shifts creative decisions usually associated with post-production to the pre-production stage. This means production-ready environments, 3D assets, textures and effects need to be built upfront. 

In a virtual production setup, visual development is handled by the virtual art department. You can explore their role in detail in this Epic Games virtual production guide. But, essentially, the virtual art department create camera-ready assets and act as a link between creatives and artists to ensure all feedback is taken on board.

Another critical step is determining the technical requirements of the production. This includes making decisions about the: 

• Camera setup

• Scale of the LED volumes 

• GPU power required for the setup

• Collaborative software tools

• Camera tracking systems 

• Pixel density of the LED volumes in order to avoid moiré issues.

How collaboration powers virtual production 

Preparing any virtual production involves a lot of moving pieces. The need to build camera-ready visuals before a shoot adds even more emphasis on collaboration in the pre-production stage. 

This is where tools like LucidLink can help.

With LucidLink, VP teams have a single source of truth where creatives can instantly access media files of any size, from anywhere. It doesn’t matter if you’re working with overseas freelancers, or need to quickly access creative away from a shoot –– you can access and edit the most up-to-date files straight away. This makes real-time collaboration a reality.

Final Pixel, a global creative studio specializing in virtual production, relies on LucidLink to work together on projects no matter where they are.

“Our founders are based in LA, New York and London and we scale to 20 or 200 artists and production talent depending on the scale of the project. LucidLink is the primary way we share all of our files…it’s the backbone of everything we do collaboratively.”

What is a virtual production workflow?

Working as a team to prepare your setup is just one part of the VP workflow. The whole virtual production workflow encompasses the processes, technologies and techniques used to combine live-action shooting with virtual environments and real-time VFX. 

While workflows differ depending on a project’s specific requirements, a filmmakers’ virtual production workflow will typically include the following:

1. Pre-visualization (Previs)

Before actual shooting begins, filmmakers use computer graphics to create a rough representation of sequences, scenes, or entire movies. This step helps directors and producers understand and plan camera angles, movements, pacing and more.

2. Technical visualizations (Techvis)

A technical extension of previs. Techvis is used to determine the feasibility of shots, considering the physical attributes of the shooting location, camera rigs and other technical aspects.

3. Asset creation

Digital assets like characters, environments and props are modeled, textured, rigged and animated if necessary. These assets are optimized for the real-time engines used in virtual production.

4. Setting up the volume

The physical space (typically surrounded by LED screens or walls) is set up. The screens display the virtual environments where the scenes will take place.

5. Calibration & tracking

Cameras are calibrated to ensure they align with the virtual environment. Real-time camera tracking systems are set up, ensuring that as the camera moves, the perspective in the virtual environment adjusts accordingly (parallax effect).

6. Shooting

Scenes are filmed with actors on the stage, with the virtual environment providing real-time backgrounds and lighting. Depending on the setup, filmmakers might see finalized (or near-finalized) shots right on set.

7. Real-time review

Directors and cinematographers can review shots in real-time together from a shared filespace, like LucidLink, to adjust lighting, camera movements or even virtual assets as needed.

8. Post-production

Traditional post-production processes like editing, color grading and sound design. Additional visual effects that weren’t achieved in real-time can be added during post. All virtual and real elements are further refined and integrated seamlessly.

9. Final rendering

For sequences or assets not rendered in real-time during the shoot, a final rendering process brings everything to its highest quality.

10. Final review & distribution

After all edits, visual effects, sound designs and other elements are finalized, the project is reviewed, approved and prepared for distribution.

While this workflow maps out a general structure, virtual production processes can flex with your project. Steps and technologies can be modified, rearranged or even skipped depending on the needs and scale of a particular project and team.

VFX in virtual production

We’ve already mentioned how pivotal VFX is to virtual production. 

It’s the foundation for filmmakers to create stunning and immersive environments that would be impossible or impractical to shoot in the real world. VFX integrates seamlessly with live-action footage, enhancing the storytelling and visual appeal of a project.

Why VFX changes for virtual production

In traditional filmmaking, VFX work is typically done in post-production. But, in VP, most of the effects are created and integrated during the pre-production and production stages. 

This shift allows directors and producers to see the final look of scenes in real-time. That way they can make adjustments on the fly and reduce the need for extensive post-production work.

Key VFX techniques in virtual production

Real-time rendering

Using powerful game engines like Unreal Engine, virtual production can render complex visual effects in real-time. This capability lets filmmakers see the final composite of live-action and CGI elements on set, allowing for immediate feedback and adjustments.

Motion capture 

Actors' movements are captured and translated into digital characters in real-time. This technique is essential for creating lifelike performances and integrating them seamlessly into virtual environments.

Green screen replacement

While LED walls are popular, green screens are still used in virtual production. The key difference is that the green screen can be replaced with virtual sets and environments in real-time, providing immediate visual context for actors and directors.

Augmented reality (AR) and mixed reality (MR)

These technologies overlay CGI elements onto the real world, allowing actors to interact with virtual objects and characters naturally. This interaction makes performances feel more realistic while improving the final visual output.

The benefits of integrating VFX early

Moving VFX closer to the start of a production workflow can also have positive overall effects on a project. 

• Coherent creative vision: Directors and VFX artists can collaborate closely from the beginning, ensuring that the creative vision is realized throughout the production process.

• Cost efficiency: By resolving visual effects challenges during the pre-production and production stages, filmmakers can reduce costly revisions and re-shoots.

• Time savings: Real-time VFX integration speeds up the production process, leading to quicker turnaround times and more efficient workflows.

Keep virtual production projects on track

That’s the end of our whistle-stop tour of virtual production.

Whether you’re creating films, ads or marketing, virtual production offers new ways to win over audiences, while giving you greater control over the elements, timelines and costs.

VFX’s early introduction in the creative process also helps directors and producers ensure projects are on track from the get-go and keep budgets in check.

As we’ve seen, the success of any virtual production hinges on meticulous preparation and teamwork. By providing real-time access to large media files and effortless collaboration across distributed teams, LucidLink’s shared filespace keeps every member of your production team on the same page, wherever they are.

Want to see how LucidLink can power your virtual production workflow? Start a free trial today.

Images source: Monesha Madison Lever