What Is XR Virtual Production?

Introduction: With the development of the virtual entertainment industry, more and more TV and online programs have begun to use virtual broadcasting technology on a large scale. Recently, XR technology has begun to appear in large-scale evening parties including the Spring Festival Gala, which is refreshing to the audience.

At the immersive virtual concert of THE9 girl group "City of Virtual Reality" last month, iQiyi, which bet on virtual production, once again demonstrated the "dreamlike" visual effects of XR virtual production technology. Similar to Jay Chou's "Mojito" at this year's Spring Festival Gala, XR technology can be said to be a revolution in the expressiveness and innovation of the content industry.

▼ THE9 girl group "City of Virtual Reality" immersive virtual concert

XR virtual performance "Mojito" completed remotely and synchronously at different locations on the Spring Festival Gala

Forbes predicted in 2019 that the XR market is expected to grow 8 times in the next two years, and the market size will exceed US$209 billion by 2022.

1. What is XR?

2. Why is it so popular recently?

3. How does XR work in production?

4. What is the connection and difference between XR and virtual production?

5. What is XR virtual production?

6. XR production case

7. In conclusion

What Is XR?

▼ Virtual performances and live broadcasts using XR technology

XR (Extended Reality) is a term used to describe an environment or interactive behavior that combines virtual and real elements. Although XR is generally considered to include AR (augmented reality), VR (virtual reality), and MR (mixed reality), it has a special meaning in film, television, broadcasting, and live entertainment production. At present, XR has become the mainstream of the virtual broadcasting industry.

Why Is It So Popular Recently?

▼ Virtual environment expansion of XR virtual stage

In the industry description a few years ago, XR was more like a general description of AR and VR technology. It can be used to refer to an integrated system that includes the virtual and real worlds, and can even be equivalent to a specific motion capture technology, AR application, or VR game. But in the field of content production, it is specifically referred to as a workflow that includes LED displays, camera tracking systems, and powerful graphics engines. It is a virtual production method that places performers in the virtual world in real time without green screens and post-production processes.

▼ Sci-fi music stage created by XR

How Does XR Work In Production?

In XR production, high-quality LED screens (usually presented as ring screens or spherical screens) simultaneously display 3D virtual environments pre-loaded by graphics engines (usually using Unreal Engine or Notch), and these LEDs also form the background for filming or live events. When a precise camera tracking system is added, the camera begins to move around the real and virtual elements that are seamlessly combined in the LED environment, giving the audience who observes through the camera's perspective an illusion of fusion and immersion.

▼ Typical layout and installation of XR virtual stage

Benefits of XR production:

Immersive and real-time production and editing process

Actors, hosts, and producers can see the virtual environment on site, allowing them to adjust and guide performances or shooting on site, reducing the time and budget spent on post-production. Simply put, it is "post-production pre-positioning". In addition, virtual production solves the crew's transportation costs and environmental waiting costs for location shooting. Simply put, you can reproduce and modify the required environment at any time without waiting for the rain or sunset.

▼ Virtual weather broadcasting using XR technology

Real on-site interactive lighting:

The lighting is provided by the LED display screen on the XR site, which helps the seamless integration of real people and virtual environments and reduces the time for further lighting adjustments. This is particularly suitable for shooting real objects with high reflectivity, which requires real environmental reflection information and lighting without the need for post-synthesis.

▼ Interaction using tracked and positioned lighting props

No color overflow and matte synthesis:

In traditional green screen shooting environments, it often takes a lot of time to perform fine matte processing and fix color overflow problems, while XR does not require matte at all, and the scene is directly out of the lens.

▼ Switching from the XR virtual environment back to the real LED screen on site

Fast lens correction:

In traditional green screen shooting, the lens correction time of the virtual camera often takes several hours, while the correction of the virtual camera tracking system on the XR camera only takes a few minutes, which greatly reduces the interruption and preparation time on site and allows shooting across multiple scenes and session scenarios in a short time.

▼ Freely assembled and expandable LED modules

What Is The Connection And Difference Between XR And Virtual Production?

In fact, many people have doubts. Many XR screen recently are associated with virtual studios, evening parties, and virtual press conferences. Is XR more used for broadcasting activities? This is a wrong view. XR is not only used in live broadcasts and other activities, it can also be used for virtual production of film and television, namely VP (Virtual Production) that we often mentioned before. Virtual production can be simply regarded as the workflow of using VR and game technologies to shoot movies.

▼ LED virtual production site of the American TV series "The Mandalorian" Similar to the XR mentioned earlier, virtual production also relies on the background provided by the LED screen as the environment, but the ground is often built with real scenery materials, and all production processes are collected through cameras, without the need for expansion outside the LED screen. The XR virtual studio environment we often see has an LED floor screen, and requires virtual expansion synthesis of the LED external environment. In addition, virtual production relies heavily on the cooperation of the art department and virtual artists to create movie-level real and virtual environment quality.

▼ Typical XR virtual production LED and equipment installation site

In fact, virtual production relying on projection is not a new thing. As early as the 1930s to 1960s in Hollywood, many movies used rear projection technology. Actors stood in front of a screen with a picture projected on it, which was often used to shoot the moving environment outside the car window under a fixed lens. But at that time, it was only used as a supplement to conventional shooting due to the lack of a camera tracking system.

▼ Rear projection technology used by James Bond in the 007 movie

Industry application of XR:

Corporate activities:

Today, companies are increasingly considering XR as a new way to solve the problem of long-distance meetings. Virtual meetings, virtual presentations and even product launches can use XR technology to enhance the immersive experience of participants and make them truly feel the culture and concept of the brand.

▼ Guests remotely transmitted to the XR LED screen through green screen synthesis

▼ Indoor meeting virtual studio installation site

Education:

XR can immerse students in remote teaching in realistic scenes, allowing students to better understand and learn knowledge in simulated virtual situations and leave a deep impression.

▼ Display texture of ground LED to AR effect in camera

Movie:

xR provides opportunities for film shooting, reducing travel time for staff and contact risks during the epidemic. In post-production, LED screens can provide faster, easier and higher-quality alternatives to green screens, shortening production cycles and making more creative visual decisions in real time.

▼ LED virtual production site of "Solo: A Star Wars Story"

TV studio:

In the studio environment, LED screens provide high-resolution clear content and lighting in real time. Although cameras are needed to collect them in the end, the real-time content allows the host to better interact with the virtual environment instead of blindly guessing and stepping on the spot as before.

▼ Virtual studio of the xite Studio team

Music screen:

VJ artists can use XR to control and present live music videos, allowing the engine to generate real-time improvisations to create unlimited narrative possibilities and let the audience feel the wonderful world beyond the scope of LED.

▼ Application of XR technology on the VJ music interactive stage

Esports:

LED screens produce real reflections and shadows, which can reduce the workload of post-production. The more avant-garde and entertaining e-sports scene is very suitable for the assistance and interpretation of XR, allowing the virtual creatures and characters in the game to appear realistically in the arena, and the audience and performers can see the CG content and interact with it more realistically.

▼ XR synthetic live broadcast at Weplay Dota2 e-sports game

Main technologies of XR production:

The technologies used by XR are simplified as follows: camera tracking system, virtual studio software platform, media server, real-time rendering engine, etc.

Since the 1998 movie "Run Lola Run", the game industry has subtly influenced and infiltrated the film and television and new media industries, and even changed the audience's aesthetic habits, making modern people accustomed to the fast pace of games and more immersive subjective motion shots. Developers of mainstream game engines (Unity 3D, Unreal) are also ambitiously entering fields outside the game industry, such as animated films, performances, etc. Real-time rendering engines specifically for stage performances have also begun to appear (such as Notch).

▼ Virtual environment and interactive virtual props made with Notch engine

Virtual scenes are generally pre-made in Unreal Engine or Notch engine. Previously, it was necessary to model in DCC software such as C4D and Maya, import the engine, add lights and textures, set animations and do interactive tests in advance.

▼ Notch's operating interface and the performance of abstract particle effects

▼ Unreal Engine's operating interface and the performance of concrete scenes

From the engine's preference, Notch can provide better real-time performance of dynamic particles, suitable for high-dynamic abstract visual projects such as VJ, or used alone for the production of XR's foreground AR effects. Unreal is more inclined to produce concrete and complex scenes, with more packaged tools and plug-ins to assist, and can quickly create realistic effects.

▼ Image mapping of virtual and real spaces of XR stage

As virtual broadcasting technology becomes more diverse and complex, functions such as 3D real-time preview, stereoscopic projection (3D Mapping), real-time interactive effects, and multimedia file broadcast control have begun to be integrated through servers, and hardware devices with different functions can be seamlessly connected with servers.

▼ Media server and broadcast control related equipment on the XR stage

In XR virtual projects, the Disguise server and platform are mainly used at home and abroad. Its media server is equivalent to a bridge between the real world and the virtual world. It matches the motion information of the virtual camera with the real camera and maps the image information to the LED screen, and expands it outside the range. Its functions also include camera and lens correction, space mapping, color correction, and support for rendering engines such as Unreal, Unity, and Notch.

▼ Mo-sys's tracking and positioning system StarTracker

In the selection of tracking equipment, similar to green screen virtual production, equipment and service providers include: Mo-sys, Stype, BlackTrax, Optitrack, Vicon, etc., which are divided into active and passive infrared optical capture, and their tracking data and positioning adjustments can be integrated into the media server.

▼ OptiTrack optical motion capture system that has been acquired by a domestic company

In addition to the virtual studio software provided by Disguise, the team can also use virtual studio software platform solutions from other brands, such as Pixotope, Zero Density, Brain Storm, etc.

▼ The operation interface of Pixotope, a virtual studio software platform

▼ The flow chart of the single-camera solution structure of Pixotope, a virtual studio software platform

The Pixotope solution is mainly used in large-scale broadcasting, sports events, e-sports and other live broadcast fields, such as the "Super Bowl", the League of Legends finals, and the European Television Network.

▼ The spatial relationship between Pixotope software virtual scenes and real-life synthetic elements

Pixotope is an open software solution for quickly creating virtual studios, augmented reality, and real-time graphics. It has powerful commercial hardware designed specifically to connect with other partners' technologies and external data sources. Like Zero Density, Pixotope is a native virtual studio software solution based on Unreal Engine.

▼ Virtual reality show "Lost In Time"

At the opening ceremony of Riot Games' 2018 League of Legends World Championship held in Incheon, South Korea, the virtual idol K/DA girl group perfectly performed the realistic AR dance music "Pop Star" through the technology provided by Pixotope, attracting more than 90 million online viewers including the audience.

▼ League of Legends finals K/DA girl group virtual live broadcast

▼ League of Legends event virtual idol AR live broadcast held in Shanghai, China

In terms of the choice of LED brands, many movies, TV series or short films including "The Mandalorian", "First Man on the Moon" and "Westworld" mainly use LED products from domestic brands LEDFUL, such as the Black Pearl series and Diamond series, whose LED dot pitch ranges from 2.6mm to 5.2mm, and is selected according to the distance from the camera to the LED panel. In addition, the LED full-color small dot pitch series of domestic manufacturers LEDFUL is also the first choice for many virtual press conferences and film and television variety shows.

▼ ROE's BP2 and CB5 series LED screens were used in "The Mandalorian"

Virtual production of Disney + Star Wars series "The Mandalorian"

In 2019, Jon Favreau, the producer who finished filming "The Lion King", used XR technology for the first time in Disney's Star Wars series "The Mandalorian", shooting half of the scenes in the LED environment. Favreau's team built a 20-foot-high LED screen (270° ring screen) to provide actors and staff with photo-realistic environments for scenes such as frozen planets and barren deserts, saving a lot of time for set construction, prop switching, and traffic transfer in a 75-foot diameter space.

What is XR virtual production?

▼ Behind the scenes of the virtual production of the first season of "The Mandalorian"

Rapper Dave's live performance at the Brit Awards:

At the 2020 Brit Awards, Dave completed a fascinating virtual and real performance next to the piano set up by XR. With the help of Mo-Sys' camera tracking system, Notch's particle effects, and Disguise's projection server, realistic 3D animations were displayed on the piano. Combined with the precise camera movement around, the melody and visual experience of the whole song reached a climax.

▼ Behind-the-scenes footage of Dave's performance

Katy Perry's performance at the American Idol finals:

The live broadcast of the American Idol finals was produced and implemented by the technologically advanced XRStudios team for XR projects, using Disguise's media server, and the special effects were produced by the Silent Partner Studios team in Notch VFX. Many people are curious about this performance

XR screens can seamlessly blend into the surrounding environment and provide stunning visual displays. By leveraging the unique features and advantages of LED screens, you can create an engaging and immersive experience.If you want to learn more, please contact LEDFUL.