Virtual-real synthesis tech stuns viewers of Spring Festival Gala

The 2024 Spring Festival Gala, a lavish celebration produced and broadcast by China Media Group (CMG) on Friday night, featured a fusion of multiple cutting-edge technologies, including extended reality (XR), virtual production (VP) and artificial intelligence (AI).

By introducing an ultra-high-definition production system that integrates XR and VP, the gala was staged via new shooting methods, presenting a lifelike world merging virtual and real scenes. These technologies enabled the production of new visualizations where physical and digital objects co-existed in real time, creating an immersive experience for the audience.

For the first time, an immersive stage interaction system was deployed in the studio hall. The free-view ultra-high-definition shooting system captured actors' movements and preserved wonderful moments in an all-round way, bringing the audience a stunningly realistic visual impression by leveraging AI production and augmented reality (AR) real-time rendering technologies.

Poet Li Bai 'comes' to sub-venue

Li Bai, a Chinese legendary poet from the Tang Dynasty (618-907), "came" to the sub-venue in Xi'an, capital of northwest China's Shaanxi Province, thanks to the XR and VP technologies.

The poet walked in the local street, flew in the night sky with cranes and stood on the top of ancient buildings chanting "the water of the Yellow River comes from the sky." Thanks to the immersive stage interaction system, Li also interacted with a Chinese actor and poured liquor.

The team adopted the animation-reality synthesis and XR real-time rendering technologies to present the realistic effect. They changed the postures of the virtual figure of Li and adjusted the light and shadow renderings in the animation to achieve a perfect match with the reality, said Jiang Yuhao, visual director of the 2024 Spring Festival Gala. 

Viewers in Xi'an could even experience the same view as presented in the gala by scanning local building outlines with their mobile phones, and interact with the poet, said Jiang.

Koi fish 'swim' on the stage

In a performance called "Koi fish," dancers moved gracefully in illusory rippling water, blending into the background scenery. The studio hall was transformed into an underwater world.

Using the XR rendering system, an underwater immersive scene was built by fusing and coordinating the virtual light with the real lighting effect in the studio, said Fan Xiaoxuan, senior engineer of CMG's Technology Bureau.

"The system generates an underwater space where virtual and real light and shadow are closely blended, creating a realistic perspective," said Fan.

When the performers danced like fish swimming nimbly, audience could see many "ripples" under their feet, following their steps and changing shapes and positions. 

The special effects were realized by trackers, according to Fan.

"The performers wore signal trackers which sent their position and posture data to the optical tracking modules. The modules, after capturing the information, create the dynamic special effects coupling the performance and the virtual ripples," he said.