AI technology is giving old Chinese movies a brand-new look and lending film restorers a helping hand to sail through an otherwise slow and tedious process.
Before the age of digitization, film preservation was a mammoth undertaking involving cleaning, repairing and patching together the best surviving pieces of a film from bygone eras. Technicians needed to manually transform footage recorded on a film stock into a digital format – frame by frame. It was time-consuming and laborious.
A frame is one of many single images, which when put in sequence create a moving picture.
"Traditionally, a technician could spend a whole day repairing only around 200 frames. One second of footage contains 25 to 30 frames. So you can imagine how long it would take to repair a movie," said Zhang Xiaoyun, an associate professor at the Cooperative Medianet Innovation Center of Shanghai Jiaotong University.
Comparison between old film (L) and repaired one (R). /Shanghai Jiaotong University
But AI helps streamline and optimize the task. Algorithms can be trained to comb through considerable amounts of materials, analyze them and extrapolate their knowledge to layer old movies with missing data and other inputs.
"The AI restoration enhancing technology we're developing is totally data-driven. By learning from tons of materials, the model knows what old films and repaired ones are like. It can automatically repair films through its algorithm," Zhang told CGTN.
"It's much more efficient than traditional methods," she added.
AI can reduce auditory and visual noise, sharpen colors, insert extra frames for smoother animation and enhance the resolution.
In 2006, China launched a project to restore 40,000 old movies shot on celluloid tape. Several film institutions and streaming websites, such as iQiyi, Youku, along with China Film Group Corporation, have each developed AI-powered image processing systems.
The Cooperative Medianet Innovation Center of Shanghai Jiaotong University has cooperated with China Media Group (CMG) in recent years to repair classic films and songs, most of which now meet the 4K or 8K ultra high-definition broadcast requirements.
More than technology
Comparison between old film (L) and repaired one (R). /Shanghai Jiaotong University
Archived films can sustain damages of different kinds. Scratches are common, and the plastic strips on which moving images are captured tend to lose their quality and even warp over time. All this makes the restoration work more difficult as results largely depend on the condition of the materials at hand.
The better and more detailed the footage, the likelier the success of the AI restoration, Zhang said. But if the film is heavily damaged, repairing it is almost mission impossible.
Film restoration is not just about technology, but also about art and history. Technicians can't rely solely on machines to salvage classic movies, especially when it comes to colorization.
The way AI turns a black-and-white film into a technicolor version or liven up the colors of an aged movie is pure guesswork, built on its knowledge of real-world objects. At times, the algorithm gets it right. At other times, it misses the mark.
Zhang expressed hope that artists and film directors can get involved in the restoration work.
"They can provide us with some colorization requirement for one keyframe, for example. Then we'll know how to repair the rest," she explained.
"AI is not omnipotent. We hope that in the future, AI will do 99 percent of the work, while humans only need to cover the remaining one percent," Zhang said.
Domestic GPU platforms urgently needed
Comparison between old film (L) and repaired one (R). /Shanghai Jiaotong University
The AI restoration of old films requires a special type of chips, called graphics processing units (GPUs), for video rendering and image editing. These cards were originally used in video games to enhance their visuals, but are now used in a variety of applications, including mining of cryptocurrencies and AI film restoration.
Currently, Chinese film restorers import most of the GPUs, with the U.S. company Nvidia dominating the market.
The Cooperative Medianet Innovation Center currently has about 200 students whose main work is to run tests and develop new AI models, including pre-treating, decoding, enhancing and output encoding footage. "We have around 100 GPU training cards – mostly imported ones. That can't meet the large practice and research work of the whole center," Zhang said.
Huawei offers technical support at the Cooperative Medianet Innovation Center of the Shanghai Jiaotong University. /CGTN
The center now works with Chinese tech firm Huawei, which provides them with its Shengteng GPU platform and a server that can support eight GPU cards.
"Through the cooperation, we and Huawei hope to develop our domestic GPU platform and make full use of our GPU resources," Zhang said.
"AI film restoration is not a solo work but requires team effort," professor Wang Yanfeng told CGTN during a separate interview.
Wang, who is the deputy director of the Shanghai Artificial Intelligence Research Institute, said at present Shanghai Jiaotong University is developing AI restoration enhancing technology, and Shanghai Yunshi Science and Technology Co. Ltd is responsible for restoration, both manually and automatically, while Huawei is providing GPU servers. All the materials will eventually be sent to CMG for post-production.
(Edited by Nadim Diab, Video by Zeng Hongen)