Meta announces new AI model for generating video with sound

Facebook owner Meta announced on Friday that it had developed a new artificial intelligence model called Movie Gen, capable of creating realistic video and audio clips in response to user prompts. The company claims it can rival tools from leading media generation startups like OpenAI and ElevenLabs.

Samples of Movie Gen's creations, provided by Meta, showcased videos of animals swimming and surfing, as well as clips using real photos of people depicting them performing actions like painting on a canvas.

According to Meta, Movie Gen can also generate background music and sound effects synchronized with the video content, and it can edit existing videos.

Videos created by Movie Gen can be up to 16 seconds long, while audio clips can reach up to 45 seconds. Meta shared data from blind tests indicating that the model performs favorably compared to offerings from startups, including Runway, OpenAI, ElevenLabs, and Kling.

The announcement comes as Hollywood grapples with how to harness generative AI video technology this year, following Microsoft-backed OpenAI's demonstration in February of its product Sora, which can create feature film-like videos in response to text prompts.

Technologists in the entertainment industry are eager to utilize such tools to enhance and expedite filmmaking, while others express concerns about systems that seem to have been trained on copyrighted works without permission.

Lawmakers have also raised concerns about how AI-generated fakes, or deepfakes, are being used in elections worldwide, including in the U.S., Pakistan, India and Indonesia.

Meta spokespeople indicated that the company is unlikely to release Movie Gen for open use by developers, unlike its Llama series of large-language models, as they assess the risks associated with each model individually. They declined to comment on the specific assessment of Movie Gen.

Instead, they stated that Meta is working directly with the entertainment community and other content creators on applications for Movie Gen and plans to incorporate it into Meta's own products sometime next year.

According to the blog post and a research paper released by Meta, the company used a combination of licensed and publicly available datasets to develop Movie Gen.

OpenAI has been meeting with Hollywood executives and agents this year to discuss potential partnerships involving Sora, though no deals have been reported yet. Concerns over the company's approach intensified in May when actor Scarlett Johansson accused the ChatGPT maker of imitating her voice without permission for its chatbot.

Lionsgate Entertainment, the company behind "The Hunger Games" and "Twilight," announced in September that it was giving AI startup Runway access to its film and television library to train an AI model. In return, it stated that the studio and its filmmakers can use the model to augment their work.