Connected & Protected
24/7 Support
HomeBlogGenerating audio for video
AI & Machine Learning
#video-to-audio technology
#generate audio for video
#soundtracks for silent films
#audiovisual generation
#video generation models
#creative video soundscapes
#text prompts for audio
#rich sound design
#immersive video experiences
#natural language audio generation
#silent video enhancement
#V2A technology
#audio generation for video
#synchronized soundtracks
#generative media

Generating audio for video

Video-to-audio research uses video pixels and text prompts to generate rich soundtracks

G
Google DeepMind News
Content Author
Content Specialist

The Generative Media team at Google DeepMind is at the forefront of AI research, specializing in innovative techniques for audiovisual content creation and enhancement.

Expertise: AI in media generation, audio and video integration, generative models, creative technology applications
Published: June 17, 2024
Updated: Dec 31, 2025
12 min read

Quick Answer

Video-to-audio technology enables the generation of synchronized soundtracks for silent videos by combining visual data with natural language prompts, enhancing the viewing experience.

TL;DR

This post explores the advancements in video-to-audio (V2A) technology, which generates soundtracks for silent videos using video pixels and text prompts. It opens new creative opportunities by pairing with video generation models for realistic and immersive audio experiences.

Key Takeaways

  • 1V2A technology creates soundtracks for previously silent videos.
  • 2It combines video data with natural language prompts for audio generation.
  • 3The technology allows for creative possibilities with traditional footage.
Generating audio for video
June 17, 2024 Research

Generating audio for video

Generative Media team

Share

Video-to-audio research uses video pixels and text prompts to generate rich soundtracks

Video generation models are advancing at an incredible pace, but many current systems can only generate silent output. One of the next major steps toward bringing generated movies to life is creating soundtracks for these silent videos.

Today, we're sharing progress on our video-to-audio (V2A) technology, which makes synchronized audiovisual generation possible. V2A combines video pixels with natural language text prompts to generate rich soundscapes for the on-screen action.

Our V2A technology is pairable with video generation models like Veo to create shots with a dramatic score, realistic sound effects or dialogue that matches the characters and tone of a video.

It can also generate soundtracks for a range of traditional footage, including archival material, silent films and more — opening a wider range of creative opportunities.

Prompt for audio: Cinematic, thriller, horror film, music, tension, ambience, footsteps on concrete

Prompt for audio: Cute baby dinosaur chirps, jungle ambience, egg cracking

Prompt for audio: Jellyfish pulsating under water, marine life, oce

an

Prompt for audio: A drummer on a stage at a concert surrounded by flashing lights and a cheering crowd

Prompt for audio: Cars skidding, car engine throttling, angelic electronic music

Prompt for audio: A slow mellow harmonica plays as the sun goes down on the prairie

Prompt for audio: Wolf howling at the moon

Enhanced creative control

Importantly, V2A can generate an unlimited number of soundtracks for any video input. Optionally, a ‘positive prompt’ can be defined to guide the generated output toward desired sounds, or a ‘negative prompt’ to guide it away from undesired sounds.

This flexibility gives users more control over V2A’s audio output, making it possible to rapidly experiment with different audio outputs and choose the best match.

Prompt for audio: A spaceship hurtles through the vastness of space, stars streaking past it, high speed, Sci-fi

Prompt for audio: Ethereal cello atmosphere

Prompt for audio: A spaceship hurtles through the vastness of space, stars streaking past it, high speed, Sci-fi

How it works

We experimented with autoregressive and diffusion approaches to discover the most scalable AI architecture, and the diffusion-based approach for audio generation gave the most realistic and compelling results for synchronizing video and audio information.

Our V2A system starts by encoding video input into a compressed representation. Then, the diffusion model iteratively refines the audio from random noise. This process is guided by the visual input and natural language prompts given to generate synchronized, realistic audio that closely aligns with the prompt. Finally, the audio output is decoded, turned into an audio waveform and combined with the video data.

Diagram of our V2A system, taking video pixel and audio prompt input to generate an audio waveform synchronized to the underlying video. First, V2A encodes the video and audio prompt input and iteratively runs it through the diffusion model. Then it generates compressed audio, which is decoded into an audio waveform.

To generate higher quality audio and add the ability to guide the model towards generating specific sounds, we added more information to the training process, including AI-generated annotations with detailed descriptions of sound and transcripts of spoken dialogue.

By training on video, audio and the additional annotations, our technology learns to associate specific audio events with various visual scenes, while responding to the information provided in the annotations or transcripts.

Further research underway

Our research stands out from existing video-to-audio solutions because it can understand raw pixels and adding a text prompt is optional.

Also, the system doesn't need manual alignment of the generated sound with the video, which involves tediously adjusting different elements of sounds, visuals and timings.

Still, there are a number of other limitations we’re trying to address and further research is underway.

Since the quality of the audio output is dependent on the quality of the video input, artifacts or distortions in the video, which are outside the model’s training distribution, can lead to a noticeable drop in audio quality.

We’re also improving lip synchronization for videos that involve speech. V2A attempts to generate speech from the input transcripts and synchronize it with characters' lip movements. But the paired video generation model may not be conditioned on transcripts. This creates a mismatch, often resulting in uncanny lip-syncing, as the video model doesn’t generate mouth movements that match the transcript.

Prompt for audio: Music, Transcript: “this turkey looks amazing, I’m so hungry”

Our commitment to safety and transparency

We’re committed to developing and deploying AI technologies responsibly. To make sure our V2A technology can have a positive impact on the creative community, we’re gathering diverse perspectives and insights from leading creators and filmmakers, and using this valuable feedback to inform our ongoing research and development.

We’ve also incorporated our SynthID toolkit into our V2A research to watermark all AI-generated content to help safeguard against the potential for misuse of this technology.

Before we consider opening access to it to the wider public, our V2A technology will undergo rigorous safety assessments and testing. Initial results are showing this technology will become a promising approach for bringing generated movies to life.

Note: All examples are generated by our V2A technology, which is paired with Veo, our most capable generative video model.

Acknowledgements

This work was made possible by the contributions of: Ankush Gupta, Nick Pezzotti, Pavel Khrushkov, Tobenna Peter Igwe, Kazuya Kawakami, Mateusz Malinowski, Jacob Kelly, Yan Wu, Xinyu Wang, Abhishek Sharma, Ali Razavi, Eric Lau, Serena Zhang, Brendan Shillingford, Yelin Kim, Eleni Shaw, Signe Nørly, Andeep Toor, Irina Blok, Gregory Shaw, Pen Li, Scott Wisdom, Aren Jansen, Zalán Borsos, Brian McWilliams, Salah Zaiem, Marco Tagliasacchi, Ron Weiss, Manoj Plakal, Hakan Erdogan, John Hershey, Jeff Donahue, Vivek Kumar, and Matt Sharifi.

We extend our gratitude to Benigno Uria, Björn Winckler, Charlie Nash, Conor Durkan, Cătălina Cangea, David Ding, Dawid Górny, Drew Jaegle, Ethan Manilow, Evgeny Gladchenko, Felix Riedel, Florian Stimberg, Henna Nandwani, Jakob Bauer, Junlin Zhang, Luis C. Cobo, Mahyar Bordbar, Miaosen Wang, Mikołaj Bińkowski, Sander Dieleman, Will Grathwohl, Yaroslav Ganin, Yusuf Aytar, and Yury Sulsky.

Special thanks to Aäron van den Oord, Andrew Zisserman, Tom Hume, RJ Mical, Douglas Eck, Nando de Freitas, Oriol Vinyals, Eli Collins, Koray Kavukcuoglu and Demis Hassabis for their insightful guidance and support throughout the research process.

We also acknowledge the many other individuals who contributed across Google DeepMind and our partners at Google.

Veo

SynthID

Expert Analysis for Orange County Businesses

How this impacts SMBs in Southern California

Tech News Roundup: Generating Audio for Video

The recent advancements in video-to-audio (V2A) technology represent a significant leap in content creation, particularly for small and medium businesses (SMBs) in Orange County and Southern California. As video marketing continues to dominate digital strategies, the ability to create rich, synchronized soundtracks using AI can enhance the overall production quality of promotional materials. This technology not only streamlines the production process but also allows businesses to convey emotions and narratives more effectively, making it an ideal tool for brands looking to stand out in a competitive market.

For local businesses aiming to leverage this innovative technology, TNC offers a suite of managed IT services that can support the infrastructure needed for video production and editing. Our cybersecurity solutions ensure that your creative assets remain protected during the production process, while our cloud backup and BCDR services keep your data safe and accessible. As you consider integrating AI-driven solutions into your marketing efforts, TNC is here to provide the support you need to maximize your technological investments.

Read the full article: Original Source Link

Frequently Asked Questions

What is video-to-audio (V2A) technology?

V2A technology generates soundtracks for silent videos by merging video pixels with text prompts.

How can V2A enhance video content?

It provides synchronized soundtracks, improving the overall audiovisual experience of videos.

Content Quality

Last Reviewed:December 27, 2025
Fact-Checked:December 27, 2025
Sources:3 verified sources
Word Count:1,060 words

Article Tags

video-to-audio technology
generate audio for video
soundtracks for silent films
audiovisual generation
video generation models
creative video soundscapes
text prompts for audio
rich sound design
immersive video experiences
natural language audio generation
silent video enhancement
V2A technology
audio generation for video
synchronized soundtracks
generative media
G

About Google DeepMind News

Content Author

The Generative Media team at Google DeepMind is at the forefront of AI research, specializing in innovative techniques for audiovisual content creation and enhancement.

Expertise: AI in media generation, audio and video integration, generative models, creative technology applications

Ready to Transform Your Business?

Discover how our expert IT solutions can help your business thrive.