A banner year for AI-generated video

This year has seen rapid advances in AI video generation. Tools such as OpenAI’s Sora, Google DeepMind’s Veo 3, and Runway’s Gen-4 can produce clips that sometimes rival real filmed footage or CGI. Even mainstream media has started to use the technology, with Netflix deploying an AI visual effect in The Eternaut for the first big mass market TV use case. With access in apps like ChatGPT and Gemini for subscribers, casual creators as well as professionals are experimenting and producing striking results.

How a user typically generates a clip

Most people interact with video generation through an app or a website. A typical workflow is prompt driven: you give a model a text prompt, review the result, and iterate. Outputs are often hit or miss; the model may need many passes before producing something that matches the intent. Demo reels show peak results, but everyday feeds also fill with low quality or misleading footage. Video generation is also energy intensive, consuming far more computation than text or image generation.

Diffusion models in a nutshell

Diffusion models work by learning to reverse a process of adding noise. During training the model sees images progressively corrupted by random pixels and learns how to undo each corruption step. To generate an image, the model starts from random static and iteratively cleans it into something that resembles the images it was trained on.

Text guidance is added by pairing the diffusion model with a separate model that maps text to images, typically a large language model or a multimodal model trained on huge datasets of text paired with images or video. That guide steers each denoising step toward outputs that match the prompt.

Extending diffusion to video

Generating video means dealing with sequences of frames rather than single images. Naive frame by frame generation tends to produce artifacts where objects or lighting jump between frames. The solution is to add temporal consistency into the generation pipeline so that objects persist and motion looks coherent.

Latent diffusion makes it tractable

Raw video frames contain millions of pixels and demand enormous compute. Latent diffusion reduces the cost by compressing frames into a smaller mathematical representation called latent space. The diffusion process then operates on these compact encodings rather than on raw pixels, much like how video streaming uses compression to move data efficiently. Once the latent frames are produced, a decoder decompresses them back into watchable video. Latent diffusion is more efficient than operating directly on pixel data, but video generation remains computationally heavy.

Why transformers matter for video

To maintain consistency across frames, the latest systems combine diffusion with transformers. Transformers excel at processing long sequences, which is why they power modern large language models. For video, models break the clip into spatiotemporal chunks. As Tim Brooks, a lead researcher on Sora, put it, “It is like if you were to have a stack of all the video frames and you cut little cubes from it.” Using transformers to process those chunks helps the model keep objects, lighting, and camera motion coherent across time. This approach also allows training on varied video formats, from vertical phone clips to widescreen films, improving robustness and flexibility.

Adding audio: Veo 3 and synchronized sound

A major step forward is generating audio together with video. Google DeepMind’s Veo 3 can produce synchronized audio and visuals, including lip synced dialogue, sound effects, and background noise. The key innovation was compressing audio and video into a single joint representation so the diffusion process operates on both modalities jointly. That lockstep generation helps ensure that sound aligns with the images.

Blurring model boundaries and the road ahead

Diffusion methods are commonly used for images, audio, and video, while transformers remain dominant for text. But the boundaries are shifting. Researchers have started experimenting with diffusion for text generation, and transformers are now integral to video diffusion pipelines. Diffusion methods can be more efficient than transformers in some scenarios, so we may see more hybrid architectures and new efficiency gains. Expect continued improvements in realism, format versatility, and multimodal alignment, alongside ongoing debates about energy use, ethics, and training data provenance.