W.A.L.T Video Diffusion

W.A.L.T is a transformer-based method for photorealistic video generation via diffusion modeling. It uses a causal encoder to compress images and videos into a unified latent space, and a window attention architecture for joint spatial and spatiotemporal generative modeling.

This design allows for top performance on video (UCF-101 and Kinetics-600) and image (ImageNet) generation benchmarks without classifier free guidance. We also use a three-model cascade for text-to-video generation, producing 512 x 896 resolution videos at 8 frames per second.