Tempotokens
[AAAI 2024] The official PyTorch implementation of "Diverse and Aligned Audio-to-Video Generation via Text-to-Video Model Adaptation"
Last verified:
What is Tempotokens?
TempoTokens is a PyTorch implementation for diverse and aligned audio-to-video generation via text-to-video model adaptation. Developed by Guy Yariv, Itai Gat, Sagie Benaim, Lior Wolf, Idan Schwartz, and Yossi Adi at the Adi Yoss Lab at Hebrew University of Jerusalem, this tool generates diverse and realistic videos guided by natural audio samples from a wide variety of semantic classes. The videos are aligned both globally (semantically associated with the entire output video) and temporally (each audio segment corresponds to a video segment) with the input audio.
The system uses a lightweight adaptor network that learns to map audio-based representations to the input representation expected by text-to-video generation models. It leverages an existing text-conditioned video generation model and a pre-trained audio encoder (BEATs). Uniquely, it enables video generation conditioned on text only, audio only, and for the first time, on both text and audio simultaneously. The method was validated extensively on three datasets: VGGSound, Landscape, and AudioSet-Drum.
This tool is designed for researchers and developers working in generative AI, audio-visual synthesis, deep learning, and diffusion models. It is particularly useful for those creating audio-driven video content, studying audio-visual alignment, or building applications that require synchronized audio-video generation such as animation, content creation, and multimedia research.
The repository includes pre-trained weights, training scripts, inference code, and configuration files. It also introduces AV-Align, a novel evaluation metric based on detection and comparison of energy peaks in both modalities to assess alignment quality between generated videos and input audio.
Tempotokens pricing
Pricing model: Freemium
Free and open-source under MIT license. The code, pre-trained weights, and samples are available at no cost. Users only need their own computational resources (GPU with CUDA) and to download the BEATs pre-trained model from the provided link.
Tempotokens pros
- Generates videos better aligned with input sound on content and temporal axis
- Higher visual quality compared to recent state-of-the-art approaches
- Produces more diverse video outputs
- Supports text-only, audio-only, and combined text-audio conditioning
- First method to enable combined text and audio conditioning for video generation
- Lightweight adaptor network requires less computational resources
- Pre-trained weights available for three datasets (VGGSound, Landscape, AudioSet-Drum)
- Novel AV-Align evaluation metric for audio-video alignment assessment
- Open-source under MIT license
- Official PyTorch implementation with complete code and samples
- Works with ModelScope text-to-video base model
- Can generate videos from user's own audio files
- Supports LoRA for parameter-efficient fine-tuning
- Includes XFormers and SDP attention for memory efficiency
- Watermark removal option with LAMA inpainting
Tempotokens cons
- Requires GPU with CUDA for inference and training
- Needs to download large BEATs pre-trained model separately
- Only generates short video clips matching audio duration
- Limited to datasets it was trained on without retraining
- Requires accelerate config initialization before use
- Multiple configuration files to review and customize
- No pre-built Docker container or one-click deployment
- Limited documentation beyond README and paper
Frequently asked questions about Tempotokens
What is TempoTokens?
TempoTokens is a PyTorch implementation for diverse and aligned audio-to-video generation. It uses a lightweight adaptor network to map audio representations to text-to-video model inputs, enabling video generation conditioned on audio, text, or both simultaneously.
What datasets is TempoTokens trained on?
The method was validated on three datasets: VGGSound, Landscape, and AudioSet-Drum. Pre-trained weights are available for all three datasets.
Can I generate videos from my own audio files?
Yes, the inference.py script allows you to generate videos from your own audio files by specifying the --audio_path parameter with the path to your audio file.
What audio formats are supported?
The tool uses the BEATs audio encoder which processes audio input. The specific supported formats depend on BEATs encoder compatibility, typically including common formats like WAV.
What is AV-Align?
AV-Align is a novel evaluation metric proposed by the authors to assess the alignment of generated videos with input audio samples. It is based on the detection and comparison of energy peaks in both audio and video modalities.
Do I need to train the model myself?
No, pre-trained weights are available for download from Google Drive for all three datasets (VGGSound, Landscape, AudioSet-Drum). You can use these directly for inference without training.
What are the system requirements?
You need Python, PyTorch, CUDA-compatible GPU for training and inference, and the accelerate library. The BEATs pre-trained model must also be downloaded separately.
Can I use text prompts with audio?
Yes, TempoTokens supports video generation conditioned on text only, audio only, and jointly on both text and audio simultaneously, which is a first in the field.
How do I install TempoTokens?
Clone the repository, run pip install -r requirements.txt, and initialize an accelerate environment with accelerate config. Then download the BEATs pre-trained model and optionally the pre-trained weights.
What license is TempoTokens under?
The repository is released under the MIT license, allowing free use, modification, and distribution for both personal and commercial projects.