ModelScope Text-To-Video
The ModelScope Text To Video Synthesis tool is a machine learning application developed by the community of developers at Hugging Face. Thi...
Last verified:
Visit ModelScope Text-To-Video
What is ModelScope Text-To-Video?
ModelScope Text To Video is an open-source text-to-video generation tool developed by Alibaba's DAMO Academy. It uses a multi-stage diffusion model that inputs English text descriptions and returns video clips that match the text. The model consists of three sub-networks: text feature extraction, text feature-to-video latent space diffusion, and video latent space to video visual space conversion, with approximately 1.7 billion parameters overall.
Key features include Unet3D architecture for video generation through iterative denoising from pure Gaussian noise, support for generating videos up to 25 seconds long on GPUs with 16GB VRAM, temporal consistency through spatio-temporal blocks, and the ability to handle imaginative prompts like
ModelScope Text-To-Video pricing
Pricing model: Free
The ModelScope Text To Video model is open-source and free for research purposes. It is available on Hugging Face Spaces with a free tier that has slow waiting times for users without paid accounts. For faster generation, users can create a Hugging Face account, add credit card details, and duplicate the space with their own cloud GPU render (costs approximately $1-$4 per hour, with users reporting $2 for half an hour). The model can also be deployed locally using the provided Code examples and Colab notebook at no additional cost beyond hardware requirements. There are no official paid plans from DAMO Academy - pricing only applies when using paid GPU infrastructure through Hugging Face's Inference Providers or duplicating spaces.
ModelScope Text-To-Video pros
- Open-source model available for free research use
- Only 1.7 billion parameters - relatively lightweight
- Generates videos up to 25 seconds on 16GB VRAM GPUs
- Unet3D architecture provides smooth temporal transitions
- Spatio-temporal blocks ensure frame consistency
- Accepts arbitrary imaginative English text prompts
- Available on Hugging Face Spaces for easy online access
- Also deployed on ModelScope Studio platform
- Simple Pipeline API for ModelScope framework users
- Works with standard Python packages (pip installable)
- Can be deployed locally via Google Colab
- Outputs MP4 format videos compatible with most players
- 576x1024 resolution suitable for many use cases
- Adapts to varying frame numbers during training/inference
- Superior performance over state-of-the-art methods on evaluation metrics
- VAE slicing and CPU offloading optimize memory usage
- Wide range of creative applications possible
- Code and online demo publicly available
ModelScope Text-To-Video cons
- Only English input supported - no other languages
- Designed for research purposes only, not commercial use
- Cannot achieve film and television quality generation
- Cannot generate clear or readable text in videos
- Performance limited on complex compositional generation tasks
- Generated videos have noticeable Shutterstock-style watermark
- Requires 16GB CPU RAM and 16GB GPU RAM minimum
- Only supports inference on GPU - no CPU option
- Free Hugging Face account has very slow waiting times
- Generated results may have biases from training data distribution
- Videos can appear overtrained with shuttering artifacts
- Prompts must be kept short for proper movement
- Not trained to realistically represent people or events
- Prohibited for pornographic, violent, or harmful content
- 14 frames default may feel choppy for some uses
Frequently asked questions about ModelScope Text-To-Video
What languages does ModelScope Text To Video support?
The model only supports English input. It is mainly trained with English corpus and does not support other languages at the moment. This is because the model was trained primarily on English datasets like Webvid, LAION5B, and ImageNet.
What is the maximum video length I can generate?
The model can generate videos up to 25 seconds long on GPUs with 16GB VRAM. The default output is typically 14 frames at 576x1024 resolution, but the model can adapt to varying frame numbers during training and inference.
What hardware do I need to run this model locally?
The demo requires approximately 16GB CPU RAM and 16GB GPU RAM. The model currently only supports inference on GPU, not CPU. For local deployment, you need to install modelscope==1.4.2, open_clip_torch, and pytorch-lightning via pip.
Can I generate videos with text or words in them?
No, the model cannot generate clear or readable text in videos. This is one of the documented model limitations. Attempts to generate text content will not produce legible results.
Is this model suitable for commercial use?
No, this model is meant for research purposes only. It is not trained for commercial applications, and the documentation explicitly states it should not be used to realistically represent people or events, nor for pornographic, violent, or harmful content generation.
How do I access the model online without installing it?
You can access the model directly through the Hugging Face Spaces demo at https://huggingface.co/spaces/damo-vilab/modelscope-text-to-video-synthesis or through ModelScope Studio. Both platforms allow you to experience the model without local installation.
What video format does the model output?
The model outputs MP4 format video clips that match the input text description. The videos are typically 576x1024 resolution with 14 frames, compatible with most video players and editing software.
Why are my generated videos slow on Hugging Face?
Hugging Face generation is super slow for free accounts. Waiting times are very slow with the free account. To speed up generation, you need to create a Hugging Face account, add credit card details, duplicate the space, and choose your own cloud GPU render, which costs approximately $1-$4 per hour.
What are the main limitations of this model?
Key limitations include: only English support, cannot achieve film/TV quality, cannot generate clear text, limited performance on complex compositional tasks, visible Shutterstock-style watermark on videos, potential biases from training data distribution, and requires GPU-only inference with 16GB minimum VRAM.
How do I use the model with the ModelScope Pipeline?
Under the ModelScope framework, call a simple Pipeline where input must be in dictionary format with the legal key 'text' containing short text. Example: pipe({'text': 'A panda eating bamboo on a rock.'})[OutputKeys.OUTPUT_VIDEO]. The model requires GPU and the input must be English text.