Eagle
Eagle: Frontier Vision-Language Models with Data-Centric Strategies
Last verified:
What is Eagle?
Eagle is a family of frontier vision-language models (VLMs) developed by NVIDIA Research for long-context multimodal learning. The latest version, Eagle 2.5, is specifically designed to address challenges in long video comprehension and high-resolution image understanding, supporting up to 512 video frames and 4K HD image inputs. The model family includes multiple sizes from 1B to 34B parameters, with Eagle 2.5-8B being the flagship model that matches or beats larger models like Qwen2.5-VL-72B on key datasets.
Key features include Information-First Sampling with Image Area Preservation and Automatic Degrade Sampling, progressive mixed post-training expanding from 32K to 128K context length, and a diversity-driven data recipe combining open-source data with the self-curated Eagle-Video-110K dataset containing over 110K annotated samples. The model supports various input types including pure text, single/multiple images, video sequences, and multi-page documents.
Eagle is designed for researchers, developers, and organizations working on multimodal AI applications requiring long-form visual reasoning. It excels at tasks like video QA, localization, summarization, document understanding, OCR, chart analysis, and embodied AI benchmarks. The model is fully open-source with data and training recipes available, making it suitable for the open-source community aiming to develop competitive VLMs.
The model achieves state-of-the-art results on 6 out of 10 long video benchmarks, outperforms GPT-4o on 3/5 video tasks, outperforms Gemini 1.5 Pro on 4/6 video tasks, and achieves 72.4% on Video-MME with 512 input frames. It includes efficiency optimizations like Triton-based fused operators, GPU memory optimization, distributed context parallelism, and vLLM deployment support.
Eagle pricing
Pricing model: Freemium
Eagle is completely free and open-source. The code is released under Apache 2.0 license, and pretrained model weights are released under the NVIDIA License. The service is a research preview intended for non-commercial use only. No paid tiers or subscription plans exist - users can download models from Hugging Face and deploy locally at no cost.
Eagle pros
- State-of-the-art on 6 out of 10 long video benchmarks
- Supports up to 512 video frames for long-context understanding
- Handles 4K HD high-resolution image inputs
- 128K token context length with progressive training
- Fully open-source with data and training recipes available
- Outperforms GPT-4o on 3/5 video tasks
- Outperforms Gemini 1.5 Pro on 4/6 video tasks
- Matches Qwen2.5-VL-72B performance with only 8B parameters
- Eagle-Video-110K dataset with 110K+ annotated samples included
- Triton-based fused operators for GPU memory optimization
- Supports vLLM deployment for accelerated inference
- Multiple model sizes from 1B to 34B parameters available
- Information-First Sampling preserves visual details
- Supports text, images, video, and multi-page documents
- Apache 2.0 license for code usage
Eagle cons
- Research preview intended for non-commercial use only
- Requires specific transformers version 4.37.2 (not latest)
- Needs flash-attn installation for proper functionality
- Large model sizes require significant GPU memory
- Latest transformers version not compatible with model
- 8B model still needs multiple GPUs for optimal performance
- Video decoding requires decord dependency
- Training recipes require substantial computational resources
- Model weights under NVIDIA License with usage restrictions
Frequently asked questions about Eagle
What is Eagle and what does it do?
Eagle is a family of frontier vision-language models (VLMs) developed by NVIDIA Research for long-context multimodal learning. It excels at long video comprehension (up to 512 frames), high-resolution image understanding (up to 4K HD), and processes text, images, videos, and multi-page documents. The model achieves state-of-the-art results on long video benchmarks and matches much larger models like Qwen2.5-VL-72B with only 8B parameters.
What are the different Eagle model sizes available?
The Eagle family includes multiple model sizes: Eagle 2.5-8B (latest flagship with Qwen2.5-7B-Instruct backbone, 128K context), Eagle2-1B (Qwen2.5-0.5B-Instruct, 16K context), Eagle2-2B (Qwen2.5-1.5B-Instruct, 16K context), Eagle2-9B (Qwen2.5-7B-Instruct with Siglip+ConvNext, 16K context), and Eagle2-34B (Qwen2.5-32B-Instruct, 16K context). All models are available on Hugging Face.
How does Eagle 2.5 compare to GPT-4o and Gemini 1.5 Pro?
Eagle 2.5-8B outperforms GPT-4o (0806) on 3 out of 5 video tasks and outperforms Gemini 1.5 Pro on 4 out of 6 video tasks. It achieves 72.4% on Video-MME with 512 input frames (w/o subtitle) and 75.7% (w/ subtitle). It also matches or beats Qwen2.5-VL-72B on multiple key datasets while using only 8B parameters instead of 72B.
What is the Eagle-Video-110K dataset?
Eagle-Video-110K is a novel benchmark-scale dataset with over 110K annotated samples specifically curated for long video understanding. It contains both story-level and clip-level annotations including QA, localization, and summarization. Videos range from a few minutes to 3 hours long, pushing the limits of long-form visual reasoning. The dataset was collected via a diversity-driven strategy.
What are the key innovations in Eagle 2.5?
Key innovations include: Information-First Sampling with Image Area Preservation (IAP) for optimal image tiling and Automatic Degrade Sampling (ADS) for balancing visual/textual input; Progressive Mixed Post-Training gradually increasing context from 32K to 128K; Diversity-Driven Data Recipe combining open-source data with Eagle-Video-110K; GPU Memory Optimization with Triton-based fused operators; and Distributed Context Parallelism using Ulysses and Ring/Context Parallelism.
How do I install and run Eagle locally?
Install dependencies with: pip install transformers==4.37.2 and pip install flash-attn. Note that latest transformers version is not compatible. Prepare a model worker using the provided script, then use inference scripts supporting pure text, single image, multiple images, or video input. The model can be deployed locally using Streamlit for a chat demo with video input support.
What input types does Eagle support?
Eagle supports four input types: pure text input, single image input, multiple image input, and video input. For video, it can process up to 512 frames with optimized sparse frame sampling. It also handles high-resolution images up to 4K HD, multi-page documents, and long text sequences up to 128K tokens.
What benchmarks does Eagle excel at?
Eagle achieves SOTA on 6/10 long video benchmarks including MVBench (74.8), Perception_test (82.0), EgoSchema (72.2), MLVU (77.6), Video-MME (72.4), and CG-Bench. For image benchmarks, it scores 94.1 on DocVQA, 87.5 on ChartQA, 869 on OCRBench, and 66.2 on MMstar. It also performs well on embodied benchmarks like OpenEQA (63.5) and EgoPlan (45.3).
What is the license for Eagle?
The code is released under Apache 2.0 license. Pretrained model weights are released under the NVIDIA License. The service is a research preview intended for non-commercial use only. Components include Qwen2.5-7B-Instruct (Apache-2.0) and SigLIP2 (Apache-2.0). Users must ensure compliance with all applicable laws and regulations when using the dataset and checkpoints.
How does Eagle optimize inference efficiency?
Eagle uses multiple efficiency optimizations: Triton-based fused operators replacing PyTorch's MLP, RMSNorm, and RoPE implementations; fused linear layers with cross-entropy loss removing intermediate logit storage; CPU-offloading of hidden states; distributed context parallelism with two-layer communication groups; ZigZag Llama3-style Context Parallelism with all-gather KV to reduce latency; optimized sparse video frame sampling; and vLLM deployment support for accelerated inference with reduced memory.