Opencompass
OpenCompass is an LLM evaluation platform, supporting a wide range of models (Llama3, Mistral, InternLM2,GPT-4,LLaMa2, Qwen,GLM, Claude, etc) over 100+ datasets.
Last verified:
What is Opencompass?
OpenCompass is a one-stop open-source platform for evaluating large language models (LLMs) and large vision-language models (LVLMs). Developed by Shanghai AI Laboratory, it provides a complete,开源, and reproducible evaluation framework that supports over 70 datasets with approximately 400,000 questions, comprehensively assessing model capabilities across five dimensions including knowledge, reasoning, coding, and more.
Key features include efficient distributed evaluation that can complete billion-scale model evaluations in just a few hours using a single command, diversified evaluation paradigms supporting zero-shot, few-shot, and chain-of-thought evaluations, and modular design with high extensibility for adding new models, datasets, or custom evaluation strategies. The platform offers predefined configurations for 20+ HuggingFace and API models including Llama3, Qwen, InternLM, GPT-4, Claude, and Gemini.
OpenCompass includes CompassHub (a benchmark browser interface for exploring benchmarks), CompassRank (leaderboards for open-source and proprietary model benchmarks), and CompassKit (evaluation toolkits for LLMs and LVLMs). It supports both open-source models (Alpaca, Baichuan, ChatGLM, Gemma, LLaMA, Qwen, Yi, etc.) and API models (OpenAI, Gemini, Claude, ZhipuAI, Baichuan, Baidu ERNIEBot, etc.).
The platform is designed for AI researchers, machine learning engineers, MLOps practitioners, and organizations developing or deploying large language models who need objective, reproducible benchmarking for model comparison and performance tracking.
Opencompass pricing
Pricing model: Freemium
OpenCompass is completely free and open-source under the Apache-2.0 license. The platform is free to use with no paid tiers. Users can install via pip (pip install -U opencompass) or from source (git clone + pip install -e .). The full installation with support for more datasets is available via pip install "opencompass[full]". API evaluation requires users to provide their own API keys (e.g., OPENAI_API_KEY) but the framework itself has no cost. Leaderboard submission is free - users can join by providing model repository URL or API interface to [email protected].
Opencompass pros
- Fully open-source with Apache-2.0 license
- Supports 70+ datasets with 400,000+ questions
- Evaluates 20+ pre-configured models out of the box
- Distributed evaluation completes in few hours
- Supports both open-source and API models
- Zero-shot, few-shot, and chain-of-thought evaluation
- Modular design with high extensibility
- CompassHub benchmark browser interface
- CompassRank leaderboards for model comparison
- Math reasoning evaluation with MATHVerifyEvaluator
- Long-context evaluation with RULER and BABILong benchmarks
- LLM-as-judge evaluation with GenericLLMEvaluator
- One-click switching between inference backends (vLLM, LMDeploy)
- Real-time result reporting mechanism
- ModelScope integration for on-demand dataset loading
- CascadeEvaluator for multi-evaluator pipelines
- 6.2k GitHub stars with active community
Opencompass cons
- Requires network connection to HuggingFace initially
- GPU memory constraints for large models
- Complex configuration for custom datasets
- No direct GPU usage limit setting
- Model loading repeated for many tasks can be slow
- Opencv-python dependency issues in some environments
- Some third-party features require additional setup steps
- clp evaluation method affected by tokenizer issues
Frequently asked questions about Opencompass
What are the differences between ppl and gen evaluation modes?
ppl (perplexity) evaluates multiple-choice questions by concatenating options with context and selecting the option with lowest perplexity. It's simple and direct with high certainty. gen (generate) uses the model's continuation writing as the reasoning result, requiring heavier post-processing to extract answers. Base models use ppl for MCQ tasks and gen for other tasks. Chat models use gen for all tasks since many commercial APIs don't expose ppl interface.
How does OpenCompass control the number of shots in few-shot evaluations?
In dataset configuration files, the retriever field controls shot count. FixKRetriever uses a fixed k samples (k-shot), while ZeroRetriever uses no samples (0-shot). In-context samples can also be specified directly in the dataset template, in which case ZeroRetriever is used but evaluation is not 0-shot - it depends on the specific template configuration.
How does OpenCompass allocate GPUs during evaluation?
OpenCompass processes evaluations using 'tasks' (independent model+dataset combinations). GPU needs are determined by the model's num_gpus parameter. Multiple workers execute tasks in parallel, continuously trying to secure GPU resources. With 8 GPUs and tasks requiring 4 GPUs each, OpenCompass runs 2 tasks concurrently by default. The --max-num-workers setting controls parallelism - setting it to 1 means only one task processes at a time.
How can I control how many GPUs OpenCompass occupies?
Locally: Use CUDA_VISIBLE_DEVICES environment variable (e.g., CUDA_VISIBLE_DEVICES=0,1,2,3 limits to first 4 GPUs). With Slurm or DLC: Adjust --max-num-workers to restrict concurrent evaluation tasks. If each task requires 4 GPUs and you want 8 total GPUs, set --max-num-workers to 2.
Why does OpenCompass partition evaluation requests into tasks?
Comprehensive LLM evaluation is time-consuming due to extensive datasets. OpenCompass divides requests into multiple independent tasks dispatched to various GPU groups/nodes, achieving full parallelism and maximizing computational resource efficiency. This approach significantly reduces total evaluation time compared to linear evaluation.
How do I use downloaded HuggingFace models with OpenCompass?
Specify the local path of downloaded checkpoints using --hf-path parameter. Example: python run.py --datasets siqa_gen winograd_ppl --hf-type base --hf-path /path/to/model. This allows evaluation without downloading models during runtime.
What should I do if I get connection errors with HuggingFace?
Work behind a proxy by setting http_proxy and https_proxy environment variables. Use cache files from machines with internet access (located at ~/.cache/huggingface/), then run in offline mode with HF_DATASETS_OFFLINE=1 TRANSFORMERS_OFFLINE=1 HF_EVALUATE_OFFLINE=1. Alternatively, use mirrors like HF_ENDPOINT=https://hf-mirror.com.
How can I build a new custom dataset?
OpenCompass provides documentation for building new objective and subjective datasets. You can use CustomDataset with CustomDataset and configure reader_cfg, infer_cfg, and eval_cfg. For math evaluation specifically, use CustomDataset with MATHVerifyEvaluator, providing dataset in JSON Lines (.jsonl) or CSV format with problem statement and solution columns.
What inference acceleration backends does OpenCompass support?
OpenCompass supports one-click switching between inference backends including the default HuggingFace backend, LMDeploy, and vLLM. Use -a lmdeploy or -a vllm flag for command-line switching, or configure via deployment APIs. Ensure necessary packages are installed and the model supports accelerated inference with the chosen backend.
How do I join the OpenCompass leaderboard?
To join the evaluation, provide your model repository URL or standard API interface to [email protected]. The leaderboard includes both open-source models and API models. You can also refer to the Guide to Reproducing CompassAcademic Leaderboard Results to quickly reproduce leaderboard results.