Evalscope
A streamlined and customizable framework for efficient large model (LLM, VLM, AIGC) evaluation and performance benchmarking.
Last verified:
What is Evalscope?
EvalScope is a comprehensive model evaluation and benchmarking framework created by the ModelScope community. It provides an all-in-one solution for assessing various types of AI models including Large Language Models (LLMs), Multimodal Models, Embedding Models, Reranker Models, CLIP Models, and AIGC Models (Text-to-Image/Video). The framework serves as a valuable ally in model optimization journeys, offering end-to-end support from training to assessment.
Key features include preloaded industry-recognized benchmarks like MMLU, CMMLU, C-Eval, GSM8K, ARC, HellaSwag, TruthfulQA, MATH, and HumanEval. EvalScope implements multiple evaluation metrics, supports automated evaluation of objective questions using expert models, and generates automatic evaluation reports. It features Arena Mode for pairwise model comparisons, visualization tools through Gradio/WandB/SwanLab, and performance stress testing for model inference services. The framework seamlessly integrates with the ms-swift training framework for one-click evaluations and supports multiple evaluation backends including Native, OpenCompass, VLMEvalKit, and RAGEval.
EvalScope is designed for large model developers, AI researchers, and machine learning engineers who need to evaluate model capabilities before deployment. It's particularly useful for teams developing LLMs, multimodal systems, RAG applications, and AIGC models who require comprehensive benchmarking against industry standards. The framework supports both CLI and Python code usage, making it accessible for various workflows.
Evalscope pricing
Pricing model: Freemium
EvalScope is completely free and open-source under the Apache-2.0 license. It can be installed via pip (pip install evalscope) or from source with no paid tiers. Optional dependency groups include evalscope[perf] for performance testing, evalscope[app] for visualization features, evalscope[opencompass] for OpenCompass backend, evalscope[vlmeval] for VLMEvalKit backend, and evalscope[rag] for RAG evaluation. All features are available without cost, with no premium plans or subscription fees.
Evalscope pros
- Comprehensive support for multiple model types (LLM, VLM, Embedding, Reranker, CLIP, AIGC)
- Preloaded with industry-recognized benchmarks (MMLU, CMMLU, C-Eval, GSM8K, ARC, HumanEval)
- Seamless integration with ms-swift training framework for one-click evaluations
- Multiple evaluation backends (Native, OpenCompass, VLMEvalKit, RAGEval)
- Arena Mode for pairwise model comparisons with win rate rankings
- Automatic evaluation report generation in JSON/Tables/Logs formats
- Visualization support through Gradio, WandB, and SwanLab
- Performance stress testing with metrics like TTFT, TPOP, and stability
- Supports custom datasets for evaluation (text and multimodal)
- API-compatible model evaluation (OpenAI API format)
- Pass@k evaluation metric support for coding tasks
- Needle-in-a-Haystack test built-in for long-context evaluation
- Multimodal RAG evaluation with end-to-end metrics
- Easy pip installation with optional dependency groups
- Open source under Apache-2.0 license
- Active community with regular updates and new benchmark additions
- Supports both CLI and Python code task submission
- Custom parameter configuration for model and dataset args
- Judge model support for subjective evaluation tasks
- Multi-GPU evaluation through inference services or device_map
Evalscope cons
- No native Data Parallel support for multi-GPU evaluation
- Notebook environment issues with event loop (requires .py script)
Frequently asked questions about Evalscope
How to use EvalScope through Docker?
You can use the official ModelScope image which includes EvalScope. Refer to the ModelScope Docker documentation for details on pulling and running the image with EvalScope pre-installed.
How to conduct pass@k evaluation or generate multiple answers for a single sample?
Set repeats=k in TaskConfig and set the aggregation method in dataset_args as '<dataset_name>': {'aggregation': 'mean_and_pass_at_k'}. Each sample will be repeated k times and all pass@n metrics for 1<=n<=k will be calculated. Supported aggregation methods also include mean_and_vote_at_k and mean_and_pass_hat_k.
How to use a local model as a Judge Model?
Deploy the local model as an API service using frameworks like vLLM, then specify its service address in --judge-model-args. You can also set the timeout parameter in the generation_config of --judge-model-args.
How to set up multi-GPU evaluation?
EvalScope doesn't support Data Parallel natively. Achieve model parallelism through: (1) Using inference services - start a multi-GPU service with vLLM using --tensor-parallel-size and evaluate through API, or (2) Local loading - specify device_map=auto in --model-args to automatically distribute weights across devices.
How to troubleshoot obviously abnormal evaluation results like extremely low accuracy?
Follow these steps: (1) Check model interface - confirm the model service or local model generates responses normally, (2) Review prediction files in outputs/<timestamp>/predictions/ directory to verify outputs meet expectations, (3) Use evalscope app to start the visualization interface for intuitive result analysis.
How to evaluate multimodal models like Qwen-VL or Gemma3?
Deploy multimodal models as API services using frameworks like vLLM, then evaluate through API. Direct local loading of multimodal models for pure text evaluation may not be fully supported.
How to continue from checkpoint after evaluation interruption?
Checkpoint resumption is supported. Use the --use-cache parameter and specify the output directory path from the previous evaluation to reuse completed model predictions and evaluation results.
What should I fill in for the --url parameter in evalscope perf?
For general OpenAI API-compatible services, use the /v1/chat/completions endpoint. For the speed_benchmark dataset specifically (used for testing completion performance), use the /v1/completions endpoint to avoid Chat Template processing overhead.
Why is TTFT (Time To First Token) the same as Latency (Total Latency)?
To accurately measure TTFT, you must add the --stream parameter in the stress test command to enable streaming output. Without streaming, TTFT will equal total latency when receiving complete responses.
How to cite EvalScope in my work?
Use the following BibTeX format: @misc{evalscope_2024, title={{EvalScope}: Evaluation Framework for Large Models}, author={ModelScope Team}, year={2024}, url={https://github.com/modelscope/evalscope}}