Researchclawbench
🦞 ResearchClawBench: Evaluating AI Agents for Automated Research from Re-Discovery to New-Discovery
Last verified:
What is Researchclawbench?
ResearchClawBench is an end-to-end scientific research benchmark designed to evaluate AI agents and LLMs on their ability to conduct automated scientific research from re-discovery to new-discovery. It measures whether AI coding agents can independently conduct scientific research — from reading raw data and related work to producing publication-quality reports with figures, methodology, results, and discussion — and then rigorously evaluates the results against real human-authored papers.
The benchmark operates in two distinct stages: Stage 1 (Auto Research) where the AI agent receives a workspace containing raw datasets, reference materials, and task instructions, and must independently write and execute code, produce figures, and generate a publication-style report (report/report.md); and Stage 2 (Evaluation) where an LLM judge scores the agent's report against expert checklists derived from target papers with per-item scores and reasoning.
Key features include evaluation across 10 scientific disciplines, expert-driven task construction with human validation, fine-grained evaluation checklists with weighted criteria and technical keywords, sandboxed agent workspaces with full tool access, and a public leaderboard showing model performance scores. Every task is built through a rigorous pipeline including high-quality paper collection, expert task extraction, checklist design, data collection, and human reproduction validation.
ResearchClawBench is designed for AI researchers, scientists, developers working with AI agents, and anyone who needs to objectively measure an AI agent's capability to perform real-world scientific research. It is particularly useful for evaluating auto-research agents, comparing different LLM backbones for scientific tasks, and verifying AI agents' ability to perform complex scientific tasks from data analysis to report generation.
Unlike benchmarks that test coding ability or factual recall, ResearchClawBench asks whether given a curated scientific workspace and the same research goal, an AI agent can arrive at the same (or better) scientific conclusions. Most AI benchmarks evaluate what models know, but ResearchClawBench evaluates what agents can do.
Researchclawbench pricing
Pricing model: Freemium
ResearchClawBench is free and open-source. It is released by the InternScience organization (AI for Science Center @Shanghai AI Laboratory) as an open-source benchmark. The benchmark dataset is available on Hugging Face, and the code is on GitHub under the InternScience organization. No paid plans are mentioned - it is designed as an open-source tool to advance and democratize artificial intelligence through open source and open science.
Researchclawbench pros
- Evaluates what agents can do rather than what models know
- End-to-end scientific research benchmark from data to publication
- Rigorous expert-driven task construction pipeline
- Validated by human reproduction of results
- Evaluation against real human-authored target papers
- Fine-grained checklists with weighted criteria and keywords
- Covers 10 scientific disciplines
- Sandboxed workspace with full tool access for agents
- No hand-holding or chain-of-thought hints
- Produces publication-quality reports with figures
- Public leaderboard for model comparison
- Open-source benchmark from InternScience organization
- Two-stage protocol for research and evaluation
- Scientist-aligned workflow benchmark
- Fair and reproducible evaluation grounded in reality
Researchclawbench cons
- Very low scores for current models (around 13-18 out of 100)
- Requires significant computational resources for evaluation
- Complex setup requiring expert knowledge
- Only 40 tasks in the benchmark
- Async job processing requires polling for results
- No clear free tier or pricing information available
- Limited documentation compared to other benchmarks
- Experimental status with 19 stars on GitHub
Frequently asked questions about Researchclawbench
What is ResearchClawBench?
ResearchClawBench is an end-to-end scientific research benchmark for evaluating AI agents and LLMs on tasks that require reading task data and related work, writing and executing code, producing figures, and generating publication-style reports. Final reports are scored against expert checklists derived from human-authored target papers.
What makes ResearchClawBench different from other AI benchmarks?
Most AI benchmarks evaluate what models know (factual recall or coding ability). ResearchClawBench evaluates what agents can do - given a curated scientific workspace and research goal, can an AI agent arrive at the same or better scientific conclusions as human researchers? It tests independent scientific research capability from raw data to publication.
How does the two-stage evaluation work?
Stage 1 (Auto Research): The AI agent receives raw data, reference materials, and instructions, then independently writes code, produces figures, and generates a publication-style report. Stage 2 (Evaluation): An LLM judge scores the agent's report against expert checklists from target papers, providing per-item scores with reasoning.
What scientific disciplines are covered?
ResearchClawBench covers 10 scientific disciplines. Domain experts select recent, high-impact publications with clear methodology and reproducible results across these disciplines for the benchmark tasks.
How are tasks constructed and validated?
Tasks go through a rigorous pipeline: (1) High-Quality Paper Collection - experts select impactful publications, (2) Expert Task Extraction - humans distill core research tasks, (3) Checklist Design - fine-grained evaluation criteria with weights and keywords, (4) Data & Related Work Collection, (5) Human Reproduction & Validation - researchers verify results are achievable.
What is the current performance of AI models on ResearchClawBench?
Current models score very low on ResearchClawBench. DeepSeek-V4-Pro scored 17.12/100 (39/40 tasks completed), Kimi-K2.5 scored 13.96/100 (39/40 tasks), and GLM-5.1 scored 18.19/100. This shows even advanced agents struggle profoundly with scientific research tasks.
How is the final report scored?
Reports are scored against expert checklists derived from human-authored target papers. The checklists contain weighted criteria for both text and image items, with specific technical keywords that the judge must verify. The final score is the overall mean score out of 100 over completed tasks.
Is ResearchClawBench open-source?
Yes, ResearchClawBench is open-source. It is published by the InternScience organization, the open-source hub of AI for Science Center @Shanghai AI Laboratory. The code is on GitHub and the dataset is available on Hugging Face.
Who should use ResearchClawBench?
ResearchClawBench is designed for AI researchers, scientists, developers working with AI agents, and anyone who needs to objectively measure an AI agent's capability to perform real-world scientific research. It is particularly useful for evaluating auto-research agents and comparing different LLM backbones for scientific tasks.
How do I submit my agent to the leaderboard?
To submit to the leaderboard, you need to complete ResearchClawBench tasks with your agent and submit the overall evaluation result. The detailed leaderboard is available at the ResearchClawBench-Home website. You submit your agent's score (overall mean out of 100) and number of completed tasks.