Docetl
A system for agentic LLM-powered data processing and ETL
Last verified:
What is Docetl?
DocETL is an open-source system for creating and executing LLM-powered data processing pipelines, specifically designed for complex document processing tasks. Developed by the EPIC Data Lab and Data Systems and Foundations group at UC Berkeley, it offers a low-code, declarative YAML interface that allows users to define complex data operations on unstructured documents without writing extensive code.
Key features include a rich suite of operators tailored for complex data processing (such as
Docetl pricing
Pricing model: Freemium
DocETL is completely free and open-source under the MIT license. There are no paid plans or licensing fees. Users only pay for the LLM API calls they make (e.g., OpenAI, Claude, Gemini). The tutorial example showed a sample pipeline running at $0.10 total cost using gpt-4o-mini. You can install via pip (pip install docetl) or clone from GitHub at no cost.
Docetl pros
- Open-source with MIT license - free to use and modify
- Low-code YAML interface reduces development complexity
- Automatic accuracy optimization using LLM agents
- 25-80% more accurate outputs than well-engineered baselines
- Specialized operators like resolve for entity resolution
- Interactive UI playground for iterative prompt engineering
- Handles long documents that don't fit in single prompts
- Built-in validation with automatic retry on failure
- Map-reduce pattern support for complex workflows
- 1.7k+ GitHub stars with active community
- Claude Code integration for fastest pipeline building
- 100% control over prompts in YAML configuration
- Supports multiple LLM providers via LiteLLM
- Caching and checkpointing of intermediate results
- Cross-domain applicability (law, medicine, social sciences)
Docetl cons
- Primarily tested with OpenAI models - other providers may have reduced functionality
- Requires API key setup for LLM providers (costs not included)
- Complex YAML configuration has learning curve for beginners
- Structured output capabilities depend on LLM provider support
- Not ideal for simple ETL tasks - overkill for basic use cases
- Requires Python development environment setup
- Gemini/Ollama models may need simplified output schemas
- No built-in hosted cloud service - self-hosted only
Frequently asked questions about Docetl
What is DocETL and what does it do?
DocETL is a system for creating and executing LLM-powered data processing pipelines. It offers a low-code, declarative YAML interface to define complex data operations on unstructured documents. It's designed to maximize correctness and output quality for complex tasks over collections of documents, using map-reduce patterns, entity resolution, and automatic optimization.
Is DocETL free to use?
Yes, DocETL is completely open-source under the MIT license. You can install it via pip or clone from GitHub at no cost. The only costs you incur are for the LLM API calls you make (e.g., OpenAI, Claude, Gemini API usage).
Which LLM providers does DocETL support?
DocETL uses LiteLLM under the hood and supports various LLM providers including OpenAI (primarily tested with gpt-4o and gpt-4o-mini), Gemini (use gemini/ prefix like gemini/gemini-2.0-flash), and Ollama. However, it has been primarily tested with OpenAI models, and other providers may have reduced functionality especially with structured output operations.
How do I get started with DocETL?
To get started: 1) Install the package via pip (pip install docetl), 2) Set up your LLM API key as an environment variable (e.g., export OPENAI_API_KEY=your_api_key), 3) Define your pipeline in a YAML file, and 4) Run your pipeline using the DocETL command-line interface (docetl run pipeline.yaml). The fastest way is using Claude Code with the built-in DocETL skill.
What types of documents can DocETL process?
DocETL can process various document types across domains including legal documents (contract clauses, force majeure, indemnification clauses), medical transcripts (doctor-patient conversations, medication information), customer support tickets, YouTube transcripts, and any unstructured text data. It handles long documents that don't fit in a single prompt through document splitting and context maintenance.
How does the accuracy optimization work?
DocETL's optimizer leverages LLM agents to automatically experiment with different logically-equivalent rewrites of your pipeline (called rewrite directives) and selects the most accurate version. It uses agent-guided plan evaluation with task-specific validation prompts and an optimization algorithm that efficiently finds promising plans while considering latencies. This can improve accuracy by 25-80% compared to well-engineered baselines.
What operators does DocETL provide?
DocETL provides a rich suite of operators including map (process each document), reduce (group and aggregate), resolve (entity resolution to match similar items), gather (maintain context when splitting documents), unnest (flatten lists like pandas explode), filter, and merge. Specialized operators like resolve use blocking keys and comparison prompts to identify and consolidate similar entries.
Can I use DocETL without coding?
DocETL is low-code, not no-code. You define pipelines using YAML configuration files, which requires some technical familiarity. However, the fastest way to build pipelines is using Claude Code with the built-in DocETL skill - you simply describe your data processing task in natural language, and Claude will create and run the pipeline for you automatically.
How do I handle validation and errors in DocETL?
DocETL supports validation criteria where tasks automatically retry when validation fails. You can define validation prompts in your pipeline configuration, and the system will re-execute operations that don't meet your validation criteria. The DSLRunner also includes caching/checkpointing of intermediate results, allowing you to resume from where failures occurred.
Who created DocETL and is it actively maintained?
DocETL was created by members of the EPIC Data Lab and Data Systems and Foundations group at UC Berkeley. As of March 2025, it has over 1.7k GitHub stars with users spanning various domains. The project is actively maintained as an open-source research project, with the EPIC Lab focusing on developing low-code interfaces for data work powered by predictive programming techniques.