Cocoindex
Incremental engine for long horizon agents 🌟 Star if you like it!
Last verified:
What is Cocoindex?
CocoIndex is an open-source incremental data framework for AI agents that transforms raw data into continuously fresh, optimized indexes for retrieval. It turns codebases, meeting notes, inboxes, videos, and documents into live context for AI agents to reason over effectively, with minimal incremental processing. The core engine is written in Rust with a Python API, delivering ultra-high performance for real-time data transformation.
Key features include incremental processing (only delta is reindexed with sub-second freshness at any repo size), semantic search capabilities, call graphs and blast radius analysis for code, knowledge graph extraction from meeting notes, and end-to-end data lineage. CocoInsight provides a comprehensive web UI to inspect flow schema and data, helping engineers understand and debug their data pipelines visually. The framework follows a Dataflow programming model where developers declare target state as Target = F(Source), and the engine computes the minimum work to keep it synchronized.
CocoIndex is designed for engineers building AI workloads, particularly coding agents, code-review agents, and security-review agents. It's ideal for teams building RAG systems, vector indexes, knowledge graphs, or any custom data transformation beyond SQL. The framework supports plug-and-play building blocks for different sources (LocalFile, S3, Azure Blob, Google Drive), targets (Postgres, Qdrant, LanceDB), and transformations, making it easy to assemble data pipelines like LEGO blocks.
Cocoindex pricing
Pricing model: Freemium
CocoIndex is completely free and open-source under the Apache 2.0 license. There are no paid plans or tiers - the entire framework including incremental processing, CocoInsight web UI, and all building blocks are available at no cost. Users only need to provide their own infrastructure (Postgres database, vector database like Qdrant or LanceDB, and any API services like embedding models or LLMs).
Cocoindex pros
- Open-source with Apache 2.0 license
- Incremental processing by default - only delta reindexed
- Sub-second freshness at any repo size
- Rust core engine for ultra-high performance
- Python API for easy developer velocity
- End-to-end data lineage out of the box
- CocoInsight web UI for visual pipeline debugging
- No hidden states or value mutation - pure Dataflow model
- Schema auto-evolution without migration meetings
- No index swap or downtime on changes
- Plug-and-play building blocks for sources, targets, transformations
- Native support for semantic search and vector indexes
- Call graphs and blast radius analysis for code changes
- Supports multiple data sources (LocalFile, S3, Azure, Google Drive)
- Supports multiple targets (Postgres, Qdrant, LanceDB, graph DB)
- Production-ready from day zero
- Cached computation - reuses cached results when input hash matches
- Parallel chunking for maximum efficiency
- Live updates while server is running with -L flag
- Built-in HTTP server for CocoInsight access
Cocoindex cons
- Requires Postgres installation for incremental processing
- Internal REST API is subject to change and not stable
- Default server binds to localhost only (127.0.0.1) for safety
- Python 3.11+ required
- Primarily focused on AI/ML workloads, not general ETL
- Learning curve for Dataflow programming model
- CocoInsight requires explicit CORS configuration to access
Frequently asked questions about Cocoindex
What is CocoIndex?
CocoIndex is an ultra-performant real-time data transformation framework for AI with incremental processing. It transforms raw data from sources like codebases, meeting notes, inboxes, and videos into continuously fresh indexes optimized for AI agent retrieval. The core engine is written in Rust with a Python API, and it follows a Dataflow programming model where you declare the target state and the engine keeps it synchronized with minimal recomputation.
How does incremental processing work in CocoIndex?
Incremental processing is a core value of CocoIndex. When source data changes, only the delta is reindexed - not the full dataset. The framework watches the source, computes the delta, and reconciles the target at any scale in parallel. When code changes, only affected rows re-run while cached results are reused when the input hash still matches. This delivers sub-second freshness at any repo size with minimal computation.
What programming languages does CocoIndex support?
CocoIndex has a Rust core engine (77% of the codebase) for performance and a Python API (22.9%) for developer velocity. Users write transformations in Python using a simple dataflow syntax, and the engine derives the computation graph automatically. Python 3.11+ is required.
What is CocoInsight?
CocoInsight is a comprehensive web UI to help you understand your flows and interact with the index. It lets you see the shape of your data at every stage of the flow, trace a single vector back to the paragraph it came from, and debug with your eyes instead of grep. CocoInsight talks to your CocoIndex HTTP server over APIs and retains zero data - it only retrieves flow schema and data when you're using it.
How do I start the CocoIndex HTTP server?
You can start the server via CLI with 'cocoindex server path/to/app.py' or programmatically via Python API. The default bind address is 127.0.0.1:49344 (local only for safety). To allow CocoInsight access, use the -ci flag: 'cocoindex server path/to/app.py -a 0.0.0.0:49344 -ci'. Then open https://cocoindex.io/cocoinsight and point it to your server address.
What data sources does CocoIndex support?
CocoIndex supports multiple built-in sources including LocalFile (files from a directory), Amazon S3, Azure Blob Storage, and Google Drive. The plug-and-play building block architecture makes it easy to add custom sources. Examples include indexing markdown files, PDFs, images, code repositories, and meeting notes.
What data targets does CocoIndex support?
CocoIndex supports exporting to Postgres (required for incremental processing), vector databases like Qdrant and LanceDB for semantic search, graph databases for knowledge graphs, and custom output files. The standardized interface makes it a one-line code switch between different target components.
Is CocoIndex production-ready?
Yes, CocoIndex is production-ready from day zero. It has exceptional developer velocity with ~100 lines of Python to declare transformations, incremental processing for always-fresh data, end-to-end lineage for explainability, and optimizations like parallel chunking and maximum efficiency built right in. The framework has 3k GitHub stars and 101 releases.
What is the Dataflow programming model in CocoIndex?
CocoIndex follows the Dataflow programming model where each transformation creates a new field solely based on input fields, without hidden states and value mutation. You declare the target state as Target = F(Source), write the transform in Python (not a DAG), and the engine derives the graph. All data before/after each transformation is observable with lineage out of the box, and only the delta runs - never the full recompute.