Autorag
AutoRAG: An Open-Source Framework for Retrieval-Augmented Generation (RAG) Evaluation & Optimization with AutoML-Style Automation
Last verified:
What is Autorag?
AutoRAG is an open-source AutoML tool for automatically finding an optimal Retrieval-Augmented Generation (RAG) pipeline for your data. It evaluates various RAG modules automatically with your own evaluation data to find the best RAG pipeline for your specific use-case. The tool addresses the challenge that there are numerous RAG pipelines and modules available, but it's difficult to know which pipeline works best for your own data without extensive testing.
AutoRAG supports three main capabilities: Data Creation (create RAG evaluation data with your own raw documents through parsing, chunking, and QA creation), Optimization (automatically run experiments to find the best RAG pipeline for your data), and Deployment (deploy the best RAG pipeline with a single YAML file, supporting Flask server and API endpoints). It provides extensive metrics for evaluation including retrieval metrics (retrieval_f1, retrieval_recall, retrieval_ndcg, retrieval_mrr) and generation metrics (meteor, rouge, sem_score).
The tool is designed for developers and teams building RAG applications who need to efficiently evaluate and optimize their RAG models. It supports Python 3.10 or higher, works with local LLMs and embedding models, includes Docker support for production deployment, and offers a dashboard for visualizing results. The framework is written in Python (99.6%) and is licensed under Apache-2.0.
Autorag pricing
Pricing model: Freemium
AutoRAG is completely free and open-source under Apache-2.0 license. It can be installed via pip with 'pip install AutoRAG'. There is a GPU version available with 'pip install AutoRAG[gpu]' and a parsing version with 'pip install AutoRAG[gpu,parse]'. The tool itself has no licensing costs, but using external services like OpenAI API for LLM calls will incur their standard usage costs. Docker images are available for free at autoraghq/autorag.
Autorag pros
- Open-source and free to use
- Automates RAG pipeline optimization with AutoML
- Supports multiple retrieval modules (BM25, vectordb, hybrid)
- Comprehensive evaluation metrics for retrieval and generation
- Easy deployment with single YAML file
- Supports local LLMs and embedding models
- Docker support for production deployment
- Provides dashboard for visualizing results
- Command-line interface for easy execution
- Supports parsing multiple file formats (PDF, etc.)
- Flexible YAML configuration for experiments
- API server deployment option with Flask
- Web interface deployment option
- HuggingFace Space integration available
- Large community with 4.3k GitHub stars
Autorag cons
- Requires Python 3.10 or higher
- May require high computational resources for GPU models
- Steeper learning curve for beginners
- AutoRAG GUI is no longer supported
- Requires creating QA and corpus datasets separately
- API calls (like OpenAI) may incur costs
- Complex YAML configuration for advanced users
- Validation process cannot be skipped in some cases
- Batch size limitations with certain models (e.g., OpenAI)
- Requires separate project directory for each dataset
Frequently asked questions about Autorag
What is AutoRAG?
AutoRAG is an open-source AutoML framework for automatically finding an optimal RAG (Retrieval-Augmented Generation) pipeline for your data. It evaluates various RAG modules automatically with your own evaluation data and finds the best RAG pipeline for your specific use-case.
How do I install AutoRAG?
You can install AutoRAG using pip with 'pip install AutoRAG'. For local models, install the GPU version with 'pip install AutoRAG[gpu]'. For parsing capabilities, use 'pip install AutoRAG[gpu,parse]'. Python 3.10 or higher is required.
What data do I need for RAG optimization?
RAG Optimization requires two types of data: QA dataset file (qa.parquet) and Corpus dataset file (corpus.parquet). The QA dataset is important for accurate and reliable evaluation, while the Corpus dataset is critical for RAG performance as it's used to retrieve documents.
What metrics does AutoRAG support?
AutoRAG supports retrieval metrics (retrieval_f1, retrieval_recall, retrieval_ndcg, retrieval_mrr), retrieval token metrics, and generation metrics (meteor, rouge, sem_score with embedding_model parameter).
How do I deploy my optimal RAG pipeline?
You can deploy in three ways: 1) Run as code using Runner.from_trial_folder(), 2) Run as API server using ApiRunner.from_trial_folder() or 'autorag run_api' command, 3) Run as web interface using 'autorag run_web' command.
Can I use my own local LLM or embedding model?
Yes, you can use your own local LLM or embedding model with AutoRAG. You need to install the GPU version with 'pip install AutoRAG[gpu]' and configure it in your YAML file.
What is the AutoRAG GUI status?
AutoRAG GUI is no longer supported. The developers will focus on maintaining only the AutoRAG core library in the future.
How do I run the dashboard?
You can run the dashboard using the command 'autorag dashboard --trial_dir /your/path/to/trial_dir' where trial_dir is the directory containing your trial results (like 0, 1, 2, etc.).
What RAG nodes does AutoRAG support?
AutoRAG supports multiple nodes including lexical_retrieval (BM25), semantic_retrieval (vectordb), hybrid_retrieval (hybrid_rrf), prompt_maker, and generator nodes with various module options for each.
How do I handle API rate limit errors?
For rate limit errors, adjust the batch size in your config YAML file. Set batch under 3 when using OpenAI models. You can also add the batch parameter to modules like 'llama_index_llm' with batch: 16 or lower.