Langroid
Harness LLMs with Multi-Agent Programming
Last verified:
What is Langroid?
Langroid is a Python framework for building LLM-powered applications using multi-agent programming. It was explicitly designed with agents as first-class citizens and multi-agent programming as its core design principle, making it the first Python LLM-application framework built around this paradigm. The framework is inspired by the Actor Framework and provides intuitive definitions for agents, tasks, and task-delegation among agents.
Key features include: Agents that encapsulate LLM conversation state with access to long-term memory (vector-stores like Qdrant, Chroma, LanceDB) and tools/functions; a Task class that wraps agents, manages iteration, and orchestrates multi-agent interactions via hierarchical task-delegation; native tool support using Pydantic that eliminates the need to write JSON schemas; RAG (Retrieval-Augmented-Generation) capabilities through DocChatAgent for chatting with documents; groundings and source-citation from external documents; observability with detailed logging and message lineage tracking; caching of LLM prompts and responses via Redis; and support for both JSON-based and XML-based tools (with XML being more reliable for code generation).
Langroid works with practically any LLM - local/open source or remote/proprietary/API-based including OpenAI models (default GPT-4o), non-OpenAI proprietary models, and local models via libraries like LiteLLM, ollama, llama-cpp-python, and ooba-TGW. It is designed for developers, data scientists, and engineers building complex LLM applications such as chatbots, document Q&A systems, multi-agent collaboration systems, data analysis tools, SQL chat interfaces, web search integrations, and structured information extraction systems.
Langroid pricing
Pricing model: Freemium
Langroid is an open-source Python framework available for free. It can be installed via pip and has no paid tiers. The framework itself is free to use, but users must pay for any LLM API calls they make (e.g., OpenAI API costs for GPT-4o, GPT-3.5-turbo). Local/open-source LLMs can be used at no cost beyond compute/electricity. Some optional integrations like Seltz Search Tool require additional API keys and pip install langroid[seltz].
Langroid pros
- Agents are first-class citizens with dedicated Agent class
- Multi-agent programming as core design principle
- Native tool support using Pydantic - no JSON schema writing needed
- Works with any LLM: local, open-source, or proprietary APIs
- Built-in Redis caching for LLM prompts and responses
- Supports Qdrant, Chroma, and LanceDB vector stores for RAG
- XML-based tools more reliable than JSON for code generation
- Detailed logging and message lineage/provenance tracking
- Grounding and source-citation from external documents
- Hierarchical task-delegation with principled orchestration
- DocChatAgent for easy document chat with RAG
- Pydantic error messages sent back to LLM for self-correction
- Lightweight, transparent, and flexible architecture
- Supports concurrent batch task execution via batch.py
- Can be deployed in FastAPI server
- GlobalState mechanism for sharing state between agents
- Built-in JSON repair for handling malformed LLM output
- single_round mode for tasks that exit after first LLM response
Langroid cons
- Many examples require GPT-4 compatible API key, weaker models may not work
- No built-in mechanism for persisting agent state across runs (requires manual pickle)
- top_k parameter not supported in OpenAIGPTConfig
- Weaker LLMs may require more detailed prompting and iterative approaches
- XML tools require defining custom subclasses for existing ToolMessages
- LLM may still forget to generate ToolMessage despite remedies
- Relevance extraction is the biggest latency step in DocChatAgent
- Some edge cases of bad JSON from weak LLMs may not be repairable
Frequently asked questions about Langroid
Does Langroid work with non-OpenAI LLMs?
Yes! Langroid works with practically any LLM, local or remote, closed or open. You can use local/open LLMs via libraries like LiteLLM OpenAI Proxy Server, ooba-TGW, llama-cpp-python, and ollama by setting the chat_model parameter in the LLM config. For non-OpenAI proprietary LLMs, similar configuration applies. The framework handles the api_base setting automatically in most cases.
Can I view the reasoning text when using Reasoning LLMs like R1 or o1?
Yes, Langroid supports viewing the reasoning (thinking) text when using Reasoning LLMs like R1 or o1. You can access this through the reasoning-content feature documented in the framework.
How can I limit the number of output tokens generated by the LLM?
You can set the max_output_tokens parameter in the LLMConfig class or OpenAIGPTConfig class. For example, setting max_output_tokens=100 limits output to 100 tokens. If omitted, it defaults to 8192. Set max_output_tokens=None to use the model-specific maximum from model_info.py.
How does Langroid handle long chat histories?
Langroid handles long chat histories by checking if tokens(H) + M ≤ C (where H is history, M is max_output_tokens, C is context length). If the context is too small, it tries to shorten output or drop early messages. It won't drop the system message or last user message. Methods like llm_response_forget can erase query/response pairs to manage tokens.
Can I persist agent state across multiple runs?
Currently there is no built-in Langroid mechanism for persistence, but you can achieve basic persistence by saving agent.message_history using Python's pickle module. Set restart=False in Task constructor to prevent state reset. For complex persistence, use GlobalState to store message histories of multiple agents indexed by name.
Is it possible to share state between agents/tasks?
Yes, the GlobalState mechanism can be used to share state between agents/tasks. GlobalState stores message histories of multiple agents indexed by their name. Examples are in chat-tree.py and test_global_state.py.
How can I deal with LLMs generating bad JSON in tools?
Langroid attempts to repair bad JSON using the json-repair library and custom methods before parsing. For better reliability, especially with weak LLMs, use XMLToolMessage instead of ToolMessage - XML with CDATA tags is far more reliable than JSON. Strict decoding leveraging Structured JSON outputs from OpenAI, llama.cpp, and vllm is coming soon.
How can I improve DocChatAgent (RAG) latency?
DocChatAgent latency can be improved by: setting assistant_mode=True to turn off query rephrasing to StandAlone; setting relevance_extractor_config to None to turn off relevance extraction (the biggest latency step); and controlling various retrieval settings in DocChatAgentConfig. The retrieval step alone can be used via RetrievalTool.
Is there support to run multiple tasks concurrently?
Yes, use run_batch_tasks and related functions in batch.py for concurrent task execution. DocChatAgent also uses batch tasks for relevance extraction, running k tasks concurrently when there are k relevant passages. Examples include test_batch.py, test_relevance_extractor.py, and multi-agent-round-table.py.
Can a sub-task end all parent tasks and return a result?
Yes, using FinalResultTool. From a ChatAgent's tool-handler or agent_response method, return a FinalResultTool with arbitrary field types - this ends current and all parent tasks. From ChatAgent's llm_response method, define a subclass of FinalResultTool and enable the agent to use it. Examples are in test_tool_messages.py and multi-agent-return-result.py.