Rl
A modular, primitive-first, python-first PyTorch library for Reinforcement Learning.
Last verified:
What is Rl?
TorchRL is an open-source Reinforcement Learning (RL) library built specifically for PyTorch. It provides Python-first, low and high level abstractions for RL that are efficient, modular, well-documented, and properly tested. The library is designed to support research in RL, with most code written in Python in a highly modular way that allows researchers to easily swap components, transform them, or write new ones with little effort.
Key features include TensorDict for streamlined RL codebase, a common interface for environments supporting Gym, DeepMind Control Lab, and more, multiprocess and distributed data collectors, efficient replay buffers with modularized storage, cross-library environment transforms executed on device, various model architectures including actor-critic, exploration wrappers, efficient loss modules, and a comprehensive LLM API for language model fine-tuning with RLHF support. The library also includes state-of-the-art algorithm implementations for DQN, PPO, SAC, TD3, DDPG, and many others.
TorchRL is intended for RL researchers, practitioners, and developers who want to build reinforcement learning agents using PyTorch. It is particularly suited for those who value modularity, want to minimize dependencies, need distributed training capabilities, or are working on cutting-edge research involving LLMs and multi-agent RL.
Rl pricing
Pricing model: Freemium
TorchRL is completely free and open-source under the MIT license. There are no paid plans or tiers. The library can be installed directly from PyPI with 'pip install torchrl'. Optional dependencies for specific features (Atari, utils, tests) can be installed via extras like 'pip install torchrl[utils]' which includes hydra-core for the experimental training interface.
Rl pros
- Open-source and free to use under MIT license
- Python-first design for ease of use and flexibility
- Highly modular architecture for easy component swapping
- TensorDict enables complete PPO training in under 100 lines
- Aligns with PyTorch ecosystem conventions
- Minimal dependencies (only Python stdlib, NumPy, PyTorch)
- Supports distributed and multiprocess data collection
- Comprehensive LLM API for RLHF and fine-tuning
- State-of-the-art algorithm implementations included
- Cross-library environment transforms on device
- Efficient prioritized replay buffers with C++ binaries
- Vectorized advantage and return computation
- Experimental command-line training interface (no Python coding needed)
- Works with Gym, DM Control, Brax, and more environments
- torch.compile and functorch compatible
- Thorough documentation with tutorials and API reference
- Active community with 3.1k GitHub stars and 190 contributors
Rl cons
- Still in PyTorch beta feature status with BC-breaking changes likely
- Requires Python 3.10 or higher
- C++ binaries only work with PyTorch 2.7.0 and above
- Nightly builds only shipped for Linux machines
- Some features limited with older PyTorch versions
- Experimental CLI interface API may change in future versions
- Windows 11 machines may require local build
- Nested jagged tensors feature limited with older PyTorch
- tensordict requires latest PyTorch (requirement being loosened)
- Experimental features like PPOTrainer are prototypes
Frequently asked questions about Rl
What is TorchRL?
TorchRL is an open-source Reinforcement Learning (RL) library for PyTorch that provides Python-first, low and high level abstractions for RL. It is designed to be efficient, modular, documented, and properly tested, with code aimed at supporting research in RL.
How do I install TorchRL?
You can install the latest stable release using pip: 'pip install torchrl'. For the experimental training interface with Hydra configuration, use 'pip install torchrl[utils]'. Nightly builds can be installed via 'pip install tensordict-nightly torchrl-nightly'.
What Python and PyTorch versions does TorchRL require?
TorchRL requires Python 3.10 or higher and is roughly compatible with any PyTorch version >= 2.1. The C++ binaries for prioritized replay buffers only work with PyTorch 2.7.0 and above. Using the latest TorchRL with the latest PyTorch is recommended.
What algorithms are implemented in TorchRL?
TorchRL includes state-of-the-art implementations for DQN, DDPG, IQL, CQL, TD3, TD3+BC, A2C, PPO, SAC, REDQ, Dreamer v1, Decision Transformers, CrossQ, Gail, Impala, and multi-agent variants (IQL MARL, DDPG MARL, PPO MARL, QMIX-VDN MARL, SAC MARL), plus RLHF and LLM API (GRPO).
What is TensorDict and why is it important?
TensorDict is a convenient data structure that streamlines RL codebases by carrying data between environments, models, and algorithms. It allows writing complete PPO training scripts in less than 100 lines, supports multiple tensor operations (stack, cat, view, indexing), and is functorch and torch.compile compatible.
Does TorchRL support distributed training?
Yes, TorchRL provides multiprocess and distributed data collectors that work synchronously or asynchronously. It includes tools for distributed learning such as memory-mapped tensors and supports parallel environment execution with batched environment containers.
What environments does TorchRL support?
TorchRL provides a common interface for environments supporting OpenAI Gym, DeepMind Control Lab, Brax, DM Control, and more. It also supports state-less execution for model-based environments. Environment libraries like Gym are optional dependencies.
Does TorchRL have an LLM API?
Yes, TorchRL includes a comprehensive LLM API for post-training and fine-tuning language models. It provides unified wrappers for Hugging Face and vLLM, conversation management with History class, tool integration (Python execution, function calling), specialized objectives (GRPO, SFT), and high-performance async collectors for RLHF and supervised fine-tuning.
Can I train RL agents without writing Python code?
Yes, TorchRL now provides an experimental command-line interface that lets you train state-of-the-art RL agents with simple bash commands. You can run 'python sota-implementations/ppo_trainer/train.py' and override parameters via command line like 'trainer.total_frames=2000000 optimizer.lr=0.0003'.
What is the license for TorchRL?
TorchRL is licensed under the MIT License, which is a permissive open-source license. The library is released as a PyTorch beta feature, and while BC-breaking changes are likely, they will be introduced with deprecation warranty after a few release cycles.