Unirl
[ICML 2026] a unified reinforcement learning toolbox for joint RL on language models and diffusion models
Last verified:
What is Unirl?
PromptRL (Prompt Matters in RL for Flow-Based Image Generation) is a research framework that jointly trains language models (LMs) and flow-matching models (FMs) within a unified reinforcement learning loop for text-to-image generation. The framework addresses critical limitations in existing RL pipelines for flow-based models, specifically sample inefficiency due to insufficient generation diversity and pronounced prompt overfitting where models memorize specific training formulations. PromptRL incorporates language models as trainable prompt refinement agents directly within the flow-based RL optimization loop, enabling dynamic prompt rewriting capabilities.
Key features include: Dynamic Prompt Refinement where an LM generates semantically-grounded prompt variants preserving core intent while introducing linguistic diversity; a Prompt Retention Mechanism that retains original prompts to ensure robust performance on training distribution; Joint Policy Gradient Updates where LM and FM share reward signals while remaining architecturally disjoint; and Multi-Reward Training via Reward Tagging that assigns categorical tags to prompts for different reward functions. The framework achieves state-of-the-art performance with 0.97 on GenEval, 0.98 on OCR accuracy, and 24.05 on PickScore benchmarks.
PromptRL is designed for researchers and practitioners working on text-to-image generation, flow-based models, and reinforcement learning for AI alignment. It is particularly valuable for those developing image editing models, as it improved FLUX.1-Kontext's EditReward from 1.19 to 1.43 with only 0.06 million rollouts. The framework is open-source with code available at GitHub's UniRL repository, targeting the ML research community working on diffusion models and unified multimodal systems.
The framework requires over 2× fewer rollouts compared to naive flow-only RL while achieving higher performance ceilings, making it computationally more efficient for training flow-based image generation models. It successfully validates effectiveness on large-scale image editing models and surpasses Gemini 2.5 Flash Image (Nano Banana) in edit performance.
Unirl pricing
Pricing model: Freemium
PromptRL is completely free and open-source. The code is available at https://github.com/G-U-N/UniRL under open-source licensing. There are no paid plans, subscription tiers, or commercial pricing since this is academic research code released for the research community.
Unirl pros
- Achieves state-of-the-art 0.97 score on GenEval benchmark
- Attains 0.98 OCR accuracy on benchmark tests
- Reaches 24.05 PickScore benchmark performance
- Requires over 2× fewer rollouts than flow-only RL
- Improves FLUX.1-Kontext EditReward from 1.19 to 1.43
- Surpasses Gemini 2.5 Flash Image with EditReward 1.37
- Achieves comparable performance to ReasonNet at 1.44 EditReward
- Uses only 0.06 million rollouts for image editing improvements
- Incorporates LMs as trainable prompt refinement agents
- Enables dynamic prompt refinement with semantic preservation
- Includes prompt retention mechanism for training distribution robustness
- Supports joint policy gradient updates for LM and FM
- Implements multi-reward training via reward tagging
- Achieves higher performance ceilings than naive flow-only RL
- Architecturally disjoint LM and FM with shared reward signals
- Rapid development of sophisticated prompt rewriting capabilities
- Synergistic training regime reshapes optimization dynamics
- Open-source code available at GitHub UniRL repository
- Addresses sample inefficiency in flow-based RL pipelines
- Solves prompt overfitting problem in existing RL methods
Unirl cons
- Requires expertise in reinforcement learning frameworks
- Needs knowledge of flow-matching models for implementation
- Computational resources required for RL training loop
- Only addresses text-to-image generation, not other modalities
- LM and FM gradients do not propagate between them
- Requires setting up unified RL optimization infrastructure
- Limited to flow-based models, not diffusion models directly
- No commercial cloud-based API available
- Requires Python programming for code usage
- No pre-trained models provided out-of-the-box
- Research-focused with limited production documentation
- Requires GPU resources for training flow-matching models
- Not suitable for users without ML research background
- No graphical user interface, CLI/code only
- Requires understanding of policy gradient methods
Frequently asked questions about Unirl
What is PromptRL?
PromptRL (Prompt Matters in RL for Flow-Based Image Generation) is a framework that jointly trains language models (LMs) and flow-matching models (FMs) within a unified reinforcement learning loop for text-to-image generation. It incorporates LMs as trainable prompt refinement agents directly within the flow-based RL optimization loop, addressing sample inefficiency and prompt overfitting in existing RL pipelines.
What benchmarks does PromptRL achieve state-of-the-art on?
PromptRL achieves state-of-the-art performance with 0.97 on GenEval, 0.98 on OCR accuracy, and 24.05 on PickScore benchmarks. It also improved FLUX.1-Kontext's EditReward from 1.19 to 1.43, surpassing Gemini 2.5 Flash Image's 1.37 score.
How does PromptRL compare to flow-only RL?
PromptRL consistently achieves higher performance ceilings while requiring over 2× fewer rollouts compared to naive flow-only RL. Even when scaling flow-only RL with 2× rollouts, it still underperforms PromptRL (GenEval: 0.93 vs. 0.97).
What are the key components of PromptRL?
The key components are: (1) Dynamic Prompt Refinement where LM generates semantically-grounded prompt variants, (2) Prompt Retention Mechanism retaining original prompts, (3) Joint Policy Gradient Updates for LM and FM with shared rewards, and (4) Multi-Reward Training via Reward Tagging with categorical tags.
Where can I get the PromptRL code?
The code is available at https://github.com/G-U-N/UniRL. It is open-source and released for the research community.
What problems does PromptRL solve?
PromptRL addresses two critical limitations in flow-based RL: sample inefficiency due to insufficient generation diversity, and pronounced prompt overfitting where models memorize specific training formulations and collapse on semantically equivalent but stylistically varied prompts.
Can PromptRL be used for image editing?
Yes, PromptRL validates effectiveness on large-scale image editing models. It improved the EditReward of FLUX.1-Kontext from 1.19 to 1.43 with only 0.06 million rollouts, surpassing Gemini 2.5 Flash Image and achieving comparable performance with ReasonNet.
What is the Prompt Retention Mechanism?
For each batch of n samples, PromptRL retains m < n samples using the original prompt without LM refinement. This ensures the flow-matching model maintains robust performance on the training distribution while benefiting from expanded exploration through prompt variants.
How does Multi-Reward Training work in PromptRL?
Rather than computing weighted reward sums, PromptRL assigns each prompt a categorical tag indicating which reward function evaluates its images. This enables multi-reward training without complex reward weighting schemes.
Who developed PromptRL?
PromptRL was developed by Fu-Yun Wang, Han Zhang, Michael Gharbi, Hongsheng Li, and Taesung Park. The paper was published for ICML 2026 and is available on arXiv as arXiv:2602.01382.