ROLL

ROLL: Reinforcement Learning Optimization for Large-Scale Learning

🚀 An Efficient and User-Friendly Scaling Library for Reinforcement Learning with Large Language Models 🚀

ROLL is an efficient and user-friendly RL library designed for Large Language Models (LLMs) utilizing Large Scale GPU resources. It significantly enhances LLM performance in key areas such as human preference alignment, complex reasoning, and multi-turn agentic interaction scenarios.

Leveraging a multi-role distributed architecture with Ray for flexible resource allocation and heterogeneous task scheduling, ROLL integrates cutting-edge technologies like Megatron-Core, SGLang and vLLM to accelerate model training and inference.

📢 News

🚀 Get Started

Documents

Quick Start

Installation
Config System Explanation
Debugging Guide
Trackers and Metrics
Checkpoint Saving and Resuming Guide
Converting MCoreAdapter Models to Hugging Face Format
Quick Start: Single-Node Deployment Guide
Quick Start: Multi-Node Deployment Guide
Quick Start: Using Alibaba Cloud Function Compute DevPod for Rapid Development Frequently Asked Questions

UserGuide

Pipeline Step by Step

RLVR Pipeline
Agentic Pipeline
Agentic Comprehensive Guide
Distill Pipeline

Algorithms

Reinforce++
TOPR
GiGPO
PPO
Lite PPO
GRPO
GSPO
RAFT++
StarPO
RewardFL

Backend

FSDP2 Megatron
vLLM
SGLang

Advanced Features

Asynchronous Parallel Rollout
Asynchronous Training Feature

Performance Optimization & Resource Management

Resource Config
GPU Time-Division Multiplexing Control

ROLL x Huawei Ascend

Huawei Ascend Usage Guide

✨ Key Features

• Multi-task RL Training (RLVR): Covers mathematics, coding, general reasoning, open-ended Q&A, instruction following, etc.
  • Flexible domain_batch_size distribution control.
  • Sample-level asynchronous parallel Rollout, asynchronous reward calculation, and dynamic sampling.
  • Asynchronous training under implementation.
• Agentic RL: Multi-turn interaction capabilities for games, multi-turn dialogues, tool use, etc.
  • Environment-level asynchronous parallel rollout.
  • Supports asynchronous training.
  • Multi-turn interaction rollout supports local debugging, improving multi-turn interaction business development efficiency.
  • Supports TrajectoryWise (StarPO) and StepWise (GiGPO) training paradigms.
• Algorithm-Friendly: Provides flexible and rich RL strategy configurations by default.
  • Over 20 rich reinforcement learning strategy options, such as reward normalization, reward clipping, various advantage estimation methods, etc.
  • Out-of-the-box support for reinforcement learning algorithms, such as PPO, GRPO, Reinforce++, TOPR, RAFT++, GSPO, etc.
• Rich Training and Inference Engine: Ray-based multi-role distributed architecture; Strategy abstraction unifies various backends, enabling easy operation from single machines to thousands-of-GPU clusters.
  • Inference/Generation supports vLLM, SGLang.
  • Training supports FSDP2, Megatron-LM 5D parallelism (mcore-adapter, dp/tp/pp/cp/ep).
  • Extreme offload/reload capabilities.
  • Supports LoRA training.
  • Supports FP8 rollout (FP8 inference for LLM as judge, FP8 rollout with BF16 training under development).
• AutoDeviceMapping: Supports custom device mapping for different roles, flexibly managing colocated and disaggregated deployments.
• Observability: Integrated with SwanLab / WandB / TensorBoard, tracking of performance for each domain and reward type.
• Rich Post-training Technical Support:
  • Agentic RL LLM & VLM
  • RLVR LLM & VLM
  • Distill Pipeline LLM & VLM
  • DPO Pipeline
  • SFT Pipeline under development

🏆 Notable work based on ROLL

• Freshness-Aware-PER: A freshness-aware prioritized experience replay framework for LLM/VLM reinforcement learning, combining reward magnitude with exponential age decay (reward_fresh priority) and asynchronous full-buffer refresh, providing fresher and higher-signal off-policy samples for both step- and trajectory-level agentic RL. code
• ComplementaryRL: Complementary RL is a learning framework that enables agents to effectively learn from experience through the seamless co-evolution of an experience extractor and a policy actor within the RL optimization loop.
• RLix: RLix is an RL job manager that lets more RL jobs run concurrently with less waiting by sharing GPU capacity across jobs, while preserving each pipeline’s training behavior and improving GPU utilization.
• TurningPoint-GRPO: A GRPO framework for Flow Matching models in text-to-image generation that alleviates step-wise reward sparsity by modeling step-level incremental rewards and explicitly captures long-term effects via turning points detection, providing dense learning signals for each denoising action.
• SocioReasoner: A vision-language method for urban socio-semantic segmentation that employs a render-and-refine mechanism optimized by RL to identify abstract social entities using satellite and map data.
• STAgent: An agentic LLM specialized for spatio-temporal understanding and complex tasks like constrained POI discovery and itinerary planning, featuring hierarchical data curation with 1:10,000 filter ratio and cascaded training (seed SFT + difficulty-aware SFT + RL), achieving strong performance on TravelBench while preserving general capabilities.
• IPRO: A novel video diffusion framework using reinforcement learning to enhance identity preservation in human-centric I2V generation, optimizing diffusion models with face identity scorer and KL-divergence regularization.
• TaoSR-SHE: Stepwise Hybrid Examination Reinforcement Learning Framework for Taobao Search Relevance, with SRPO (hybrid reward model + offline verifier), diversified data filtering, and multi-stage curriculum learning.
• EARL: Efficient Agentic RL Systems for LLMs, introducing a dynamic parallelism selector and a layout-aware data dispatcher to boost throughput, reduce memory and data movement bottlenecks, enabling stable large-scale agentic RL without hard context-length limits.
• LiveThinking: Real-time reasoning for AI-powered livestreaming by distilling a 670B teacher LLM to a 30B MoE (3B active) via Rejection Sampling Fine-Tuning, then compressing reasoning with GRPO; delivers sub-second latency and ~30x compute reduction, with gains in response correctness (3.3%), helpfulness (21.8%), and GMV in Taobao Live.
• TaoSR-AGRL: Adaptive Guided Reinforcement Learning for LLM-based e-commerce relevance, introducing Rule-aware Reward Shaping and Adaptive Guided Replay to improve long-horizon reasoning, rule adherence, and training stability in Taobao Search; deployed in main search handling hundreds of millions of users.
• RecGPT: a next-generation, LLM-driven framework that places user intent at the core of recommender systems, fostering a more sustainable and mutually beneficial ecosystem.
• TaoSR1: A novel LLM framework directly deploying Chain-of-Thought (CoT) reasoning for e-commerce query-product relevance prediction, overcoming deployment challenges for superior performance.
• AIGB-Pearl: a novel auto-bidding method that integrates generative planning and policy optimization, utilizing an LLM-enhanced trajectory evaluator to iteratively refine bidding strategies for state-of-the-art advertising performance.

🙏 Citation and Acknowledgement

ROLL is inspired by the design of OpenRLHF, VeRL, Nemo-Aligner, and RAGEN. The project is developed by Alibaba TAOBAO & TMALL Group and Alibaba Group. The code is distributed under the Apache License (Version 2.0). This product contains various third-party components under other open-source licenses. See the NOTICE file for more information.

The following repositories have been used in ROLL, either in their close-to-original form or as an inspiration:

• NVIDIA/Megatron-LM
• microsoft/DeepSpeed
• sgl-project/sglang
• vllm-project/vllm
• modelscope/DiffSynth-Studio

If you use ROLL in your research or project, please consider citing us:

@article{wang2025reinforcement,
  title={Reinforcement Learning Optimization for Large-Scale Learning: An Efficient and User-Friendly Scaling Library},
  author={Wang, Weixun and Xiong, Shaopan and Chen, Gengru and Gao, Wei and Guo, Sheng and He, Yancheng and Huang, Ju and Liu, Jiaheng and Li, Zhendong and Li, Xiaoyang and others},
  journal={arXiv preprint arXiv:2506.06122},
  year={2025}
}

🤝 About [ROCK & ROLL Team]

ROLL is a project jointly developed by Taotian Future Living Lab and Alibaba AI Engine Team, with a strong emphasis on pioneering the future of Reinforcement Learning (RL). Our mission is to explore and shape innovative forms of future living powered by advanced RL technologies. If you are passionate about the future of RL and want to be part of its evolution, we warmly welcome you to join us!👇

We are HIRING!

• Post Training Infra 研发工程师 JD link
• 大模型训练专家：
  • （社招）JD link
  • （校招）JD link
• Infra 研究型实习生 JD link