AREAL: Revolutionizing Large Reasoning Model Training with Fully Asynchronous Reinforcement Learning
AREAL is a new asynchronous reinforcement learning system that significantly speeds up training of large reasoning models by separating generation and training processes, achieving up to 2.77× faster training without loss of accuracy.
Enhancing Reasoning with Reinforcement Learning
Reinforcement Learning (RL) is playing an increasingly vital role in improving Large Language Models (LLMs), especially those focused on reasoning tasks. These Large Reasoning Models (LRMs) generate intermediate "thinking" steps before producing final answers, boosting their capabilities in complex problems such as mathematics and coding. However, scaling RL training for LRMs is challenging due to the requirements for massive parallelization and efficient system design.
Limitations of Synchronous Training Systems
Current RL training systems often rely on synchronous batch processing. This approach forces all model generations in a batch to wait for the slowest output to complete, leading to poor GPU utilization and inefficiencies. Even newer batch-based methods suffer from bottlenecks because they use outdated rollouts and cannot fully leverage system resources.
Introducing AReaL: A Fully Asynchronous Training System
A team of researchers from IIIS, Tsinghua University, Ant Research, and HKUST developed AReaL, a fully asynchronous reinforcement learning system designed to accelerate the training of large reasoning models. AReaL separates generation and training processes, allowing rollout workers to continuously generate outputs while training workers update the model in parallel as new data arrives. This innovative design maximizes GPU usage and speeds up training significantly.
Technical Innovations Behind AReaL
AReaL's architecture decouples generation and training across different GPU clusters, improving scalability and hardware efficiency. Its four core components include:
- Rollout workers capable of interruptible generation and model updates
- A reward service evaluating model outputs
- Trainer workers performing PPO updates
- A controller managing data flow
To address challenges such as data staleness and inconsistent policy versions, AReaL employs staleness-aware training strategies and a decoupled PPO objective. Additional system-level optimizations like pipelined CPU-GPU operations, non-blocking asynchronous requests, and dynamic sequence packing further enhance training speed and GPU efficiency.
Impressive Experimental Results
When tested on math and coding tasks using distilled Qwen2 models, AReaL achieved training speeds 2 to 3 times faster than previous methods like DeepScaleR and DeepCoder, without compromising accuracy. The system efficiently scales across GPUs and supports long context lengths up to 32k tokens. Features such as interruptible generation and dynamic microbatching contribute to significantly improved training speed and hardware utilization. The decoupled PPO objective ensures stable learning even when training data is stale, a limitation in standard PPO approaches.
Impact on Large-Scale Reinforcement Learning
AReaL represents a major advancement in the efficient training of large reasoning models, enabling faster, scalable reinforcement learning without sacrificing performance. By running generation and training asynchronously and incorporating staleness-aware strategies, it reduces GPU idle time and improves throughput, marking a significant step forward for large-scale RL applications in language modeling.
For more details, explore the Paper and GitHub Page.
Сменить язык
Читать эту статью на русском