Think Before You Predict: NVIDIA's RLP Brings Reinforcement to Pretraining

October 14, 2025 · 5 min

What RLP Is

NVIDIA introduces Reinforcement Learning Pretraining (RLP), a pretraining objective that rewards the model for “thinking” — generating short chains-of-thought (CoT) before predicting the next token. Instead of applying reinforcement learning after pretraining, RLP applies a dense, position-wise reinforcement signal during pretraining itself. The reward measures the information gain the thought provides about the upcoming token versus a no-think exponential moving average (EMA) baseline.

Mechanism: information-gain rewards and an EMA counterfactual

RLP uses a single network with shared parameters to both sample a CoT policy and to score the next token conditioned on that thought. A slowly updated EMA teacher model provides a no-think counterfactual. The per-token reward is the log-likelihood ratio between the CoT-conditioned likelihood and the EMA baseline, computed under teacher forcing. Training only updates the thought tokens through a clipped surrogate objective that uses per-token importance ratios and group-relative advantages; sampling multiple thoughts per context reduces variance.

The paper’s technical description includes a compact comment-like snippet that summarizes the per-thought reward and the training update approach. Reproduced verbatim below:

/* 




r(ct)=logpθ(xt∣x<t,ct)−logpϕ(xt∣x<t), computed under teacher forcing. Training updates only the thought tokens using a clipped surrogate with per-token importance ratios and group-relative advantages (multiple sampled thoughts per context reduce variance). The objective maximizes expected information gain; theoretical results connect the expected reward to reductions in cross-entropy and bound it via marginalization over thoughts.
Why this matters technically: unlike prior “reinforcement pretraining” variants that rely on sparse, binary correctness signals or proxy filters, RLP’s dense, verifier-free reward attaches position-wise credit wherever thinking improves prediction, enabling updates at every token position in general web-scale corpora without external verifiers or curated answer keys. 
Understanding the Results
Qwen3-1.7B-Base: Pretraining with RLP improved the overall math+science average by ~19% vs the base model and ~17% vs compute-matched continuous pretraining (CPT). After identical post-training (SFT + RLVR) across all variants, the RLP-initialized model retained a ~7–8% relative advantage, with the largest gains on reasoning-heavy benchmarks (AIME25, MMLU-Pro). 
Nemotron-Nano-12B v2: Applying RLP to a 12B hybrid Mamba-Transformer checkpoint yielded an overall average increase from 42.81% to 61.32% and an absolute +23% gain on scientific reasoning, even though the RLP run used ~200B fewer tokens (training for 19.8T vs 20T tokens; RLP applied for 250M tokens). This highlights data efficiency and architecture-agnostic behavior. 
https://github.com/NVlabs/RLP/blob/main/pdf/RLP_Reinforcement_as_a_Pretraining_Objective.pdf
RPT comparison: Under matched data and compute with Omni-MATH-style settings, RLP outperformed RPT on math, science, and overall averages—attributed to RLP’s continuous information-gain reward versus RPT’s sparse binary signal and entropy-filtered tokens. 
https://github.com/NVlabs/RLP/blob/main/pdf/RLP_Reinforcement_as_a_Pretraining_Objective.pdf
Positioning vs. Post-Training RL and Data Curation
Reinforcement Learning Pretraining (RLP) is orthogonal to post-training pipelines (SFT, RLVR) and shows compounding improvements after standard alignment. Because the reward is computed from model log-evidence rather than external verifiers, it scales to domain-agnostic corpora (web crawl, academic text, textbooks) and SFT-style reasoning corpora, avoiding the brittleness of narrow curated datasets. In compute-matched comparisons (including CPT with 35× more tokens to match FLOPs), RLP still led on overall averages, suggesting the improvements derive from objective design, not budget.
Key Takeaways
RLP makes reasoning a pretraining objective: sample a chain-of-thought before next-token prediction and reward it by information gain over a no-think EMA baseline.
Verifier-free, dense, position-wise signal: works on ordinary text streams without external graders, enabling scalable pretraining updates on every token.
Qwen3-1.7B results: +19% vs Base and +17% vs compute-matched CPT during pretraining; with identical SFT+RLVR, RLP retains ~7–8% gains (largest on AIME25, MMLU-Pro).
Nemotron-Nano-12B v2: overall average rises 42.81% → 61.32% (+18.51 pp; ~35–43% rel.) and +23 points on scientific reasoning, using ~200B fewer NTP tokens.
Training details that matter: update gradients only on thought tokens with a clipped surrogate and group-relative advantages; more rollouts (≈16) and longer thought lengths (≈2048) help; token-level KL anchoring offers no benefit.
Conclusion
RLP reframes pretraining to directly reward “think-before-predict” behavior using a verifier-free, information-gain signal, yielding durable reasoning gains that persist through identical SFT+RLVR and extend across architectures (Qwen3-1.7B, Nemotron-Nano-12B v2). The method’s objective—contrasting CoT-conditioned likelihood against a no-think EMA baseline—integrates cleanly into large-scale pipelines without curated verifiers, making it a practical upgrade to next-token pretraining rather than a post-training add-on.
Check out the Paper, Code and Project Page. Feel free to check out our GitHub Page for Tutorials, Codes and Notebooks. Also, feel free to follow us on Twitter and don’t forget to join our 100k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
The post NVIDIA Researchers Propose Reinforcement Learning Pretraining (RLP): Reinforcement as a Pretraining Objective for Building Reasoning During Pretraining appeared first on MarkTechPost.
*/

Why this design matters

RLP’s core advantage is practical: it yields a dense, position-wise learning signal that does not rely on external verifiers or curated answer keys. This lets it be applied at web-scale across general corpora, producing updates at every token position wherever thinking improves prediction. That contrasts with prior reinforcement-pretraining attempts that used sparse binary signals, filters, or externally judged correctness.

Empirical results

Qwen3-1.7B-Base: RLP during pretraining produced roughly +19% on math+science average vs the base model and +17% vs compute-matched continuous pretraining. After identical post-training (SFT + RLVR), the RLP-started model kept a ~7–8% relative advantage, especially on reasoning-heavy benchmarks such as AIME25 and MMLU-Pro.
Nemotron-Nano-12B v2: Applying RLP to a 12B checkpoint increased the overall average from 42.81% to 61.32% and produced an absolute +23-point gain on scientific reasoning, despite using roughly 200B fewer tokens for the RLP run. This suggests both data efficiency and architecture-agnostic benefits.

RLP also outperformed RPT under matched data and compute in Omni-MATH-style settings, attributed to RLP’s continuous information-gain reward vs RPT’s sparse binary signal.

Training considerations

Key implementation details include updating gradients only on thought tokens, using a clipped surrogate with per-token importance ratios and group-relative advantages, and drawing multiple rollouts (≈16) and longer thought lengths (≈2048) to reduce variance. The authors report token-level KL anchoring did not help.

Takeaway

RLP makes reasoning an explicit pretraining objective by rewarding thinking that increases the model’s evidence for the next token. It provides a verifier-free, dense reward that integrates with standard pipelines, scales across corpora and architectures, and yields durable reasoning improvements that persist after identical post-training alignment steps.