Sakana AI's Reinforcement-Learned Teachers: Revolutionizing Efficient Reasoning in LLMs

June 23, 2025 · 3 min

Introducing Reinforcement-Learned Teachers (RLTs)

Sakana AI presents an innovative framework called Reinforcement-Learned Teachers (RLTs) designed to enhance reasoning capabilities in large language models (LLMs) with a focus on efficiency and reusability. Unlike traditional reinforcement learning (RL) methods that require models to solve problems independently and suffer from sparse rewards and high computational costs, RLTs redefine the teacher-student paradigm by training smaller teacher models to generate detailed, step-by-step explanations rather than solving problems from scratch.

Rethinking Reinforcement Learning Objectives

Conventional RL trains models to solve problems autonomously using sparse, correctness-based rewards. These models then generate reasoning traces to teach smaller student models, causing a misalignment between RL objectives and the actual teaching use case. RLTs directly address this by providing the model both the problem and its solution, and tasking it with producing pedagogical explanations. The reward system is dense and aligned with the student’s comprehension, measuring how well the student can reproduce the solution based on the explanation.

Core Rewards: Solution and Explanation Scores

The RLT framework employs two crucial reward components:

Solution Score (rSS): Evaluates the student’s ability to reconstruct the correct solution from the explanation and problem.
Explanation Score (rKL): Assesses the logical coherence of the teacher’s explanation from the student’s perspective.

These combined form a dense reward signal encouraging clear, instructive explanations while bypassing traditional RL’s exploration bottleneck, allowing efficient training of smaller models.

Remarkable Performance of Small-Scale Teachers

Experiments demonstrate that a 7-billion parameter RLT model surpasses much larger models (32B+ parameters) on multiple challenging datasets such as AIME 2024, MATH 500, and GPQA Diamond. In a dataset comprising 17,000 questions:

RLT-7B outperforms DeepSeek R1, Bespoke-7B, and post-processed RL traces.
RLT-32B beats all 32B baseline models, despite being distilled from a smaller teacher.

RLTs not only deliver parameter efficiency but also yield better generalization, fewer formatting errors, and improved interpretability.

Cold-Starting Reinforcement Learning with RLTs

RLT-generated reasoning traces serve as superior cold-start material for RL training compared to those created by larger RL-trained models. Even without additional post-processing or refinement, these explanations significantly boost performance after RL fine-tuning.

Out-of-Domain Generalization and Zero-Shot Transfer

RLTs exhibit strong zero-shot transfer abilities. When applied to new domains like the arithmetic-based “Countdown” task, student models trained on RLT-generated explanations outperform those trained directly via RL. This suggests that teaching-focused RL models generalize better across tasks than those trained to solve problems from scratch.

Efficient and Scalable Training Pipeline

The RLT training process is computationally efficient:

Approximately 250 RL steps (~1 epoch)
Batch size of 256, group size of 64
Single-node training using Qwen2.5-7B-Instruct

All code and pretrained checkpoints are openly available on GitHub. Unlike traditional RL pipelines, RLTs produce raw outputs that require no post-processing, formatting corrections, or verification filters.

Summary

Sakana AI’s Reinforcement-Learned Teachers offer a scalable, cost-effective approach to distilling reasoning capabilities in LLMs. By focusing on teaching rather than solving, RLTs enable smaller models to outperform larger counterparts while facilitating better transferability and interpretability.