Yuxi Li · April 14, 2025 · Commentary on Silver & Sutton (2025)
Experience is critical for further progress in AI, including LLMs. Reinforcement learning is the natural framework for learning from that experience — but generalization across domains is far harder than recent successes imply.
Contents
Welcome to the Era of Experience (April 2025) by David Silver and Richard Sutton argues that AI is transitioning from an era of imitation — learning by mimicking human-generated data — into an era of experience — learning by acting in the world and using the resulting feedback as the primary training signal.
"In the era of experience, agents will learn primarily from their own experience of the world, rather than from human-generated data." — Silver & Sutton (2025)
Human data encodes the limits of human understanding. An agent trained purely on human demonstrations can at best match human performance — it inherits human biases, human blind spots, and the human knowledge horizon. AlphaZero demonstrated the escape hatch: trained entirely via self-play with no human games, it surpassed every human and every human-trained predecessor. The paper argues this template should generalize far beyond games.
Experience is anchored in real environment feedback — not human labels or model-generated pseudo-labels. The signal comes from the world itself, making it harder to hallucinate or circular-reason away from reality.
Agents act and observe consequences. This causal loop — action → observation → reward — is absent in passive supervised learning. It enables counterfactual reasoning and exploration of novel states no human has visited.
Experience accumulates without a fixed training cutoff. Unlike a static dataset, an agent in deployment continues to generate new experience, enabling lifelong learning and adaptation to distribution shift.
Each agent's experience stream is unique to its history of interactions. Agents can develop specialised expertise through personal trajectories — analogous to how individual humans develop domain mastery through their specific life experiences.
The paper is explicit: reinforcement learning is not merely one option among many — it is the natural framework for learning from experience. The Markov Decision Process formalism (states, actions, rewards, transitions) directly models experience streams. Policy gradient methods, value functions, and model-based planning all operate on this substrate.
Silver and Sutton point to a progression: AlphaGo (imitation + RL) → AlphaZero (pure RL, no human data) → AlphaProof (RL with formal verifier) → the coming generation of agents that learn from open-ended world interaction. Each step reduced dependence on human data; the next step eliminates it as the primary signal.
A key claim: reward functions need not be anchored in human preferences. In domains with objective ground truth (games, maths, science), the environment itself supplies rewards. The paper envisions agents that pursue goals defined by the structure of the domain rather than by what humans think is good — potentially discovering strategies and solutions that humans could not have specified or anticipated.
David Silver led the AlphaGo and AlphaZero teams at DeepMind. Richard Sutton co-authored the canonical RL textbook (Reinforcement Learning: An Introduction) and authored the influential "Bitter Lesson" essay (2019), arguing that general methods leveraging computation always outperform hand-crafted domain knowledge. This paper can be read as the Bitter Lesson applied specifically to the data source question: experience (general, scalable) beats human data (specific, bounded).
The Derby Mill Series Episode 02 (January 2025) featured Richard Sutton alongside economist Sendhil Mullainathan discussing DeepSeek — the Chinese startup that matched GPT-4-class performance at a fraction of the compute cost. The connection to the Era of Experience thesis is direct: DeepSeek's efficiency gains came primarily from reinforcement learning techniques, not from more data or bigger models.
DeepSeek employed RL and related techniques to build an ultra-efficient system requiring fewer chips and substantially less electrical power than OpenAI or Anthropic equivalents — while achieving comparable benchmark performance.
The economist flagged "the big cognitive error many are making about DeepSeek": treating it as a competitive threat to US labs rather than as evidence that RL-driven efficiency will democratize AI development globally. The moat is not compute — it is algorithmic insight.
For Sutton, DeepSeek validates the RL path: you do not need infinite human-labeled data or trillion-parameter dense models if you have a good reward signal and efficient RL. Experience-based learning is more sample-efficient than imitation at scale.
Li endorses Silver & Sutton's core thesis but adds two critical qualifications the paper underweights: (1) the concept of ground truth is more nuanced than the paper implies, and (2) achievements in games, maths, and coding may not transfer to the broader world.
"Games, maths and coding are special problems, with special, perfect environments. The achievements in such problems may not be straightforwardly generalizable / transferable to other problems." — Yuxi Li
Before the LLM era, ground truth data was a prerequisite for AI and machine learning. Li identifies three sources:
World models and formal game rules (e.g. chess, Go). AlphaZero epitomises this: a perfect game engine guarantees correctness of every training signal.
Theorem provers, type checkers, syntax validators. AlphaProof uses Lean as a perfect verifier. Correctness is decidable — no ambiguity in the training signal.
The general case: agents interact with reality. Noisy, partially observable, adversarial. The world is technically a perfect model of itself — but we only ever see samples.
Sources 1 and 2 can be viewed as special cases of Source 3 — interaction with perfectly specified environments. The LLM community has often blurred this boundary, producing what Li calls "information perpetual motion machines": self-improvement loops (self-play, self-critique) that lack genuine external ground truth.
Even in maths, LLM papers often score against scalar results (did the final answer match?) rather than proof correctness. In coding, passing a handful of unit tests or executing without errors is not equivalent to full specification satisfaction. These are proxies — not true ground truth. Many self-xxx (self-play, self-distillation, self-refinement) approaches implicitly assume the model itself can judge its own outputs, creating circular epistemic loops.
graph TD
GT["Ground Truth"]
GT --> S1["Source 1: Perfect Rules / World Models"]
GT --> S2["Source 2: Formal Verifiers"]
GT --> S3["Source 3: World Interaction (Noisy)"]
S1 --> AZ["AlphaZero — perfect game engine"]
S2 --> AP["AlphaProof — Lean theorem prover"]
S3 --> ROB["Robotics — simulation-reality gap"]
S3 --> HUM["Human-centric — expert feedback only"]
S1 --> OBJ["Objective reward ✓"]
S2 --> OBJ
S3 --> BOTH["Objective or Subjective"]
BOTH --> RLHF["→ RLHF / Inverse RL"]
The type of reward signal available to an agent is domain-dependent, not a design choice:
| Domain | Reward Type | Ground Truth Source | Example |
|---|---|---|---|
| Games | Objective | Rule engine | AlphaZero, Stockfish |
| Formal maths | Objective | Proof verifier | AlphaProof, Lean |
| Software engineering | Near-objective | Test suite / compiler | SWE-bench |
| Natural science | Objective | Physical measurement | AlphaFold |
| Social science | Subjective | Expert judgment | Economic forecasting |
| Arts & humanities | Subjective | Expert judgment | Creative writing, translation |
| Human-centric tasks | Subjective | User preference | Customer service, therapy |
Natural and formal sciences, and engineering, admit objective reward structures. Social sciences, arts, and humanities do not — and Li argues expert feedback may be the best signal we can ever obtain for these domains, not just a temporary substitute.
When no explicit reward function exists, two principled approaches emerge:
Behavioural cloning: learn to mimic expert demonstrations. Computationally cheap and intuitive — but insufficient. AlphaGo (imitation from human games) was surpassed by AlphaZero (pure self-play RL) in every measurable way. Imitation learning cannot exceed the performance ceiling of the demonstrating expert.
Key failure: distribution shift at test time. The agent encounters states never seen in the expert corpus and has no recovery mechanism.
When the reward function is unknown or unspecifiable, inverse RL recovers it from expert behaviour. RLHF (Reinforcement Learning from Human Feedback) is the dominant practical instantiation: learn a reward model from human preference comparisons, then optimize policy against it.
Li notes that iterative inverse RL — alternating reward model updates with policy improvement — can be interpreted as model-based RL over the reward model. This framing unifies RLHF with the broader RL literature and clarifies what makes it principled.
Limitation: reward model accuracy bounds policy quality. Reward hacking (Goodhart's Law applied to learned rewards) is a persistent failure mode.
Learn a model of environment dynamics; use it to plan. The reward model in iterative RLHF is one instance. MuZero, Dreamer, and World Models are canonical examples. Key advantage: sample efficiency — agent can "imagine" trajectories without costly real-world interaction.
Li's insight: treating the reward model as the learned model in model-based RL provides a clean theoretical grounding for iterative RLHF pipelines.
A fundamental axis of difficulty: can the environment be fully simulated at low cost?
Games, maths, coding. Perfect simulators. Zero marginal cost per interaction. Infinite parallelism possible. Ideal RL setting.
Science / engineering with simulation. Sim-to-real gap introduces systematic error. Validation still requires physical experiments. Data cost is significant.
Robotics, medicine, social interaction. No faithful simulator. Data collection is costly, slow, and sometimes ethically constrained. Hardest RL setting.
The simulation-to-reality gap is not merely a technical problem — it is a structural feature of physical and human-centric domains. Dexterous manipulation learned in simulation routinely fails on real hardware. Social dynamics are irreducible to any simulator we can build.
Games AI achieved clear superhuman performance because:
Maths and coding approach this but face fundamental theoretical barriers:
Gödel's incompleteness theorems: no consistent formal system can prove all true statements within itself. A fully autonomous, provably correct mathematical reasoner is impossible within any single axiomatic framework.
The halting problem: it is undecidable whether an arbitrary program halts. A fully autonomous, correct code-generation agent that can verify its own output for all inputs is impossible.
Practical implication: "superhuman" in maths and coding means "better than humans on the benchmark tasks we can measure" — not "solves all problems correctly."
Li uses machine translation as an illustrative example of where "superhuman" breaks down entirely as a concept:
Score 0–10. Higher = more favorable for autonomous RL agent development.
Ground truth is a first principle. Before pursuing any RL-based approach, ask: where does the training signal come from, and how reliable is it? Self-improvement loops without external ground truth are epistemically fragile.
Expect breakthroughs in maths and coding (high digitality, near-objective ground truth), likely in science and engineering (physical experiments as verifiers). These domains are structured enough for RL to find strong signals.
Achievements in games, maths, and coding do not straightforwardly transfer to human-centric domains. For translation, therapy, creative work: human expertise and iterative inverse RL (RLHF) may be structurally irreplaceable — not just current-era engineering constraints.
Call to action from Li:
| Source | Type | Key Content |
|---|---|---|
| Welcome to the Era of Experience | Paper | Silver & Sutton (2025) — the primary text being commented on |
| Is Human Data Enough? — David Silver | Talk | Silver's argument for moving beyond human-generated data |
| Towards Superhuman Intelligence — David Silver (RLC 2024) | Talk | Silver's RLC keynote on the path to AGI via experience |
| A Perspective on Intelligence — Rich Sutton | Talk | Sutton's broader philosophy on intelligence and RL |
| DeepSeek — Derby Mill Series Ep 02 | Podcast | Sutton, Mullainathan et al. on DeepSeek's RL-driven efficiency |
| Rich Sutton Brings Reinforcements | Podcast | Sutton with Littman & Ackley (Dec 2023) — foundational RL perspectives |
| Rich Sutton on Intelligence, LLMs, Scaling Laws | Short video | Yuxi Li's condensed take on Sutton's views |
| Bitter Lesson implies Scaling "Laws"? | Short video | Yuxi Li on the bitter lesson and its scaling implications |
Original commentary: yuxili.substack.com