Podcast
The Next Big Breakthrough Will Be AIs Learning on the Job
Dwarkesh Podcast
- Research Bet On Massive RL For AGI
- Labs are betting that training AIs on millions of verifiable RL tasks will yield general problem-solving agents.
- This assumes scale can overcome deficits like sample inefficiency and lack of continual learning by amortizing training across many sessions. (Time 0:00:00)
- Verifiable Domains Still Need Grindable Simulators
- Verifiability alone isn’t enough; a domain must be grindable with deterministic, replayable simulators to enable massive parallel RL training.
- Computer use lags because sites and apps aren’t easily cloned for thousands of identical rollouts without heavy engineering. (Time 0:02:24)
- Sample Efficiency Is Essential For Real World Skills
- Many real-world skills can’t be trained with replayable environments because outcomes are sparse, delayed, and require real-world interaction.
- Sample efficiency is crucial since you can’t run thousands of parallel, resettable rollouts for business, politics, or markets. (Time 0:04:25)
- Deployment Is Wasted Learning Opportunity
- Most inference compute (~30–50%) currently produces no learning since deployment sessions aren’t distilled back into model weights.
- Real-world deployment reveals the most valuable data about usage, mistakes, and tacit organizational knowledge that models aren’t yet learning from. (Time 0:07:50)
- Weights Compression Versus Context Cache Limits
- Human learning compresses experience into weights; models relying on large context caches can’t scale that way.
- Gradient updates are sample inefficient, so current online learning works only when millions of users repeat identical objectives. (Time 0:09:03)
- Using Mercury To Automate My Bill Pay
- Dwarkesh uses Mercury to automate invoice processing by routing contractor emails to a dedicated account.
- Mercury scans invoices, extracts fields, and creates draft payments for him to review, eliminating manual bill tracking. (Time 0:10:58)
- Distill Session Learning With OPSD
- Use on-policy self-distillation (OPSD) to distill session-learned behavior back into base model weights.
- OPSD trains the base model to match a veteran teacher’s per-token probabilities, giving denser supervision than sparse RL rewards. (Time 0:12:52)
- OPSD Balances Sparse RL And Dense Supervision
- OPSD preserves RL’s sparse, targeted updates while providing denser token-level supervision, avoiding naive full-session supervised replay.
- This prevents overwriting unrelated capabilities by focusing updates only on what’s necessary to match the teacher’s performance. (Time 0:14:06)
- Dreaming As Test Time Training
- Dreaming means the model builds internal simulators to rehearse skills and generate massive additional training samples.
- If accurate, dreaming becomes a fourth scaling axis: test-time training where the model writes environments and trains against them. (Time 0:15:34)
- 2027 Vision Of On The Job Learning
- A 2027 scenario: RLVR yields competent agents that gather real-world experience and then distill weekly learnings back into base models.
- Combined with longer effective context windows, continual learning from deployment becomes the primary driver of improvement. (Time 0:17:27)