Skip to content

Podcast

Why I Don’t Think AGI Is Right Around the Corner

Dwarkesh Podcast

Source ↗ ← All highlights
  • LLMs Lack Continual Learning
    • Current LLMs struggle to perform normal human-like labor despite impressive language tasks.
    • Their fundamental lack of continual learning limits transformative economic impact for Fortune 500 workflows. Transcript: Dwarkesh Patel Continual learning. Sometimes people say that even if all AI progress totally stopped, the systems of today would still be far more economically transformative than the internet. I disagree. I think that the LLMs of today are magical, but the reason that the Fortune 500 aren’t using them to transform their workflows isn’t because the management is too stodgy. Rather, I think it’s genuinely hard to get normal human-like labor out of LLMs. And this has to do with some fundamental capabilities that these models lack. I like to think that I’m AI forward here at the Thorkesh Podcast, and I’ve probably spent on the order of 100 hours trying to build these little LLM tools for my post-production setup. The experience of trying to get these LLMs to be useful has extended my timelines. I’ll try to get them to rewrite autogenerative transcripts for readability the way a human would, or I’ll get them to identify clips from the transcript to tweet out. Sometimes I’ll get them to co-write an essay with me passage by passage. Now, these are simple, self-contained, short-horizon, language-in, language-out tasks, the kinds of assignments that should be dead center in the LLM’s repertoire. And these models are 5 out of 10 at these tasks. Don’t get me wrong, that is impressive. But the fundamental problem is that LLMs don’t get better over time the way a human would. This lack of continual learning is a huge, huge problem. The LLM baseline at many tasks might be higher than the average humans, but there’s no way to give a model high-level feedback. You’re stuck with the abilities you get out of the box. You can keep messing around with the system prompt, but in practice, this does not produce anywhere close to the kind of learning and improvement that human employees actually experience On the job. (Time 0:00:27)
  • Hands-on Experience with LLMs
    • Dwarkesh spent around 100 hours integrating LLM tools into his post-production work.
    • He found LLMs only ‘5 out of 10’ effective even on simple language tasks like rewriting transcripts. Transcript: Dwarkesh Patel Now, these are simple, self-contained, short-horizon, language-in, language-out tasks, the kinds of assignments that should be dead center in the LLM’s repertoire. And these models are 5 out of 10 at these tasks. Don’t get me wrong, that is impressive. But the fundamental problem is that LLMs don’t get better over time the way a human would. (Time 0:01:22)
  • Importance of Organic Learning
    • Human value chiefly comes from continual learning and self-correction, not raw intelligence.
    • LLMs can’t learn organically from their failures like humans do during real-world practice. Transcript: Dwarkesh Patel The reason that humans are so valuable and useful is not mainly their raw intelligence. It’s their ability to build up context, interrogate their own failures, and pick up small improvements and efficiencies as they practice a task. How do you teach a kid to play a saxophone? Well, you have her try to blow into one and listen to how it sounds and then adjust. Now, imagine if teaching saxophone worked this way instead. A student takes one attempt, and the moment they make a mistake, you send them away and you write detailed instructions about what went wrong. Now the next student reads your notes and tries to play Charlie Parker cold. When they fail, you refine your instructions for the next student. This just wouldn’t work. No matter how well honed your prompt is, no kid is just going to learn how to play saxophone from reading your instructions. But this is the only modality that we as users have to teach LLMs anything. (Time 0:02:04)
  • LLMs’ Short-lived Session Memory
    • LLMs show some short-term learning within sessions but lose contextual understanding after they end.
    • Summarizing sessions to preserve context can be brittle, especially outside text-based domains. Transcript: Dwarkesh Patel However, it’s just hard for me to see how that could happen within the next few years, given that there’s no obvious way to slot in online continuous learning into the kinds of models These LLMs are. Now, LLMs actually do get kind of smart in the middle of a session. For example, sometimes I’ll co-write an essay with an LLM. I’ll give it an outline, and I’ll ask it to draft an essay passage by passage. All that suggestions up to paragraph four will be bad. And so I’ll just rewrite the whole paragraph from scratch and tell it, hey, your shit sucked. This is what I wrote instead. And at that point, you can actually start giving good suggestions for the next paragraph. But this whole subtle understanding of my preferences and style is lost by the end of the session. Maybe the easy solution to this looks like a long rolling context window, like Claude Code has, which compacts the session memory into a summary every 30 minutes. I just think that titrating all this rich tacit experience into a text summary will be brittle in domains outside of software engineering, which is very text-based. Again, think about the example of trying to teach somebody how to play the saxophone using a long text summary of your learnings. Even Cloud Code will often reverse a hard-earned optimization that we engineered together before I hit slash compact, because the explanation for why it was made didn’t make it into The summary. (Time 0:03:42)
  • Limitations in Automating White-Collar Work
    • Without continual learning, LLMs won’t replace humans as employees despite automating many subtasks.
    • Context building is the key barrier preventing full white-collar job automation. Transcript: Dwarkesh Patel If AI progress totally stalls today, I think less than 25% of white-collar employment goes away. Sure, many tasks will get automated. Cloud4 Opus can technically rewrite auto-generated transcripts for me. But since it’s not possible for me to have it improve over time and learn my preferences, I still hire a human for this. Even if we get more data, without progress and continual learning, I think that we will be in a substantially similar position with all of white-collar work. Yes, technically, AIs might be able to perform a lot of subtasks somewhat satisfactorily, but their inability to build up context will make it impossible to have them operate as actual Employees at your firm. (Time 0:05:17)
  • Continual Learning Could Trigger Explosion
    • Solving continual learning will trigger a massive leap in model value and might cause an intelligence explosion.
    • Such AIs could learn across all copies simultaneously, accelerating collective intelligence rapidly. Transcript: Dwarkesh Patel Now, while this makes me bearish on transformative AI in the next few years, it makes me especially bullish on AI over the next few decades. When we do solve continuous learning, we’ll see a huge discontinuity in the value of these models. Even if there isn’t a software-only singularity with models rapidly building smarter and smarter successor systems, we might still see something that looks like a broadly deployed Intelligence explosion. AIs will be getting broadly deployed through the economy, doing different jobs and learning while doing them in the way that humans can. But unlike humans, these models can amalgamate their learnings across all their copies. So one AI is basically learning how to do every single job in the world. An AI that is capable of online learning might functionally become a superintelligence quite rapidly without any further algorithmic progress. However, I’m not expecting to see some open AI livestream where they announce that continual learning has totally been solved. (Time 0:05:53)
  • Challenges for AI Computer Use Agents
    • Reliable AI computer-use agents face hurdles like long rollout times, multimodal data scarcity, and technical complexity.
    • Current models are far from autonomously completing complex, multi-hour computer tasks reliably. Transcript: Dwarkesh Patel Computer use. When I interviewed anthropic researchers Shilta Douglas and Trenton Brickham on my podcast, they said that they expect reliable computer use agents by the end of next year. Now, we already have computer use agents right now, but they’re pretty bad. They’re imagining something quite different. Their forecast is that by the end of next year, you should be able to tell an AI, go do my taxes. It goes to your email, Amazon orders, and Slack messages, and it emails back and forth to everybody you need invoices from, it compiles all your receipts, it decides which things are Business expenses, asks for your approval on the edge cases, and then submits Form 1040 to the IRS. I’m skeptical. I’m not an AI researcher, so far be it to contradict them on the technical details. But from what little I do know, here are three reasons I’d bet against this capability being unlocked within the next year. One, as horizon lengths increase, rollouts have to become longer. The AI needs to do two hours worth of agentic computer use tasks before we even see if it did it right. Not to mention that computer use requires processing images and videos, which is already more computer intensive, even if you don’t factor in the longer rollouts. This seems like it should slow down progress. Two, we don’t have a large pre-training corpus of multimodal computer use data. I like this quote from Mechanize’s post on Automated Ink Software Engineering. Quote, for the past decade of scaling, we’ve been spoiled by the enormous amount of internet data that was freely available to us. This was enough to crack natural language processing, but not for getting models to become reliable, competent agents. Imagine trying to train GPT-4 on all the Again, I’m not at the lab, so maybe text-only training already gives you a great prior on how different UIs work and what the relationships are Between different components. Maybe RL fine-tuning is so sample efficient that you don’t need that much data. But I haven’t seen any public evidence which makes me think that these models have suddenly become less data-hungry, especially in domains where they’re substantially less practiced. Alternatively, maybe these models are such good front-end coders that they can generate millions of toy UIs for themselves to practice on. For my reaction to this, see the bullet point below. Three, even algorithmic innovations, which seem quite simple in retrospect, seem to have taken a long time to iron out. The RL procedure, which DeepSeq explained in their R1 paper, seems simple at a high level. And it took two years from the launch of GPT-4 to the launch of O1. Now, of course, I know that it’s hilariously arrogant to say that R1 or O1 were easy. I’m sure a ton of engineering, debugging, and pruning of alternative ideas was required to revive the solution. But that’s precisely my point. Seeing how long it took to implement the idea, hey, let’s train our model to solve verifiable math and coding problems, makes me think that we’re underestimating the difficulty of Solving a much gnarlier problem of computer use, where you’re operating on a totally different modality with much less data. (Time 0:06:59)
  • AI Models Show Real Reasoning
    • Advanced models like Claude Code demonstrate genuine reasoning by breaking down problems and self-correcting.
    • This progress refutes pessimism about meaningful AI reasoning capabilities today. Transcript: Dwarkesh Patel Have you read the reasoning traces of O3 or Gemini 2.5? It’s actually reasoning. It’s bridging down the problem, it’s thinking about what the user wants, it’s reacting to its own internal monologue, and correcting itself when it notices that it’s pursuing an unproductive Direction. How are we just like, oh yeah, of course machines are going to go think a bunch, come up with a bunch of ideas, and come back with a smart answer? That’s what machines do. Part of the reason some people are too pessimistic is that they haven’t played around with the smartest models operating in the domains that they’re most competent in. Giving Claude code a vague spec and then sitting around for 10 minutes until it’s zero shots of working application is a wild experience. How did it do that? You could talk about circuits and training distributions and RL and whatever, but the most proximal, concise, and accurate explanation is simply that it’s powered by baby artificial Intelligence. (Time 0:09:51)
  • Prepare for Varied AGI Timelines
    • Prepare for a wide range of AGI timelines, including plausible misaligned superintelligence by 2028.
    • Expect AI progress to slow post-2030 as hardware scaling limits emerge, with algorithmic advances becoming critical. Transcript: Dwarkesh Patel My probability distribution is super wide, and I want to emphasize that I do believe in probability distributions, which means that work to prepare for a misaligned 2028 ASI still Makes a ton of sense. I think that’s a totally plausible outcome. But here are the timelines at which I’d make a 50-50 bet. An AI that can do taxes end-to for my small business as well as a competent general manager could in a week, including chasing down all the receipts on different websites and finding All the missing pieces and emailing back and forth with anyone we need to hassle for invoices, filling out the form and sending it to the IRS. 2028. I think we’re in the GPT-2 era for computer use, but we have no free training corporates and the models are optimizing for a much sparser reward over a much longer time horizon using action Primitives that they’re unfamiliar with. That being said, the base model is decently smart and might have a good prior over computer use tasks. Plus, there’s a lot more compute and AI researchers in the world, so it might even out. Preparing taxes for a small business feels like for computer use, what GPT-4 was for language. And it took four years to get from GPT-2 to GPT-4. Just to clarify, I’m not saying that we won’t have really cool computer use demos in 2026 and 2027. GPT-3 was super cool, but not that practically useful. I’m saying that these models won’t be capable of end-to handling a week-long and quite involved project, which involves computer use. Okay, and the other prediction is this. An AI that learns on the job as easily, organically, seamlessly, and quickly as a human, for any white-collar work. For example, if I hire an AI video editor, after six months, it has as much actionable, deep understanding of my preferences, our channel, and what works for the audience as a human would. This, I would say, 2032. Now, while I don’t see an obvious way to slot in continuous online learning into current models, seven years is a really long time. GPT-1 had just come out this time seven years ago. It doesn’t seem implausible to me that over the next seven years we’ll find some way for these models to learn on the job. Okay, at this point you might be reacting, look, you made this huge fuss about how continual learning is such a big handicap, but then your timeline is that we’re seven years away from What at a minimum is a broadly deployed intelligence explosion. And yeah, you’re right. I’m forecasting a pretty wild world within a relatively short amount of time. AGI timelines are very log normal. It’s either this decade or bust. Not really bust, more like lower marginal probability per year, but that’s less catchy. AI progress over the last decade has been driven by scaling training compute for frontier systems. Over four acts a year. This cannot continue beyond this decade, whether you look at chips, power, even the raw fraction of GDP that’s used on training. After 2030, AI progress has to mostly come from algorithmic progress. But even there, the low-hanging fruits will be plucked, at least under the deep learning paradigm, so the yearly probability of AGI craters after 2030. This means that if we end up on the longer side of my 50-50 bets, we might well be looking at a relatively normal world up to the 2030s or even the 2040s. (Time 0:10:46)