Skip to content

Podcast

Andrej Karpathy on Code Agents, AutoResearch, and the Loopy Era of AI

Technology | Startups

Source ↗ ← All highlights
  • December Marked The Agent Workflow Flip
    • Andrej Karpathy says coding flipped around December from mostly typing himself to mostly delegating to agents.
    • He claims he has barely typed code since then and now spends long days trying to “express my will” to many agents. Transcript: Sarah Guo I remember walking into the office at some point and you were like really locked in. And I was asking what you were up to. And you’re like, I just I have to code for 16 hours a day or code’s not even the right verb anymore. Right. But I have to express my will to my agents for 16 hours a day. Manifest. Because there’s been a jump in capability. What’s happening? Tell me about your experience. Andrej Karpathy Yeah, I kind of feel like I was in this perpetual, I still am often in this state of AI psychosis, just like all the time, because there was a huge unlock in what you can achieve as a person, As an individual, right? Because you were bottlenecked by, you know, your typing speed and so on. But now with these agents, it really, I would say in December is when it really just something flipped where I kind of went from 80-20 of like, you know, to like 20-80 of writing code by Myself versus just delegating to agents. And I don’t even think it’s 20-80 by now. I think it’s a lot more than that. I don’t think I’ve typed like a line of code probably since December, basically, which is like an extremely large change. I was talking to it, like, for example, I was talking about it to, for example, my parents and so on. And I don’t think like a normal person actually realizes that this happened or how dramatic it was. Like literally, like if you just find a random software engineer or something like that at their desk and what they’re doing, like their default workflow of, you know, building software Is completely different as of basically December. So I’m just like in the state of psychosis of trying to figure out like what’s possible, trying to push it to the limit. How is it? How can I have not just a single session of, you know, cloud code or codex or some of these agent harnesses? How can I have more of them? How can I do that appropriately? And then how can I use these claws? What are these claws? And so there’s like a lot of new things. I want to be at the forefront of it, you know, and I’m very antsy that I’m not at the forefront of it. And I see lots of people on Twitter doing all kinds of things and they all sound like really good ideas and I need to be at the forefront or I feel extremely nervous. And so I guess I’m just in this psychosis of like what’s possible, like because it’s unexplored fundamentally. Well, (Time 0:01:03)
  • The Bottleneck Has Shifted To Agent Orchestration
    • Karpathy argues current limits feel less like missing model capability and more like operator skill in orchestration.
    • He describes tiling many agents across repos, assigning independent features, research, and planning, then reviewing outputs as larger “macro actions.” Transcript: Sarah Guo How do you think about your own capacity now to like explore or to do projects? Like, what is it limited by? Andrej Karpathy Yeah, what is it limited by? Just, I think everything, like so many things, even if they don’t work, I think to a large extent, you feel like it’s a skill issue. It’s not that the capability is not there. It’s that you just haven’t found a way to string it together of what’s available. Like I just don’t, I didn’t give good enough instructions in the agent’s MD file or whatever it may be. I don’t have a nice enough memory tool that I put in there or something like that. So it all kind of feels like skill issue when it doesn’t work to some extent. You want to see how you can paralyze them, et cetera. And you want to be Peter Steinberg, basically. So Peter is famous. He has a funny photo where he’s in front of a monitor with lots of, like he uses codecs. So lots of codecs agents tiling the monitor. And they all take about 20 minutes if you prompt them correctly and use the high effort. And so they all take about 20 minutes. They have multiple, you know, 10 repos checked out. And so he’s just going between them and giving them work. It’s just like you can move in much larger macro actions. It’s not just like, here’s a line of code, here’s a new function. It’s like, here’s a new functionality and delegate it to agent one. Here’s a new functionality that’s not going to interfere with the other one. Give it agent two. And then try to review their work as best as you can, depending on how much you care about that code. Like, what are these macro actions that I can like manipulate my software repository by? And like, another agent is doing some like research, another agent is writing code, another one is coming up with a plan for some new implementation. And so everything just like happens in these like macro actions over your repository. And you’re just trying to become like really good at it and develop like a muscle memory for it is extremely, yeah, it’s very rewarding, number one, because it actually works. But it’s also kind of like the new thing to learn. (Time 0:03:24)
  • Token Throughput Is The New Compute Constraint
    • Agent-heavy work makes humans feel compute-bound again, except the scarce resource is tokens rather than GPUs.
    • Karpathy compares leftover subscription quota to idle PhD GPUs and says he feels nervous when token throughput is not maximized. Transcript: Sarah Guo Yeah, I do feel like my instinct is like, whenever I’m waiting for an agent to complete something, the obvious thing to do is like, well, I can do more work, right? Like if I have access to more tokens, then like I should just paralyze at more tasks. And so that’s very stressful because if you don’t feel very bounded by your ability to spend on tokens, then, you know, you are the bottleneck in the system that has max capability. Andrej Karpathy Yeah, if you’re not maximizing your subscription, at least. And ideally for multiple agents, like if you run out of the quota on Codex, you should switch to Cloud or whatnot. I don’t know. Like that’s what I’ve been trying to do a little bit. And I feel nervous when I have subscription left over. That just means I haven’t maximized my token throughput. So I actually kind of experienced this when I was a PhD student. You would feel nervous when your GPUs are not running. Like you have GPU capability and you’re not maximizing the available flops to you. But now it’s not about flops, it’s about tokens. So what is your token throughput? And what token throughput do you command? Sarah Guo I would actually argue that it’s very interesting that we had, you know, at least 10 years where in many engineering tasks, people just didn’t feel compute bound. Right. And the entire industry feels that now. They feel like they felt resource bound. And now that you have this big capability jump, you’re like, oh, actually it’s not, you know, my ability to access the compute anymore. Like I’m the binding constraint. Andrej Karpathy Yeah, it’s a skill issue. Yeah. Which is very empowering because, yeah, because you could be getting better. (Time 0:05:07)
  • Mastery Means Persistent Multi Agent Systems
    • Karpathy thinks mastery means moving up the stack from one agent session to persistent collaborating systems with memory.
    • He highlights OpenClaw’s breakthroughs as personality, better memory than simple context compaction, and a WhatsApp-style portal for automation. Transcript: Sarah Guo Where do you think it goes? Like if you just think about like, okay, you know, Andre’s iterating and everybody else is for 16 hours a day, getting better at using coding agents. Like what does it look like in a year? Of like you’ve reached mastery. Andrej Karpathy Yeah, what does mastery look like, right? At the end of the year or like two, three years, five years, 10 years, et cetera. Well, I think everyone is basically interested in like going up the stack. So I would say, yeah, it’s not about a single session with your agent, multiple agents, how do they collaborate and teams and so on. So everyone’s trying to figure out what that looks like. And then I would say claw is also kind of an interesting direction because it really, when I say a claw, I mean this like layer that kind of takes persistence to a whole new level. Like it’s something that like keeps looping. It’s like, it’s not something that you are interactively in the middle of. It kind of like has its own little sandbox, its own little, you know, it kind of like does stuff on your behalf, even if you’re not looking kind of thing. And then also has like maybe more sophisticated memory systems, et cetera, that are not yet implemented in agents. So OpenClaw has a lot more sophisticated memory, I would say, than what you would get by default, which is just a memory compaction when your context runs out, right? Sarah Guo You think that’s the piece that resonated for more users versus perhaps broader tool access? Andrej Karpathy For OpenClaw? Yeah. I think there’s at least five things that resonated with users. There’s a lot of really good ideas in here. Sarah Guo Yeah, good job, Peter. Andrej Karpathy I mean, Peter has done a really amazing job. I saw him recently and I talked to him about it and he’s very humble about it, but I think he innovated simultaneously in like five different ways and put it all together. So for example, like the soul MD document, like he actually really crafted a personality that is kind of compelling and interesting. And I feel like a lot of the current agents, they don’t get this correctly. I actually think a Claude has a pretty good personality. It feels like a teammate and it’s excited with you, et cetera. I would say, for example, Codex is a lot more dry um which is kind of interesting because in chashy pt codex is like a lot more upbeat and highly sycophantic but i would say codex the coding Agent is very dry it doesn’t it doesn’t seem to care about what you’re creating it’s kind of like oh i implemented it it’s like okay but do you understand what we’re building it’s true You know it doesn’t it and the other thing i would say is for example with claude i think they dialed the psychophancy fairly well where when Claude gives me praise, I do feel like I slightly Deserve it. Because sometimes I kind of give it like not very well formed thoughts. And I give it an idea that I don’t think is fully baked. And it doesn’t actually react very strongly. It’s like, oh, yeah, we can implement that. But when it’s a really good idea, by my own account, it does seem to reward it a bit more. And so I kind of feel like I’m trying to like earn its praise, which is really weird. And so I do think the personality matters a lot. And I think a lot of the other tools maybe don’t appreciate it as much. And I think in this aspect also, Peter really cares about this. And so that was correct. And then the memory system and then just, you know, he’s just having fun with this. And then the single WhatsApp portal tool of the automation. (Time 0:06:36)
  • Dobby Turned Six Home Apps Into One Agent
    • Karpathy built a home claw called Dobby that runs lights, HVAC, shades, pool, spa, Sonos, and security through WhatsApp.
    • It scanned his LAN, found unsecured Sonos APIs, created dashboards, and texts him when a FedEx truck arrives using change detection plus vision. Transcript: Sarah Guo Is there something that you have done personally with your claws beyond software engineering that you think is fun or interesting? Andrej Karpathy Yeah. So in January, I had a claw. I went through a period of claw psychosis. So I have a claw, basically, that takes care of my home. And I call him Dobby the Elf Claw. And basically, I used the agents to find all of the smart home subsystems of my home on the local area network, which I was kind of surprised that worked out of the box. Like I just told it that I think I have Sonos at home. Like, can you try to find it? And it goes and like IP scan of all the basically computers on the local area network. And it found the Sonos thing, the Sonos system. And it turned out that there’s no password protection or anything like that. It just logged in. And it’s like, oh, yeah, you have these Sonos systems installed. Let me try to reverse engineer how it’s working. It does some web searches and it finds like, okay, these are the API endpoints. And then it’s like, do you want to try it? And I’m like, whoa, like you just did that. And I’m like, yeah, can you try to play something in the study? And it does. And music comes out. And I’m like, I can’t believe I just… Sarah Guo That’s crazy. That’s like three prompts. Andrej Karpathy I can’t believe I just typed in like, can you find my Sonos? And that suddenly it’s playing music. And it did the same for lights. And so basically, like, it kind of hacked in, figured out the whole thing, created APIs, created a dashboard. So I could see the command kind of center of, like, all of my lights in the home. And then it was, like, switching lights on and off. And, you know, so I can ask it, like, go be at sleepy time. And when it’s sleepy time, that just means all the lights go off, et cetera, and so on. So it controls all of my lights, my HVAC, my shades, the pool and spa, and also my security system. So I have a camera pointed outside of the house. And anytime someone rolls in, I have a Quinn model that looks at the videos. So first of all, there’s change detection. And then based on change detection, it goes to Quinn. And then it actually tells me, it sends me a text to my WhatsApp. It shows an image from the outside and it says, hey, a FedEx truck just pulled up, FedEx truck just pulled up and you might want to check it and you got a new mail or something like that. And Dobby just texts me this. It’s really incredible. So Dobby is in charge of the house. I text with it through WhatsApp. And it’s been really fun to have these macro actions that maintain my house. I haven’t really pushed it way more beyond that. And I think people are doing a lot more crazy things with it. But for me, even just a home automation setup, I used to use six apps, completely different apps. And I don’t have to use these apps anymore. Adobe controls everything in natural language. It’s amazing. And so I think I haven’t even pushed a paradigm fully, but already that is so helpful and so inspiring, I would say. (Time 0:09:18)
  • Agent First Software Could Replace Many Apps
    • Karpathy thinks many consumer apps should collapse into exposed APIs that agents call directly on users’ behalf.
    • His treadmill and smart-home examples suggest today’s custom UIs are transitional, and agent-first software will unify fragmented tools. Transcript: Sarah Guo You think that’s indicative of like what people want from a user experience perspective with software? Right. Because I don’t think, you know, it’s pretty ignored that it takes humans effort to like learn new software, like new UI. Yeah. Andrej Karpathy I think to some extent, that’s right. It’s like working backwards from how people think an AI should be. Because what people have in their mind of like what an AI is, is not actually what an LLM is by like in a raw sense. Like LLM is a token generator, you know, like more tokens come out. But what they think of is like this persona identity that they can tell stuff and it remembers it, you know, and it’s just kind of an entity behind the WhatsApp. It’s like a lot more understandable. So I think to some extent, it’s like matching the expectations that humans already have for what AI should behave. But under the hood, there’s like a lot of technical details go into that. And LLMs are too raw of a primitive to actually type check as AI, I think, for most people, if that makes sense. Sarah Guo Yeah, I think that’s like how we understand what the AI is and like the description of it as Dobby or some personality obviously resonates with people. I also think that the unification that you did across your six different software systems for your home automation speaks to a different question of like, do people really want all The software that we have today? Yeah. Right. Because I would argue like, well, you have the hardware, but you’ve now thrown away the software or the UX layer of it. Do you think that’s what people want? Andrej Karpathy Yeah. I think there’s this like, there’s this sense that these apps that are in the app store for using these smart home devices, et cetera, these shouldn’t even exist kind of in a certain sense. Like, shouldn’t it just be APIs and shouldn’t agents be just using it directly? I can do all kinds of home automation stuff that any individual app will not be able to do, right? And an LLM can actually drive the tools and call all the right tools and do pretty complicated things. And so in a certain sense, it does point to this, like maybe there’s like an overproduction of lots of custom bespoke apps that shouldn’t exist because agents kind of like crumble them Up and everything should be a lot more just like exposed API endpoints and agents are the glue of the intelligence that actually like tool calls all the parts. Another example is like my treadmill. There’s an app for my treadmill and I wanted to like keep track of how often I do my cardio. But like, I don’t want to like log into a web UI and go through a flow and et cetera. Like all this should just be like make APIs available. And this is kind of, you know, going towards the agentic sort of web or like agent first tools and all this kind of stuff. So I think the industry just has to reconfigure in so many ways that it’s like the customer is not the human anymore. It’s like agents who are acting on behalf of humans. And this refactoring will probably be substantial in a certain sense. One way that people sometimes push back on this is like, do we expect people to vibe code some of these tools? Do we expect normal people to do this kind of stuff that I described? But I think to some extent, this is just, you know, technology as it exists today. And right now there is some vibe coding and I’m actually watching it and I’m working with the system, but I kind of feel like this kind of stuff that I just talked about, this should be free, Like in a year or two or three. There’s no vibe coding involved. This is trivial. This is table stakes. (Time 0:11:45)
  • AutoResearch Removes The Human From Model Tuning
    • AutoResearch aims to remove the human researcher from the optimization loop and let agents run longer autonomously.
    • Karpathy says an overnight run found better GPT-2 tuning than he had by hand, including weight decay on value embeddings and Adam beta adjustments. Transcript: Sarah Guo Was the um i mean you’ve talked about like being able to train or at least optimize a a model as a task you want to see agents do for a long time like what was the motivation behind auto research Andrej Karpathy Auto research yeah so i think like i had a tweet earlier where i kind of like said something along the lines of, to get the most out of the tools that have become available now, you have to Remove yourself as the bottleneck. You can’t be there to prompt the next thing. You need to take yourself outside. You have to arrange things such that they’re completely autonomous. And the more, you know, how can you maximize your token throughput and not be in the loop? This is the goal. And so I kind of mentioned that the name of the game now is to increase your leverage. I put in just very few tokens just once in a while and a huge amount of stuff happens on my behalf. And so auto-research, like I tweeted that and I think people liked it and whatnot, but they haven’t like maybe worked through like the implications of that. And for me, auto-research is an example of like an implication of that, where it’s like, I don’t want to be like the researcher in the loop, looking at results, et cetera. Like I’m holding the system back. So the question is, how do I refactor all the abstractions so that I’m not, I have to arrange it once and hit go. The name of the game is how can you get more agents running for longer periods of time without your involvement, doing stuff on your behalf. And auto research is just, yeah, here’s an objective, here’s a metric, here’s your boundaries of what you can and cannot do and go. And yeah. Sarah Guo Were surprised at its effectiveness. Andrej Karpathy Yeah, I didn’t expect it to work because I have the Project NanoChat. And fundamentally, I think a lot of people are very confused with my obsession for training GPT-2 models and so on. But for me, training GPT-2 models and so on is just a little harness, a little playground for training LLMs. And fundamentally, what I’m more interested in is like this idea of recursive self-improvement and to what extent you can actually have LLMs improving LLMs. Because I think all the Frontier Labs, this is like the thing for obvious reasons. And they’re all trying to recursively self-improve, roughly speaking. And so for me, this is kind of like a little playpen of that. And I guess I like tuned Net already quite a bit by hand in a good old fashioned way that I’m used to. Like I’m a researcher. I’ve done this for like, you know, two decades. I have some amount of like, what is the opposite of hubris? Yeah. Sarah Guo Earned a confidence. Andrej Karpathy Okay. I have like two decades of like, oh, I’ve trained this model like thousands of times. I’ve like, so I’ve done a bunch of experiments. I’ve done hyperprimary tuning. I’ve done all the things I’m very used to and I’ve done for two decades. Yeah. And I’ve gotten to a certain point and I thought it was like fairly well tuned. And then I let our research go for like overnight and it came back with like tunings that I didn’t see. And yeah, I did forget like the weight decay on the value embeddings and my atom betas were not sufficiently tuned. And these things jointly interact. So like once you tune one thing, the other things have to potentially change too. You know, I shouldn’t be a bottleneck. I shouldn’t be running these hyperparameters or optimizations. I shouldn’t be looking at the results. There’s objective criteria in this case. So you just have to arrange it so that it can just go forever. So that’s a single sort of version of auto research of like a single loop trying to improve. And I was surprised that it found these things that I, you know, the repo is already fairly well tuned and still found something. (Time 0:16:21)
  • Research Orgs Can Be Tuned Like Software
    • Karpathy sees research organizations as code expressed in markdown roles, queues, and procedures that can themselves be optimized.
    • Different ProgramMDs could encode risk tolerance, workflow, and coordination, then compete for improvement like any other system. Transcript: Sarah Guo So you’re saying our research efforts are going to get more efficient, like we’re going to have better direction for when we scale as well, if we can do this experimentation better. Andrej Karpathy Yeah, I would say that the most interesting project, and probably what the Frontier Labs are working on, is you experiment on the smaller models, you try to make it as autonomous as possible, Remove researchers from the loop. They have way too much, what is the opposite? Earned confidence? Yeah. Yeah, they don’t know. They shouldn’t be touching any of this really. And so you have to like rewrite the whole thing because right now, I mean, certainly they can contribute ideas, but okay, they shouldn’t actually be enacting those ideas. There is a queue of ideas and there’s maybe an automated scientist that comes up with ideas based on all the archive papers and GitHub repos. And it funnels ideas in, or researchers can contribute ideas, but it’s a single queue. And there’s workers that pull items and they try them out. And whatever works just gets sort of put on the feature branch. And maybe some people monitor the feature branch and merge to the main branch sometimes. So yeah, just removing humans from all the processes and automating as much as possible and getting high tokens per second throughputs. And it does require rethinking of all the abstractions and everything has to be reshuffled. So yeah, I think it’s very exciting. Sarah Guo If we take one more recursive step here, when is the model going to write a better program MD than you? Andrej Karpathy Yeah. So program MD is like… We’re not in the loop. Yeah, exactly. So program MD is my crappy attempt at describing how the auto researcher should work. Like, oh, do this, then do that, and that, and then try these kinds of ideas. And then here’s maybe some ideas like look at architecture, look at optimizer, etc. But I just came up with this in Markdown, right? And so, yeah, exactly. You want some kind of an auto-research loop maybe that looks for… You can imagine that different program.mds would give you different progress. So basically every research organization is described by ProgramMD. A research organization is a set of Markdown files that describe all the roles and how the whole thing connects. And you can imagine having a better research organization. So maybe they do fewer stand-ups in the morning because they’re useless. And this is all just code, right? And so one organization can have fewer stand-ups. One organization can have more. One organization can be very risk-taking. One organization can be less. And so you can definitely imagine that you have multiple research orgs. And then they all have code. And once you have code, then you can imagine tuning the code. So 100%, there’s like the meta layer of it. Sarah Guo Did you see my text about my contest idea? My contest idea was like, let people write different program MDs, right? And so for SameHardware, where do you get most improvement? Andrej Karpathy Oh, I see. Sarah Guo And then you can take all that data and then give it to the model and say, write a better program MD. Andrej Karpathy Yes, yes. Yeah, exactly. Sarah Guo We’re going to get something better. Like there’s no way we don’t, right? Andrej Karpathy You can 100% look at where the improvements came from and like, can I change the program MD such that more of these kinds of things would be done? Or like things that didn’t work. Sarah Guo Just meta optimization. Andrej Karpathy Yeah, you can 100% imagine doing that. So I think this is a great idea. But it’s like, you know, I think like you sort of go one step at a time where you sort of have one process and then second process and then the next process. And these are all layers of an onion. Like the LLM sort of part is now taken for granted. The agent part is now taken for granted. Now the claw-like entities are taken for granted. And now you can have multiple of them. And now you can have instructions to them. And now you can have optimization over the instructions. And it’s just like a little too much, you know? But I mean, this is why it gets to the psychosis is that this is like infinite and everything is skill issue. And that’s why I feel like, yeah, that’s just coming back to, this is why it’s so insane. (Time 0:19:42)
  • AI Progress Is Strong But Deeply Jagged
    • Karpathy says autonomous loops work best where outcomes are objectively measurable and verifiable, such as kernel optimization or training loss.
    • He argues model intelligence remains jagged: the same system can act like a brilliant programmer yet still tell stale jokes from years ago. Transcript: Sarah Guo Okay. Well, if we’re just trying to like diagnose the current moment uh what is a relevant skill right now what do you like what do you think is the implication that this um that this is the loop We should be trying to achieve in different areas and then it works right like you know remove create the metric or create the ability for um agents to continue working on it without you Yeah do we still have performance engineering? Andrej Karpathy Yeah. I mean, so there’s a few caveats that I would put on top of the LM psychosis. Number one, this is extremely well suited to anything that has objective metrics that are easy to evaluate. So for example, like writing kernels for more efficient CUDA code for various parts of a model, like such a chart, the perfect fit. Because you have inefficient code and then you want efficient code that has the exact same behavior, but it’s much faster. Perfect fit. So a lot of things that are perfect fit for auto-research, but many things will not be. And so they, it’s just, if you can’t evaluate it, then you can’t auto-research it, right? So that’s like caveat number one. And then maybe caveat number two, I would say is, you know, we’re kind of talking about the next steps and we kind of see what the next steps are, But fundamentally, the whole thing still Doesn’t, it’s still kind of like bursting at the seams a little bit and there’s cracks and it doesn’t fully work. And if you kind of try to go too far ahead, the whole thing is actually net not useful, if that makes sense. Because these models like still are not, you know, they’ve improved a lot, but they’re still like rough around the edges is maybe the way I would describe it. I simultaneously feel like I’m talking to an extremely brilliant PhD student who’s been like a systems programmer for their entire life and a 10-year And it’s so weird because humans, Like there’s, I feel like they’re a lot more coupled. Like you have, you know, everything is a lot more coupled. Yes, you wouldn’t encounter that combination. This jaggedness is really strange. And humans have a lot less of that kind of jaggedness, although they definitely have some. But humans have a lot more jaggedness. Sorry, the agents have a lot more jaggedness where sometimes like, you know, I ask for functionality and it like comes back with something that’s just like totally wrong. And then we get into loops that are totally wrong. And then I’m just, I get so frustrated with the agents all the time still. Because you feel the power of it, but you also, they’re still like it does nonsensical things once in a while for me still as well i get very annoyed when um uh i feel like the agent wasted A lot of compute on something it should have recognized was an obvious problem yeah i think like some of the bigger things is like maybe what’s under underneath it if i could hypothesize Is fundamentally these models are trained via reinforcement learning so they’re actually struggling with the exact same thing we just talked about which is the labs can improve the Models and anything that is verifiable but that has rewards so did you write the program correctly and does it do the unit test check out yes or no but some of the things where they’re struggling Is like for example i think they have a tough time with like nuance of maybe what i what in mind or what I intended and when to ask clarifying questions. Like, yeah, it’s just anything that feels softer is like worse. And so you’re kind of like you’re either on Rails and you’re part of the super intelligence circuits or you’re not on Rails and you’re outside of the verifiable domains. And suddenly everything kind of just like meanders. Maybe another way to put it is if you go to if today if you go to like state-of model chat gpt and you ask it tell me a joke um do you know what joke you’re gonna get there’s the joke the joke i Sarah Guo Do feel i i can’t tell you like the you know standard form of it but i do feel like chat gpt has like three jokes yeah yeah so the the joke that apparently all the elements like left the most Andrej Karpathy Is, why do scientists not trust atoms? Sarah Guo Okay. Andrej Karpathy Because they make everything up. Okay. They make everything up. Sarah Guo Why did that emerge? Andrej Karpathy So this is the joke you would get three or four years ago. And this is the joke you still get today. Okay. So even though the models have improved tremendously. Yeah. And if you give them an agentic task, they will just go for hours and move mountains for you. And then you ask for like a joke and it has a stupid joke, a crappy joke from five years ago. And it’s because it’s outside of the, it’s outside of the RL. It’s outside of the reinforcement learning. It’s outside of what’s being improved. It’s like, and it’s part of the jaggedness of like, shouldn’t you expect models as they get better to also have like better jokes or more diversity of them? Sarah Guo Or it’s just, it’s not being optimized and it’s stuck do you uh uh think that that implies that we are not seeing like generalization in the sense of like broader intelligence of joke Andrej Karpathy Smartness being attached to code smartness yeah i think there’s some decoupling where some things are verifiable and some things are not and some things are optimized for arbitrarily By the labs, depending on what data went in. And some things are not. Sarah Guo But I mean, the premise, there’s a premise from some research groups that if you are smarter at code generation or in these verifiable fields, you should be better at everything. And the joke situation suggests that that’s not happening in all fields. Andrej Karpathy I don’t think that’s happening. Yeah, I don’t think that’s happening. I think maybe we’re seeing like a little bit of that, but not like a satisfying amount. Sarah Guo Yeah, that jaggedness exists in humans. You can be very, very good at math and still tell a really bad joke. Andrej Karpathy Yeah, that’s true. Yeah, but it just, it still means that we’re not getting like, the story is that we’re getting a lot of the intelligence and capabilities and all the domains of society, like for free As we get better and better models. And it’s not like exactly fundamentally what’s going on. And there’s some blind spots and some things are not being optimized for. And this is all clustered up in these neural net opaque models, right? So you’re either on rails of what it was trained for and everything is like, you’re going at speed of light or you’re not. (Time 0:23:15)
  • Model Speciation Will Arrive Before Perfect Generality
    • Karpathy expects more model speciation rather than one universal oracle, with smaller systems specializing by task and latency needs.
    • He says the field still lacks mature ways to touch model weights safely without losing broad capabilities, so context remains the main control surface. Transcript: Sarah Guo I ask kind of a blasphemous question, which is like, if this jaggedness is persisting and it’s all rolled up in a at least monolithic interface, right? But, you know, single model. Does that make sense? Or should it be unbundled in things that can be optimized and improved against different domains of intelligence? Andrej Karpathy Like unbundling the models into multiple experts in different areas, etc. Sarah Guo More directly, yeah. Instead of just MOE that we have no exposure to. Because that can be confusing as a user from the outside, which is like, why is it so good at this, but not at this other thing? Andrej Karpathy Yeah, I think currently my impression is the labs are trying to have a single sort of like monoculture of a model that is arbitrarily intelligent in all these different domains and they Just stuff into the parameters i do think that we will we i do think we should expect more speciation in the intelligences um like you know the animal kingdom is extremely diverse in the Brains that exist and there’s lots of different niches of of nature and some animals have overdeveloped visual cortex or other kind of parts and i think we we should be able to see more Speciation and um you don’t need like this oracle that knows everything you kind of speciate it and then you put it on a specific task and we should be seeing some of that because you should Be able to have like much smaller models that still have the cognitive core like they’re still competent but then they specialize and then um and then they can become more efficient In terms of latency or throughput on specific tasks that you really care about. Like if you’re a mathematician working in Lean, I saw, for example, there’s a few releases that really target that as a domain. So there’s probably going to be a few examples like that where the unbundling kind of makes sense. Sarah Guo One question I have is whether or not the capacity constraint on available compute infrastructure drives more of this because efficiency actually matters more. Yeah. Right. Like you’re, if you financing aside, no financing that’s involved in all of this, if you have access to full compute for anything you do, like leaving one single model. Right. But if you actually feel pressure where you’re like, I can’t serve a model of massive size for every use case. Like, do you think that leads to any speciation? Does that question make sense to you? Andrej Karpathy The question makes sense. And I guess like what I’m struggling with is I don’t think we’ve seen too much speciation just yet. Right. No. We’re seeing a monoculture of models. Yeah. Sarah Guo And there, clearly pressure for, like, make a good code model, put it back in the main merge again. Yeah, yeah. Yeah. Um… Andrej Karpathy Even though there already is pressure on the models. Sarah Guo Mm-hmm. I guess perhaps I feel like there’s a lot of very short-term supply crunch. Uh-huh. And, like, maybe that causes more speciation now. Andrej Karpathy Yeah, I think fundamentally, like, the labs are serving a model, and they don’t really know what the end user is going to be asking about. So maybe that’s like some part of it because they kind of have to multitask over all the possible things that could be asked. But I think if you’re coming to a business and maybe partnering on some specific problems you care about, then maybe you would see that there. Or there will be some very high value applications that are like more niche. But I think right now they’re kind of like going after the totality of what’s available. I don’t think that the science of manipulating the brains is like fully developed yet, partly. Sarah Guo What do you mean manipulating? Andrej Karpathy So like, so fine tuning without losing capabilities, as an example. And we don’t have these primitives for actually like working with the intelligences in ways other than just context windows. Like context windows kind of just work and it’s very cheap to manipulate, et cetera. And this is how we’re getting some of the customization, et cetera. But I think if it was, I think it’s a bit more of a developing science of how you like more deeply adjust the models, how you have continual learning maybe, or how you fine tune in a certain Area, how you get better in a certain area, or like how you actually touch the weights, not just the context windows. And so it’s a lot more tricky, I would say, to touch the weights than just the context windows, because you’re actually fundamentally changing the full model and potentially its intelligence. And so maybe it’s just like not a fully developed science, if that makes sense, of speciation. Sarah Guo And it also has to be like cheap enough for that speciation to be worthwhile in these given contexts. Andrej Karpathy Can (Time 0:28:55)
  • AutoResearch Could Become A Swarm Compute Market
    • Karpathy imagines AutoResearch expanding into an internet-scale swarm where untrusted workers search for useful commits and trusted nodes verify them.
    • He compares it loosely to blockchains, Folding at Home, and SETI at Home because solutions are expensive to find but cheap to check. Transcript: Sarah Guo I ask a question about an extension to auto research that you described in terms of open ground? You say, okay, well, we have this thing. We need more collaboration surface around it, essentially, for people to contribute to research overall. Can you talk about that? Andrej Karpathy Yeah. So we talked about our research has a single thread of like, I’m going to try stuff in loop. But fundamentally, the paralyzation of this is like the interesting component. And I guess I was trying to like play around with a few ideas, but I don’t have anything that like clicks as simply as like, I don’t have something that I’m like super happy with just yet, But it’s something I’m like working on inside when I’m not working on my claw. So I think like one issue is if you have a bunch of nodes of paralyzation available to you, then it’s very easy to just have multiple auto researchers talking through a common system or Something like that. What I was more interested in is how you can have an untrusted pool of workers out there on the internet. So for example, in auto research, uh, you’re just trying to find, um, the piece of code that trains a model to a very low validation loss. If anyone gives you a candidate commit, it’s very easy to verify that that commit is correct, is good. Like someone could claim from the internet that this piece of code will optimize much better and give you a much better performance. You could just check. It’s very easy. But probably a lot of work goes into that checking. But fundamentally they could lie and et cetera. So you’re basically dealing with a similar kind of problem. It’s almost actually like looks a little bit like my designs that incorporate an untrusted pool of workers actually look a little bit more like a blockchain a little bit. Because instead of blocks, you have commits. And these commits can build on each other and they contain like changes to the code as you’re improving it. And the proof of work is basically doing tons of experimentation to find the commits that work. And that’s hard. And then the reward is just being on the leaderboard right now. There’s no monetary reward whatsoever. But I don’t want to push the analogy too far, but it fundamentally has this issue where a huge amount of search goes into it, but it’s very cheap to verify that a candidate solution is indeed Good because you can just train a single, you know, someone had to try 10,000 ideas, but you just have to check that the thing that they produced actually works because the 99,000 of them Didn’t work, you know? And so basically, long story short, it’s like you have to come up with a system where an untrusted pool of workers can collaborate with a trusted pool of workers that do the verification. And the whole thing is kind of like asynchronous and works and so on. And it’s like safe from a security perspective, because if anyone sends you arbitrary code and you’re going to run it, that’s very sketchy and dodgy. But fundamentally, it should be totally possible. So you’re familiar with projects like SETI at Home and Folding at Home. All of these problems have a similar kind of setup. So Folding at Home, you’re folding a protein, and it’s very hard to find a configuration that is low energy. But if someone finds a configuration that they evaluate to be low energy, that’s perfect. You can just use it. You can easily verify it. So a lot of things have this property that, you know, very expensive to come up with, but very cheap to verify. And so in all those cases, things like folding at home or SETI at home or auto research at home will be good fits. And so long story short, a swarm of agents on the internet could collaborate to improve LLMs and could potentially even like run circles around Frontier Labs. Like who knows, you know? Yeah, like maybe that’s even possible. Like Frontier Labs have a huge amount of trusted compute, but the earth is much bigger and has a huge amount of untrusted compute. But if you put systems in check, systems in place that, you know, deal with this, then maybe it is possible that the swarm out there could come up with better solutions. And people kind of like contribute cycles to a thing that they care about. And so, sorry, so the last thought is lots of companies or whatnot, they could maybe have like their own things that they care about. And you, if you have compute capacity, you could contribute to different kind of auto research tracks. Like maybe you care about certain, you know, like you care about like cancer or something like that of certain type. You don’t have to just donate money to an institution. You actually could like purchase compute and then you could join the auto research forum for that project, you know? (Time 0:32:58)
  • Software Jobs May Grow Before They Disappear
    • Karpathy expects AI to reshape digital-information jobs first because bits move faster than atoms and software demand may rise as creation gets cheaper.
    • He invokes Jevons paradox and notes frontier-lab researchers are effectively trying to automate their own work away. Transcript: Sarah Guo The last thing you released was like a little bit of jobs data analysis. Is that right? What, and might have touched on her, even though you’re just like visualizing some public data. What was, you know, what were you curious about? Andrej Karpathy Yeah, I guess I was curious too. I mean, everyone is like, everyone’s really thinking about the impacts of AI on the job market and what it’s going to look like. So I was just interested to take a look, like what does the job market look like? Where are the different roles? And how many people are in different professions? And I was like, really just interested to like look through the individual cases and try to think myself about like, you know, with these AIs and how they’re likely to evolve, like are These going to be tools that people are using? Are these going to be displacing tools for these professions? And like what are the current professions and how are they going to change? Are they going to grow or adjust to a large extent? Or like what could be new professions? So it’s really just like a way to fuel my own chain of thought about the industry, I suppose. And so, yeah, the jobs data basically is just a Bureau of Labor Statistics. They actually have a percent outlook for each profession about how much it’s expected to grow over the next, I think almost a decade. Yeah, I think it’s a decade, but it was made in 2024. Sarah Guo We need a lot of healthcare workers. Yeah. Andrej Karpathy So they’ve already made those projections. And I’m not sure actually 100% what the methodology was that they put into the projections. I guess I was interested to color things by, like, if people think that what’s primarily being developed now is this kind of more digital AI, that it’s kind of almost like these ghosts Or spirit entities that can like interact in the digital world and manipulate a lot of like digital information and they currently don’t really have a physical embodiment or presence And the physical stuff is probably going to go slightly slower because you’re manipulating atoms so flipping flipping bits and and the ability to copy paste digital information is Like makes everything a million times faster than accelerating matter, you know? So, so energetically, I just think we’re going to see a huge amount of activity in digital space, huge amount of rewriting, huge amount of activity, boiling soup. And I think the, we’re going to see something that in the digital space goes at the speed of light compared to, I think what’s going to happen in the physical world, to some extent, it would Be the extrapolationation and so i think like there’s currently kind of like i think overhang where there can be like a lot of unhobbling almost potentially of like a lot of digital information Processing that used to be done by computers and people and now with ai’s as like a third kind of manipulator of digital information there’s going to be a lot of refactoring in those in Those uh disciplines um but the physical world is actually going going to be like, I think, behind that by some amount of time. And so I think what’s really fascinating to me is like, so that’s why I was highlighting the professions that fundamentally manipulate digital information. This is work you could do from your home, et cetera, because I feel like those will be, like things will change. And it doesn’t mean that there’s going to be less of those jobs or more of those jobs, because that has to do with like demand elasticity and many other factors but things will change in These professions because of these new tools and because of this upgrade to the nervous system of the human superorganism if you want to think about it that way given the look you had Sarah Guo At the data do you have either any observations or guidance for people facing the job market or thinking about what to study now or what skills to develop. I mean, we can all go get like, I’m very thankful that I have to like meet people for my job right now. Yeah. More physical. Yeah. Andrej Karpathy Could you do your work from home though? I could. Sarah Guo I think there are relationship parts of it that are hard, but most of it I could. Andrej Karpathy Yeah. I think it’s really hard to tell because again, like the job market is extremely diverse. I think the answers will probably vary, but to a large extent, like these tools are extremely new, extremely powerful. And so just being, you know, just trying to keep up with it is like the first thing. And yeah, because I think a lot of people kind of like dismiss it or… Sarah Guo Or they’re afraid of it. Andrej Karpathy Or they’re afraid of it, etc. Which is totally understandable, of course. Yeah, I think like it’s fundamentally an empowering tool at the moment. And these jobs are bundles of tasks, and some of these tasks can go a lot faster. And so people should think of it as primarily a tool that it is right now. And I think the long-term future of that is uncertain. Yeah, it’s kind of really hard to forecast, to be honest. And I’m not professionally doing that, really. And I think there’s a job of economists to do it properly. Sarah Guo You are an engineer, though. And one thing I thought was interesting is that the demand for engineering jobs is continuing to increase. I can’t tell if that’s a temporary phenomenon. I’m not sure how I feel about it yet. Do you know? Andrej Karpathy Yeah, that’s like the demand elasticity, almost. Software was scarce, right? And so the reason we don’t have more demand for software is just it’s scarcity and it’s too expensive. It’s too expensive, yeah. So if the barrier comes down, then actually you have the Jevons paradox, which is like, you know, you actually, the demand for software actually goes up. It’s cheaper and there’s more. More powerful, yeah. The classical example of this always is the ATMs and the bank tellers, because there was a lot of like fear that ATMs and computers basically would displace tellers. But what happened is they made like the cost of operation of a bank branch much cheaper. As there were more bank branches, so there were more tellers is like the canonical example people cite. But basically it’s just Jevons paradox, like something becomes cheaper. So there’s a lot of unlocked demand for it. So I do think that that’s probably, I do have like cautiously optimistic view of this in software engineering, where I do, it does seem to me like the demand for software will be extremely Large. And it’s just become a lot cheaper. And so I do think that for quite some time, it’s very hard to forecast, but it does seem to me like right now, at least locally, there’s gonna be more demand for software. Because software is amazing. It’s like, you know, digital information processing. You’re not forced to use like arbitrary tools that were given to you. There are imperfect in various ways. You’re not forced to subscribe to what exists. Code is now ephemeral and it can change and it can be modified. And so I think there’s going to be a lot of activity in the digital space to like rewire everything in a certain sense. And I think it’s going to create a lot of demand for this kind of stuff. I think long term, yeah, obviously, even with other research, like OpenAI or Anthropic or these other labs, they’re employing, what, like a thousand something researchers, right? These researchers are basically glorified. They’re automating themselves away actively. And this is the thing they’re all trying to do yeah i think like i went around um some of those researchers also fear that feel the psychosis right because they can it’s working yeah right And so they’re like oh it’s over for me too i did spend a bunch of time going around opening i and i was like you guys realize if we’re successful like we’re all out of jobs like like i… (Time 0:37:53)
  • Frontier Lab Work Trades Access For Independence
    • Karpathy argues people can have major impact both inside and outside frontier labs, but inside roles carry financial and social alignment pressures.
    • He worries outsiders’ judgment drifts without frontier visibility, so moving in and out of labs may balance independence with access. Transcript: Sarah Guo It okay if I ask you Noam’s question? You know, you could be doing that, right? Auto-researching with a lot of compute scale and a bunch of colleagues at one of the frontier labs. Like, why not? Andrej Karpathy Well, I was there for a while, right? And I did re-enter. So to some extent, I agree. And I think that there are many ways to slice this question. It’s a very loaded question a little bit. I will say that I feel very good about like what people can contribute and their impact outside of the frontier labs, obviously not in the industry but also in like more like ecosystem Level roles um so your role for example is more like ecosystem level my role currently is also kind of more on ecosystem level and i feel very good about like impact that people can have In those kinds of uh roles i think conversely there’s there are definite problems in my mind for um for basically aligning yourself way too much with the Frontier Labs too. So fundamentally, I mean, you have a huge financial incentive with these Frontier Labs. And by your own admission, the AIs are going to really change humanity and society in very dramatic ways. And here you are basically building the technology and benefiting from it and being very allied to it through financial means. This was a conundrum that was at the heart of how opening I was started in the beginning. Like this was the conundrum that we were trying to solve. And so, you know, so it’s kind of- It’s still not resolved. The conundrum is still not like fully resolved. So that’s number one. You’re not a completely free agent and you can’t actually like be part of that conversation in a fully autonomous, free way. Like if you’re inside one of the frontier labs, like there are some things that you can’t say. And conversely, there are certain things that the organization wants you to say. And, you know, they’re not going to twist your arm, but you feel the pressure of like what you should be saying, you know, because like, obviously, otherwise it’s like really awkward Conversations, strange side eyes, like, what are you doing? You know, so you can’t like really be an independent agent. And I feel like a bit more aligned with humanity in a certain sense outside of the frontier lab, because I don’t I’m not subject to those pressures almost. Right. And I can’t say whatever I want. Yeah, I would say in the frontier labs, like you can have like impact there, of course, as well. So but there’s many researchers and maybe you’re one of them. Maybe your ideas are really good, et cetera. And maybe there’s a lot of decision-making to do and you want to be in a position where you are in the room with those conversations when they come up. I do think that currently the stakes are like overall fairly low. And so everything is kind of like nice, but ultimately at the end of the day, like when the stakes are really high, et cetera, if you’re an employee at an organization, I don’t actually Know how much sway you’re going to have on your organization, what it’s going to do. Like fundamentally at the end of the day, it not like really in charge. You’re like, you’re in a room and you’re contributing ideas, but you’re not really in charge of that entity that you’re a part of. So those are like some sources of misalignment, I think, to some extent. I will say that like in one way, I do agree a lot with that sentiment that I do feel like the labs for better or worse, they’re opaque and a lot of work is there. And they’re kind of like at the edge of capability in what’s possible. And they’re working on what’s coming down the line. And I think if you’re outside of that frontier lab, your judgment fundamentally will start to drift because you’re not part of what’s coming down the line. And so I feel like my judgment will inevitably start to drift as well. And I won’t actually have an understanding of how these systems actually work under the hood. That’s an opaque system. I won’t have a good understanding of how it’s going to develop and etc and so i do think that in that sense i agree and something i’m nervous about i think it’s worth basically uh being in Touch with what’s actually happening and actually being in the frontier lab and if if some of the frontier labs would have me come for you know some amount of time and do really good work For them, and then maybe come in and out. Sarah Guo Guys who’s looking for a job, this is super exciting. Andrej Karpathy Then I think that’s maybe a good setup, because I kind of feel like it kind of, you know, maybe that’s like one way to actually be connected to what’s actually happening, but also not feel Like you’re necessarily fully controlled by those entities. So I think, honestly, in my mind, like, Noam can probably do extremely good work at OAI, but also I think his most impactful work could very well be outside of OpenAI. Sarah Guo Noam, that’s a call to be an independent researcher with auto-research. Andrej Karpathy Yeah, there’s many things to do on the outside. And I think, ultimately, I think the ideal solution maybe is like, yeah, going back and forth. Or, yeah, and I think fundamentally, you can have really amazing impact in both places. (Time 0:44:35)
  • A Lagging Open Source Tier Is Healthy
    • Karpathy expects the pattern of closed frontier models with open models trailing by months to continue and considers that healthy.
    • He argues society needs an open common platform, because centralized control of powerful intelligence has a poor historical track record. Transcript: Sarah Guo One question related to what visibility does the world or the AI ecosystem have into the frontier is like how close open source is to the frontier and how sustainable that is. I think it is quite surprising the entire sequence of events actually from like having a handful of Chinese models and global models. And I think people are going to continue releasing here in the near term that are closer than much of the industry anticipated from a capability perspective. I don’t know if you’re surprised by that, but you’re a long-term contributor to open source. Like, what’s your prediction here? Andrej Karpathy Yeah. So roughly speaking, basically the, yeah, the closed models are ahead, but like people are monitoring the number of months that sort of like open source models are behind. Sarah Guo And it started with there’s nothing and then it went to 18 months. Andrej Karpathy Yeah, and there’s been a convergence, right? So maybe they’re behind by like, what is the latest? Maybe like six months, eight months kind of thing right now. Yeah, I’m a huge fan of open source, obviously. So for example, in operating systems, you have like Windows and Mac OS. These are large software projects, kind of like what LLMs are going to become. And there’s Linux, but Linux is very easy. Like actually Linux is an extremely successful project. It runs on the vast majority of computers. Like last time I checked, was it like 60% or something? Like run Linux. And that’s because there is a need in the industry to have a common open platform that everyone feels sort of safe using. I would say like the industry has always felt a demand for that kind of a project to exist. And I think the same is true now. And that’s why businesses actually want, there’s demand for this kind of a thing to exist. The big difference is that everything is capital. There’s a lot of capex that goes into this. So I think that’s where things like fall apart a little bit, make it a bit harder to compete in this sense. I do think that the current models are very good. The other thing that I think is like really interesting is that for the vast majority of like consumer use cases and things like that, even the term open source models are actually quite Good, I would say. And I think like if you go forward like more, more years, it does seem to me like a huge amount of like simple use cases are going to be well covered and actually even run locally. But there’s going to be always like some demand for like frontier intelligence and that that can actually be extremely large piece of the pie. But could be that the frontier the need for frontier intelligence is going to be like you know nobel prize kind of work or like let’s move linux from c to rust there’s going to be like bigger Projects you know like scoped in that kind of a way and there’s going to be maybe more um and maybe that’s where a lot of the frontier closed intelligences are going to be interacting with And open source is kind of like going to eat through a lot of the more basic use cases or something like that. You know, at some point, what is frontier today is going to be, you know, probably later this year. What’s frontier today in terms of what I’m using right now from the closed labs might be open source. And that’s going to be doing a lot of work. So I kind of expect that this dynamic will actually basically continue. Like we’ll have frontier labs that have closed AIs that are kind of like these oracles, and then we’ll have open source kind of like behind by some amount of months. And I kind of expect that to continue. And I actually think that’s like a pretty good setup overall. Because I’m a little bit hesitant of having, I don’t actually think it’s like structurally, I think there’s some systemic risk attached to just having intelligences that are closed. And that’s like, that’s it. And I think that that’s, you know, centralization has a very poor track record in my view in the past. And has… Sarah Guo You mean like in political or economic systems in general? Yes. Andrej Karpathy Exactly. I think there’s like a lot of pretty bad person. Spoken like an Eastern European. A lot of pretty bad person. And so I want there to be a thing that is maybe not at the edge of capability because it’s new and unexplored, etc. But I want there to be a thing that’s behind and that is kind of like a common working space for intelligences that the entire industry has access to. Yeah, that seems to me like a pretty decent power balance for the industry. Sarah Guo Advancing intelligence from the frontier, we can do new things. And there are a lot of like very big problems for humanity. Right. And so like, it seems that that will continue to be a very expensive game. And so I want to like root for labs that are doing that because there are problems we cannot solve without continuing to advance the models in a very expensive way. And yet, as you point out, like if what we have today as frontier is that’s a lot of capability. And so I think the power of that or the democratization of that seems very useful and also healthy. Andrej Karpathy Yeah, I think basically by accident we’re actually in an okay spot. An optimal one, yeah. By accident we happen to be in a good spot in a certain sense. Sarah Guo Well, and to some degree, the longer this endures, like this dynamic, the healthier of a spot, like the ecosystem might be in. Yeah. Right. Because you have more and more area under the curve. Andrej Karpathy And I will say that even on the closed side, I almost feel like it’s been like even further centralizing recently because I think a lot of the front runners are like not necessarily like The top tier. And so, yeah, like in that sense, I think it’s it’s not super ideal. I would love there to be more frontier labs. Because I’m by default very suspicious of like, I want there to be more people in the room. I think in machine learning, ensembles always are performing an individual model. And so I want there to be ensembles of people thinking about all the hardest problems. And I want there to be ensembles of people in a room to they, to be all well-informed and to make all those decisions. You know, so I don’t want it to be like a closed doors with two people or three people. I feel like that’s like not a good future. I almost wish like there were more labs, long story short, and I do think that open source has a place to play. I hope it sticks around. (Time 0:48:54)
  • The Next Wave Is At The Digital Physical Interface
    • Karpathy thinks digital work will be transformed first, then interfaces that connect intelligence to the physical world, and finally robotics itself.
    • He frames sensors and actuators as the next bottleneck, from lab instruments in materials science to paid human data collection. Transcript: Sarah Guo Okay, you worked on the precursor to generalized robotics autonomy in cars, right? A lot has happened in the last couple of months with robotics companies as well. Like acceleration of really impressive generalization of environment, of tasks, like increasing long horizon tasks, lots of money going into the space. Like, is it going to happen? Has anything in your view changed recently? Andrej Karpathy So like my view is kind of informed by what I saw in self-driving. And I do feel like self-driving is the first robotics application. So probably what I saw is at the time, like 10 years ago, there were a large number of startups. And I kind of feel like most of them basically didn’t long term make it. And what I saw is that like a lot of capital expenditure had to go in and a lot of time. And so I think it’s like, I think robotics, because it’s so difficult and so messy and requires a huge amount of capital investment and a lot of like conviction, just it’s like a big problem. And I think items are really hard. So I kind of feel like it will lag behind what’s going to happen in digital space. And in digital space, there’s going to be a huge amount of unhobbling, basically like things that weren’t super efficient, becoming a lot more efficient by like a factor of a hundred, Because bits are so much easier. And so I think currently in terms of what’s going to change and like where the activity is, I kind of feel like digital space is going to like change a huge amount. And then the physical space will lag behind. And what I find very interesting is like interface in between them as well. Because I think in this, if we do have more agents acting on behalf of humans and more agents talking to each other and doing tasks and participating in the economy of agents, et cetera, You’re going to run out of things that you’re going to do purely in a digital space. At some point, you have to go to the universe and you have to ask it questions. You have to run an experiment and see what the universe tells you to get back to learn something. And so we currently have a huge amount of digital work because there’s an overhang in how much we collectively thought about what already is digital. So we just didn’t have enough thinking cycles among the humans to think about all the information that’s already digital and already uploaded. And so we’re going to start running out of stuff that is actually already uploaded. So you’re going to at some point read all the papers and process them and have some ideas about what to try but um yeah we’re just gonna i don’t actually know how much you can like get intelligence That’s like fully closed off and was just information that’s available to it you know and so i think what’s going to happen is first there’s going to be a huge amount of unhobbling and I think there’s a huge amount of work there then actually it’s going to move to like the interfaces between physical and digital so and that’s like sensors of like seeing the world and Actuators of like doing something to the world so i think a lot of interesting companies will actually come from that interface of like can we feed the super intelligence in a certain Sense uh data and can we actually like take data out and manipulate the physical world per its bidding if you want to like interpromorphize the whole thing right and then the physical World actually i almost feel like the the total addressable market etc in terms of like the amount of work and so on is is massive possibly even much larger maybe what can happen in digital Space so i actually think it’s like a much bigger opportunity as well but um i do feel like it’s a huge amount of work and and in my in my mind the atoms are just like a million times harder So um so it will lag behind but it’s also i think a little bit of bigger market so it’s kind of like uh yeah i think the opportunities kind of like follow that kind of trajectory so right now This digital is like my main interest then interfaces would be like after that and then maybe some of the physical things. Like their time will come and they’ll be huge when they do come. Sarah Guo Well, it’s an interesting framework for it too, because certain things, not the things I’m working on right now, but certain things are much easier even in the world of atoms, right? Like if you just think about read and write to the physical world, like read, sensors, cameras, there’s a lot of existing hardware. And you imagine like enriching agent capabilities or capturing a lot of new data if you’re just clever about it. And like you don’t necessarily have to invest a lot to like get something valuable. Andrej Karpathy Yeah. So like examples of this that I saw, for example, are, you know, a friend of mine, Liam, is a CEO of Periodic. I visited them last week. So it’s just on top of mind. Like they’re trying to do auto research for material science. And so in that case, it’s like the sensors to the intelligence are actually like pretty expensive lab equipment. And the same is true in biology. I think a lot of people are very interested in engineering biology. And, you know, the sensors will be more than just like video cameras, if that makes sense. And then the other thing I saw, for example, is companies that are trying to have, like you basically pay people for training data, as an example. Sarah Guo Yeah, programmatically. Andrej Karpathy Yeah, to feed the Borg. And so these are all examples of sensors in a certain sense. So they take many diverse shapes and forms, if that makes sense. Sarah Guo Yeah, so I’m looking forward to the point where I can ask for a task in the physical world and I can put a price on it. I just tell the agent, like, you know, you figure out how to do it. Go get the data. Andrej Karpathy I’m actually kind of surprised we don’t have enough like information markets. Like if, for example, if Polymarket or other betting markets or even stocks, et cetera, if they have so much autonomous activity and rising amount of activity, like why should, like, For example, if Iran was just happening now, like how come there isn’t a process where like taking a photo or video from somewhere in Tehran should cost like 10 bucks? Someone should be able to pay for that you know like and that’s an example of like feeding the intelligence there’s not going to be a human looking at it it’s going to be like agents who Are trying to guess the betting games and stock markets and so on so i kind of feel like the agentic web is still like fairly new that there’s no like mechanisms for this but this is an example Of what i think might happen there’s a good book that maybe is inspiring called Daemon. You potentially read it. In Daemon, the intelligence ends up like puppeteering almost a little bit like humanity in a certain sense, you know? And so humans are kind of like its actuators, but humans are also like its censors. (Time 0:54:18)
  • Education Shifts From Teaching People To Teaching Agents
    • Karpathy says education is shifting from humans writing explanations for humans to creators writing for agents that teach humans.
    • MicroGPT compresses LLM training into about 200 Python lines, and he now sees his value in producing the irreducible ideas, not the tutorials. Transcript: Sarah Guo Want to talk about a little tiny side project you have before we end. Tell me about the microGPT. Oh, yeah. Andrej Karpathy Okay, so microGPT. So I have this like running obsession of like maybe a decade or two of just like simplifying and boiling down the basically LLMs to like their bare essence. And I’ve had a number of projects along these lines. So like NanoGPT and MakeMore and MicroGPT, MicroGrad, et cetera. So I feel like micro-GPT is now the state of the art of me trying to like just boil it down to just the essence. Because the thing is like training neural nets and LLMs specifically is a huge amount of code, but all of that code is actually complexity from efficiency. It’s just because you need it to go fast. If you don’t need it to go fast and you just care about the algorithm, then that algorithm actually is 200 lines of Python. Very simple to read. And this includes comments everything. Because you just have like your data set, which is a text, and you need your neural network architecture, which is like 50 lines. You need to do your forward pass. And then you have to do your backward pass to calculate the gradients. And so an old Autograd engine to calculate the gradients is like 100 lines. And then you need an optimizer, an atom, for example, which is a very state-of optimizer. It’s like, again, 10 lines, really. And so putting everything together in the training loop is like, yeah, 200 lines. And it was interesting to me, like normally before, like maybe a year ago or more, if I had come up with micro GPT, I would be tempted to basically explain to people. Like I have a video, like stepping through it or something like that. And I actually tried to make that video a little bit. And I tried to make like a little guide to it and so on i kind of realized that this is is not really it’s not really adding too much because people because it’s already so simple that it’s 200 lines that anyone could ask their agent to explain it in various ways and the agents like i’m not explaining to people anymore i’m explaining it to agents if you can explain it to agents Then agents can be the router and they can actually target it to the human in their language with infinite, you know, patience and just at their capability and so on. Sarah Guo Right. If I don’t understand this particular function, I can ask the agent to explain it to me like three different ways. Andrej Karpathy And I’m not going to get that from you. Exactly. And so I kind of feel like, you know, what is education? Like it used to be guides, it used to be lectures, it used to be this thing, but I feel like now more I’m explaining things to agents and maybe I’m coming up with skills where like, so basically Skill is just a way to instruct the agent how to teach the thing. So maybe I could have a skill for micro GPT of the progression I imagine the agent should take you through if you’re interested in understanding the code base. And it’s just like hints to the model to like, oh, first start off with this and then with that and so i could just script the curriculum a little bit as a skill uh so um so i don’t feel like Um yeah i feel like there’s gonna be less of like explaining things directly to people and it’s gonna be more of just like does the agent get it and if the agent gets it they’ll do the explanation And we’re not fully there yet because they i still can i still think i can probably explain things a little bit better than the agents, but I still feel like the models are improving so Rapidly that I feel like it’s a losing battle to some extent. And so I think education is going to be kind of like reshuffled by this quite substantially, where it’s the end of like teaching each other things almost a little bit. Like if I have a library, for example, of code or something like that, it used to be that you have documentation for other people who are going to use your library. But like, you shouldn’t do that anymore. Like you should have, instead of HTML documents for humans, you have Markdown documents for agents. Because if agents get it, then they can just explain all the different parts of it. So it’s this redirection through agents, you know? Sarah Guo And that’s like, so I think we’re going to see a lot more of that playing out well we’ll see if the great teachers know like to develop intuition for how to explain things to agents differently Andrej Karpathy Ultimately so for example micro gpt like i asked i tried to get an agent to write micro gpt so i told it like try to boil down the simplest things like try to boil down my um neural network Stream to the simplest thing and can’t do it. Like micro GPT is like my, it’s like my end of my obsession. It’s the 200 lies. I thought about this for a long time. I’ve obsessed about this for a long time. This is the solution. Trust me, it can’t get simpler. And this is my value add. Everything else, like agent gets it. It just can’t come up with it, but it totally gets it and understands why it’s done in a certain way, et cetera. So like my contribution is kind of like these few bits, but everything else in terms of like the education that goes on after that is like not my domain anymore. So maybe, yeah, it’s like education kind of changes in those ways where you kind of have to infuse the few bits that you feel strongly about the curriculum or the better way of explaining It or something like that. The things that agents can’t do is your job now. The things that agents can do, they can probably do better than you or like very soon. And so you should be strategic about what you’re actually spending time on. (Time 1:01:30)