Skip to content

Podcast

#490 – State of AI in 2026- LLMs, Coding, Scaling Laws, China, Agents, GPUs, AGI

Lex Fridman Podcast

Source ↗ ← All highlights
  • Pick Coding Tools By Control Level
    • Use IDE-integrated assistants (e.g., Codex plugin) when you want control and traceable diffs.
    • Use agentic tools (e.g., Cloud Code) when you prefer macro-level guidance and project-level automation. Transcript: Sebastian Raschka So I use the Codex plugin for VS Code. You know, it’s very convenient. It’s just like a plugin and then it’s a chat interface that has access to your repository i know that cloud code is i think a bit different it’s a bit more agentic it touches more things It does a whole project for you i’m not quite there yet where i’m comfortable with that because maybe i’m a control freak but i still would like to see a bit what’s going on. And Codex is kind of like right now for me, like the sweet spot where it is helping me, but it is not taking completely over. Lex Fridman I should mention one of the reasons I do use Cloud Code is to build the skill of programming with English. I mean, the experience is fundamentally different. As opposed to micromanaging the details of the process of the generation of the code and looking at the diff, which you can in cursor, if that’s the ID you use, and changing, altering, Looking and reading the code and understanding the code deeply as you progress versus just kind of like thinking in this design space and just guiding it at this macro level, which I Think is another way of thinking about the programming process. Also, we should say that Cloud Code, it just seems to be somehow a better utilization of Cloud Opus 4.5. Nathan Lambert It’s a good side-by for people to do. So you can have Cloud Code open, you can have Cursor open, you can have VS code open, and you can select the same models on all of them and ask questions. It’s very interesting. Like the cloud code is way better in that domain. It’s remarkable. All (Time 0:36:32)
  • Use LLMs To Frame, Not Distract
    • Use LLMs as a focused research assistant, but avoid rabbit holes; prompt them for structured context before diving into the wider internet.
    • Treat LLMs as a readable home rather than another distracting browser tab. Transcript: Lex Fridman My way of recommendation is to say I do the opposite. I like to use the LLM at the beginning to lay out the full context of like, what is this world that I’m now stepping into? But I try to avoid clicking out of the LLM into the world of like Twitter and blogs. And because then you’re now down this rabbit hole, you’re reading somebody’s opinion. There’s a flame war about a particular topic. And all of a sudden, you’re no longer, you’re now in the realm of the internet and Reddit and so on. But if you’re purely letting the LLM give you the context of why this matters, what are the big picture ideas? But sometimes books themselves are good at doing that, but not always. Nathan Lambert So that’s why I like the ChatGPT app. It gives the AI a home in your computer when you can focus on it rather than just being another tab in my mess of internet options. And I think Cloud Code in particular does a good job of making that a joy, where it seems very engaging as a product designed to be an interface that your AI will then go out into the world. (Time 0:41:02)
  • Open Models Catalyzed Global Research
    • Open-weight models exploded in 2025 with many Chinese and Western entries, driving research and tooling.
    • Nathan Lambert notes MOE and large-parameter open models are becoming common and shifting the landscape. Transcript: Nathan Lambert You want to see how many we can name off the the top of our head? Lex Fridman Yeah, without looking at notes. Nathan Lambert DeepSeq, Kimmy, Minimax, Z.ai, Antling. We’re just going Chinese. Sebastian Raschka Let’s throw in Mistral AI, Gemma. Yeah, GPT OSS, the open source model by jet gpt actually nvidia nemotron had a nvidia had a very cool one a nemotron 3 um there’s a lot of stuff especially at the end of the year quen one maybe The one oh yeah quen was the name the obvious name i was trying to get through that you can get at least 10 chinese and at least 10 western i think that i mean opening i released their first Nathan Lambert Open model since gpt2 that was when i when i meant to talk when i was writing about opening eyes open model release they’re all like don’t forget about gpt2 which i thought was really funny Because it’s just such a different time but gpt oss is actually a very strong model and does some things that the other models don’t do very well and i think that selfishly i’ll promote A bunch of like western companies so both in the u.s and europe have these like fully open models so i work at allen institute for ai where we’ve been building ulmo which releases data And code and all of this and now we have actual competition for people that are trying to release everything so that other people can train these models so there’s the institute for foundation Models or slash l360, which is like had their K2 models of various types. Aperdis is a Swiss research consortium. Hugging Face has small LLM, which is very popular. And NVIDIA’s Nemetron has started releasing data as well. And then Stanford’s Marin Community Project, which is kind of making it so there’s a pipeline for people to openithub issue and implement a new idea and then have it run in a stable language Modeling stack so this space that list was way smaller in 2024 so i think it was like just ai2 so that’s a great thing for more people to get involved and to understand language models which Doesn’t really have a like a chinese company that is has an analog while i’m talking i’ll say that the chinese open language models tend to be much bigger and that gives them this higher Peak performance as moe is where a lot of these things that we like a lot whether it was gemma and nematron have tended to be smaller models from the u.s which is which is starting to change From u.s and europe mr large three came, which was a giant MOE model, very similar to DeepSeek architecture in December. And then a startup, RCAI, and both Nemetron and NVIDIA have teased MOE models of this way bigger than 100 billion parameters, like this 400 billion parameter range coming in this Q1 2026 timeline. (Time 0:43:17)
  • Transformers Evolved, Not Replaced
    • Small architectural tweaks (attention variants, normalization, activations) multiply into big empirical gains.
    • Sebastian Raschka emphasizes lineage from GPT-2: same core but many practical, effective refinements. Transcript: Sebastian Raschka You can go chronologically i mean there was of course deep seek um deep seek r1 that came out in january if we just focus on 2025 however this was based on deep seek version 3 which came out The year before in december 2024 there are multiple things on the architecture side what is fascinating is you can still i mean that’s what i do in my from scratch coding projects you Can still start with gpt2 and you get can add things to that model to make it into this other model so it’s all still kind of like the same lineage the same it is a very close relationship Between those but top of my head deep seek what was uh unique there is the mixture of x i know i mean they were not inventing mixture of experts we can maybe talk a bit more what mixture of Experts means. But just to list these things first before we dive into detail, mixture of experts, but then they also had multi-head latent attention, which is a tweak to the attention mechanism, Where this was, I would say, 2025, the main distinguishing factor between these open weight models, different tweaks to inference or KV cache size. We can also define KV cache in a few moments, but to kind of make it more economical to have long context, to shrink the KV cache size. So what are tweaks that we can do? And most of them focused on the attention mechanism. There is multi-head latent attention in DeepSeq. There is group query attention, which is still very popular. It’s not invented by any of those models. It goes back a few years, but that would be the other option. Sliding window attention, I think almost reuses it, if I remember correctly. So there are these different tweaks that make the models different. Otherwise, I put them all together in an article once where I just compared them. They are very surprisingly similar it’s just different numbers in terms of how many repetitions of the transformer block you have in the center and like just little little knobs that People tune but what’s so nice about it is it’s it works no matter what you can tweak things you can move the normalization layers around. You get some performance gains. And I almost always very good in ablation studies showing what actually what it does to the model if you move something around. Ablation studies doesn’t make it better or worse. But there are so many, let’s say, ways you can implement a transformer and make it still work. Big ideas that are still prevalent is mixture of experts, multi-hatent attention, sliding window attention, group query attention. And then at the end of the year, we saw a focus on making the attention mechanism scale linearly with inference token prediction. (Time 0:51:54)
  • Scaling Laws Still Provide Value
    • Multiple scaling axes exist: pre-training, RL/post-training, and inference-time compute — all still produce gains.
    • Nathan Lambert is bullish: scaling laws broadly hold though trade-offs and costs matter. Transcript: Lex Fridman Are the scaling laws holding strong across pre-training, post-training, inference, context size, data, synthetic data? Nathan Lambert I like to start with the technical definition of scaling law which kind of informs all this the scaling law is a power law relationship between you can think of the x-axis so kind of what You are scaling is a combination of compute and data which are kind of similar and then the y-axis is like the held out prediction accuracy over an x token so we talk about models being Auto-regressive it’s like if you keep a set of text that the model has not seen how accurate will it get when you don’t train and the idea of scaling laws came when people figured out that That was a very predictable relationship and i think that that technical term is continuing and then the question is like what do users get out of it and then there are more types of scaling Where um openai’s 01 was famous for introducing inference time scaling and i think less famously for also showing that you can scale reinforcement learning training and get kind of This log x-axis and then a linear increase in performance on y-axis so there’s kind of these three axes now where the traditional scaling laws are talked about for pre-training which Is how big your model is and how big your data set is and then scaling reinforcement learning which is like how long can you do this trial and error learning that we will talk about we’ll Define more of this and then this inference time compute which is just letting the model generate more tokens on a specific problem so i’m kind of bullish where they’re all really still Working but the low-hanging fruit has mostly been taken especially in the last year on um reinforced learning with verifiable rewards which is this rl vr and then inference time scaling Which is just why these models feel so different to use where previously you would get that first token immediately and now they’ll go off for seconds minutes or even hours generating Hidden thoughts before giving you the first word of your answer and that’s all about this inference time scaling which is such a wonderful kind of step function in terms of how the models Change abilities they kind of enabled this tool use stuff and enabled this much better software engineering that we were talking about and this is when we say enabled almost entirely Downstream of the fact that this reinforced learning with verifiable rewards training just kind of let the models pick up these skills very easily so let the models learn so if you look At the reasoning process when the models are generating a lot of tokens what it will be often doing is it tries a tool it looks at what it gets back it tries another api it sees what it gets Back and if it solves the problem. So the models, when you’re training them, very quickly learn to do this. And then at the end of the day, that gives this kind of general foundation where the model can use CLI commands very nicely in your repo and handle Git for you and move things around and Organize things or search to find more information. Which if we’re sitting in these chairs a year ago it’s something that we didn’t really think of the models being doing so this is just kind of something that has happened this year and Is totally transformed how we think of using ai which i think is very magical it’s such an interesting evolution and just so unlock so much value but it’s like it’s not clear what the next Avenue will be in terms of unlocking stuff like this i think that there’s there’s we’ll get to continual learning later but there’s a lot of buzz around certain areas of ai but no one knows Lex Fridman When the next step function will really come so you you’ve actually said quite a lot of things there and said profound things quickly it would be nice to unpack them a little bit you said You’re bullish basically on every version of scaling. So we just even start at the beginning. Pre-training, are we kind of implying that the low-hanging fruit on pre-training scaling has been picked? Is pre-training hit a plateau or is even pre-training still you’re bullish on? Nathan Lambert Pre-training has gotten extremely expensive. I think to scale up pre-training, it’s also implying that you’re going to serve a very large model to the users. So I think that it’s been loosely established the likes of GPT-4 and similar models were around one trillion, like this order of trillion parameters at the biggest size. There’s a lot of rumors that they’ve actually gotten smaller as training has gotten more efficient. You want to make the model smaller because then your costs of serving go down proportionally. These models, the cost of training them is really low relative to the cost of serving them to hundreds of millions of users. I think DeepSeek had this famous number of about $5 million for pre-training at cloud market rates. I think Olmo 3, Section 2.4 in the paper we just detailed how long we had the gpu clusters sitting around for training which includes engineering issues multiple seeds and it was like About two million dollars to rent the cluster to like deal with all the problems and headaches of training a model so these models are pretty like a lot of people could get one to ten million Dollars to train a model but the recurring costs of serving millions of users is really billions of dollars of compute i think that you can look at close like a thousand gpu rental you Can pay a hundred grand a day for and these companies could have millions of gpus like you can look at how much these things cost to sit around so that’s kind of a big thing. And then it’s like, if scaling is actually giving you a better model, is it going to be financially worth it? And I think it’ll kind of slowly will push it out as AI solves more compelling tasks. So like the likes of Cloud Opus 4.5, making cloud code just work for things. I think I launched this project called like the Atom Project, which is like American Truly Open Models in July. And that was like a true vibe coded website. And like I have a job, make plots and stuff. And then I came back to refresh it in the last few weeks. And it’s like Claude Opus 4.5 versus whatever model at the time was like just crushed all the issues that it had from building in June, July. And like it might be a bigger model. There’s a lot of things that go into this, but that’s like, there’s still progress coming. Lex Fridman So what you’re speaking to is the nuance of the y-axis of the scaling laws, that the way it’s experienced versus on a benchmark, the actual intelligence might be different. But still, your intuition about pre-training, if you scale the size of compute, will the models get better? Not whether it’s financially viable, but just from the law aspect of it. Do you think the models will get smarter? (Time 1:02:45)
  • Follow The Three-Stage Training Recipe
    • Structure training into pre‑training (knowledge), mid‑training (specialized data), then post‑training (RLHF/RLVR).
    • Use mid‑training to inject reasoning traces and post‑training to unlock skills via verifiable rewards. Transcript: Sebastian Raschka So pre-training is the classic training of one next token prediction at a time. Have a big corpus of data and nathan also has very interesting insights there because of almost three it’s a big portion of the paper focuses on the right data mix so pre-training is essentially Just you know train across entropy loss training on next token prediction on a vast corpus of internet data books papers and so forth it has changed a little bit over the years in the sense People used to throw in everything they can. Now, it’s not just raw data, it’s also synthetic data where people, let’s say, rephrase certain things. So synthetic data doesn’t necessarily mean purely AI made up data. It’s also taking something from an article wikipedia article and then rephrasing it as a q a question or um summarizing it rewording it and making uh better data that way because i think Of it also like with humans if someone let’s say reads a book compared to a messy i don’t know no offense but like reddit post or something like that i do think you learn no offense, but I Lex Fridman Think… There’s going to be a post about this, Suresh. Nathan Lambert Some Reddit data is very coveted and excellent for training. You just have to filter it. Yeah. I think that’s the idea. Sebastian Raschka I think it’s like, if someone took that and rephrases that in a, let’s say, more concise and structured way i think it’s higher quality data that gets the lm maybe the same you get the same Lm out of it at the end but it gets there faster it trains faster because the let’s say if the grammar and the punctuation is correct it already learns the correct way versus getting information From a messy way and then learning later how to correct that and stuff like that so i think that is how pre-training evolved and how um how still while why scaling still works is that it’s Not about just the amount of data it’s also the tricks to make that data better for you in a sense and mid-training is i mean it used to be called a pre-training it’s i think it’s called mid-training Because it was awkward to have pre-training and post-training, but nothing in the middle, right? It sounds a bit weird to have pre-training and post-training, but what’s the actual training? So the mid-training is usually similar to pre-training, but now it’s a bit more, I would say, specialized in pre-training. It’s the same algorithm, but what you do is you focus, for example, on long context. One example you have long context documents the reason you don’t do that during just pure pre-training is because you don’t have that many long context documents you have a specific Phase and one problem of LLMs is also still it’s a neural network it has the problem of catastrophic forgetting so you teach it something it forgets other things and you want to it’s not 100 forgetting but you know it’s like no free lunch you can’t it’s also the same with humans if you ask me some math i learned 10 years ago i don’t know i would have to look at it again nathan Lex Fridman Was actually saying that he’s consuming so much content that there’s a catastrophic forgetting issue i’m like trying to learn so much about ai i was like i was learning about pre-training Nathan Lambert Parallelism i’m like i lost something and i don’t know what it was i don’t want to anthropomorphize LLMs, but it’s, I think, the same kind of in that sense, how humans learn. Sebastian Raschka I mean, the quantity is not always better because, yeah, it’s like being selective. And the mid-training is being selective in terms of quality content at the end. So the last thing the LLM has seen is the quality stuff. And then post-training is all the fine-tuning, supervised fine-tuning, DPO, reinforcement learning with verifiable rewards, with human feedback and so forth. (Time 1:18:50)
  • RLVR Unlocked Stepwise Reasoning
    • RL with verifiable rewards (RLVR) amplifies stepwise reasoning and tool use by grading correctness on verifiable tasks.
    • Nathan Lambert credits RLVR scaling for major capability gains in 2025. Transcript: Nathan Lambert Biggest one from 2025 is learning this reinforcement learning with verifiable rewards. You can scale up the training there, which means doing a lot of this kind of iterative generate grade loop and that lets the models learn both interesting behaviors on the tool use and Software side this could be searching running commands on their own and seeing the outputs and then also that training enables this inference time scaling very nicely and it just turned Out that this paradigm was very nicely linked in this where it’s this kind of rl training enables inference time scaling but inference time scaling could have been found in different Ways so it’s kind of this perfect storm of the models change a lot and the way that they’re trained is a major factor in doing so and this has changed how people approach post-training Lex Fridman Dramatically describe rl v popularized by DeepSeq R1? Can you describe how it works? Nathan Lambert Yeah, fun fact, I was on the team that came up with the term RLVR, which is from our two to three work before DeepSeq, which is, we don’t take a lot of credit for the, being the people to popularize The scaling RL, but as fun as what academics get as an aside, is the ability to name and influence the discourse because the closed labs can only say so much that one of the things you can Do as an academic is like you might not have the compute to train the model but you can frame things in a way that ends up being i describe it as like a community can come together around this Rl vr term which is very fun and then deep seek is the people that did the training breakthrough, which is they scaled the reinforcement learning, which was you have the model generate Answers and then grade the completion if it was right. And then that accuracy is your reward for reinforcement learning. So reinforcement learning is classically an agent that acts in an environment and the environment gives it state and a reward back and you try to maximize this reward. In the case of language models, the reward is normally accuracy on a set of verifiable tasks, whether it’s math problems, coding tests, and it starts get blurry with things like factual Domains, like that is also in some ways verifiable or constraints on your instruction like respond only with words that start with a like all of these things are verifiable in some way And the core idea of this is you find a lot more of these problems that are verifiable and you let the model try it many times while taking these rl steps these rl gradient updates the infrastructure Evolved from this reinforced learning from human feedback where in that era the score they were trying to optimize was a learned reward model of aggregate human preferences so you Kind of change the problem domains and that let the optimization go on to much bigger scales which kind of kick-started a major change in what the models can do and how people use them. Lex Fridman What kind of domains is RLVR amenable to? Nathan Lambert Math and code are the famous ones. And then there’s a lot of work kind of on what is called rubrics, which is related to a word people might have heard as L, I’m as a judge, which is like for each problem, I’ll have a set of problems In my training data set. I’ll then have another language model and ask it, what would a good answer to this problem look like? And then you can try the problem a bunch of times over and over again and assign a score based on this rubric. So that’s not necessarily verifiable like a math and code domain, but this rubrics idea and other scientific problems that it might be a little bit more vague is where a lot of the attention Is, where they’re trying to push this set of methods into these kind of more open-ended domains where the models can learn a lot more. Sebastian Raschka I think that’s called reinforcement learning with AI feedback, right? Nathan Lambert That’s the older term from it that was coined in Anthropics constitutional AI paper. So it’s like a lot of these things come in cycles. Sebastian Raschka Also just one step back for the RLVR. So I think the interesting, beautiful thing here is that you ask the LLM, let’s say a math question, and then you know the correct answer. And you let the LLM, like you said, figure it out. But how it does it, I mean, you don’t really constrain it much. (Time 1:52:04)
  • RLVR Often Unlocks Latent Knowledge
    • RLVR often unlocks abilities already latent from pre-training rather than creating new knowledge from scratch.
    • Quick accuracy jumps reflect formatting, rubrics, and data overlap as much as ‘new reasoning’. Transcript: Nathan Lambert There’s a lot in here. I think some of the debate, there’s been a lot of debate this year on if the language models like these aha, I think the aha moments are kind of fake because in pre-training, you essentially Have seen the whole internet. So you have definitely seen people explaining their work, even verbally, like a transcript of a math lecture. You try this, oh, I messed this up. And what reinforcement learning is this RLVR is very good at doing is amplifying these behaviors because they’re very useful in enabling the model to think longer and to check its work And i agree that it is very beautiful that this training kind of the model learns to amplify this in a way that is just so useful at the final answers being better i can give you also a hands-on Sebastian Raschka Example i was training the gren 3 base model with rl vr on math 500 the base model had an accuracy of about 15 just 50 steps like in a few minutes with rl vr the model went from 15 to 50 accuracy Nathan Lambert And the model you can’t tell me it’s learning anything about fundamentally about math and the quen example is weird because there’s been two papers this year one of which i was on that Talks about data contamination in Quinn, and specifically that they train on a lot of this special mid-training phase that we just have like a minute on because it’s weird. So they train on problems that are almost identical to math. Exactly. Sebastian Raschka And so you can see that basically the RL, it’s not teaching the model any new knowledge about math. You can’t do that in 50 steps. So the knowledge is already there in the pre-training you’re just unlocking it i (Time 1:58:12)
  • Learn By Building A Small Model First
    • Build models from scratch on one GPU to learn internals; validate by matching outputs to reference implementations.
    • Reverse-engineer configs and weights to understand real production architectures. Transcript: Sebastian Raschka So can you just take me through what you would recommend people do? Computer the goal is not if you build a model from scratch to have like something you use every day for your personal projects like it’s not going to be your personal assistant replacing An existing open weight model or chachapiti it’s to see what exactly goes into the llm what exactly comes out of the llm how the pre-training works in that sense on your own computer preferably And then you learn about the pre-training the supervised fine-tuning the attention mechanism you get a solid understanding of how things work but at some point you will reach a limit Because small models can only do so much and the problem with learning about llms at scale is i would say it’s exponentially more complex to make a larger model because it’s not that the Model just becomes larger you have to now think about sharding your parameters across multiple gpus even for the kvcache there are multiple ways you can implement it one is just to understand How it works just to grow the cache that’s like a cache you grow step by step by let’s say concatetating lists um growing it but then that wouldn’t be optimal in gpus you wouldn’t do that You would pre-allocate a tensor and then fill it in but that adds again another 20 30 lines of code and for each thing you add so much code and i think the trick with the book is basically To understand how the llm works it’s not going to be your production level llm but once you have that you can understand the production level so you’re trying to always build an llm that’s Going to fit on one GPU? Yes, most of them I have. I have some bonus materials on some MOE models. I think one or two of them, they may require multiple GPUs, but the goal is to have it on one GPU. And the beautiful thing is also you can self-verify. It’s almost like RLVR. When you code these from scratch, you can take an existing model from the Hugging Face Transformer library. So the Hugging Face Transformer library is great. But if you want to learn about LLMs, I think that’s not the best place to start because the code is so complex because it has to fit so many use cases. Also, some people use it in production. It has to be really sophisticated and it’s really intertwined and really hard. Nathan Lambert It’s not linear to read. Was started as a fine-tuning library and then it grew to be like the standard representation of every model architecture and the way it’s loaded so hugging face is like the default place To get a model and transformers is the software that enables it so people can easily load a model and do something basic with it and all frontier labs that have open weight models have Sebastian Raschka A hugging face transformers version of it, like from DeepSeq to GPT OSS. That’s like the canonical weight that you can load there. But again, also even transformers, the library is not used in production. People use then SGLang or VLLM and it adds another layer of complexity. Lex Fridman We should say that the transformers library has like 400 models. Sebastian Raschka So it’s one library that tries to implement a lot of LLMs. And so you have a huge code base, basically. It’s like huge. It’s like, it’s, I don’t know, maybe millions, hundreds of thousands of lines of code. And it’s like understanding the part that you want to understand is finding the needle in the haystack. But what’s beautiful about it is you have a working implementation. And so you can work backwards from it. What I would recommend doing or what I also do is if i want to understand for example how almost three is implemented i would look at the weights in the model hub the config file and then You can see oh they used so many layers they use let’s say group query attention or multi-head attention in that case and you see all the components in like a human readable i don’t know 100 lines of config file and then you start let’s say with your gpd2 model and add these things you know. And the cool thing here is you can then load the pre-trained weights and see if they work in your model. And you want to match the same output that you get with a transformer model. And then you can use it as a, basically as a verifiable reward to make your architecture correct. And then it’s kind of, sometimes it takes me a day to, with almost three, the challenge was a rope for the position embeddings. They had a yarn extension and there was some custom scaling there and I couldn’t quite match these things. And in this struggle, you kind of understand things. But the cool thing is, at the end, you know you have it correct because you can unit test it you can check against the reference implementation and i think that’s maybe one of the best ways To learn really like to basically reverse engineer something yeah i (Time 2:13:00)
  • Re-derive The Math Yourself
    • When learning RLHF or RLVR, read the practical derivations and re-derive math yourself to truly understand algorithms.
    • Use LLMs to help, but verify derivations manually to avoid superficial understanding. Transcript: Lex Fridman Are some of the just to go through some of the table of contents, some of the ideas we might have missed in the bigger picture of the post-training. So first of all, you do the problem setup, training overview, what are preferences, preferences data, and the optimization tools, reward modeling, regularization, instruction Tuning, rejection sampling, reinforcement learning, i.e. Policy gradients, direct alignment algorithms, then constitutional AI and AI feedback, reasoning and inference time scaling, tool use and function calling, synthetic data and Distillation evaluation, and then open question section over optimization style and information, and then product UX, character and post training. So what are some ideas worth mentioning that connect both the educational component and the research component? You mentioned the character training. This is pretty interesting. Nathan Lambert Character training is interesting because there’s so little out of it, but we talk about how people engage with these models and we feel good using them because they’re positive, but That can go too far. It could be too positive. And it’s like, essentially, how do you change your data and or decision making to make it exactly what you want and i open ai has this thing called a model spec which is essentially their Internal guideline for what they want to model to do and they publish this to developers so essentially you can know what is a failure of open ai’s training which is like they have the Intentions and they haven’t met it yet versus what is something that they like actually wanted to do and that you don’t like. And that transparency is very nice, but all the methods for curating these documents and how easy it is to follow them is not very well known. I think the way the book is designed is that the Reinforced Learning chapter is obviously what people want because everybody hears about it with RLVR. And it’s the same algorithms and the same math, but it’s just like you can use it in very different documents so i think the core preference of rlhf is like how messy preferences are is Essentially rehash of a paper i wrote years ago but this is essentially the chapter that’ll tell you why rlhf is never ever fully solvable because like the way that even rl is set up is That um it assumes that preferences can be quantified and that multiple preferences can be reduced to single values and i think it relates in the economics literature to the von neumann Morgenstein utility theorem and like that is the chapter where all of that philosophical economic and like psychological context it tells you what gets compressed into doing rlhf So it’s like you have all of this and then later in the book it’s like you use this rl math to make the number go up and i think that that’s why i think it would be very rewarding for people to Do research on is because it’s like quantifying preferences is something that is just like humans have designed the problem in order to make preferences studyable but there’s kind Of fundamental debates on like an example is in a language model response you have different things you care about whether it’s accuracy or in style and when you’re collecting the data They all get compressed into like i like this more than another and it’s like like that is happening and there’s a lot of philosophy there’s a lot of research in other areas of the world That go into like, how should you actually do this? I think social choice theory is the subfield of economics around how you should aggregate preferences. And there’s like, I was, I went to a workshop that published a white paper on like, how can you think about using social choice theory for RLHF? So I mostly would want people that get excited about the math to come and have things that they can stumble into and learn this kind of broader context. I think there’s a fun thing. I just keep a list of all the tech reports that I like of reasoning models. So in chapter 14, which is kind of like a short summary of RLVR, there’s just like a gigantic table where I just like list every single reasoning model that I like. So there’s just like, think in education a lot of it needs to be like at this point it’s like what i like because the language models are so good at the math where it’s like famous paper direct Preference optimization which is like a much simpler way of solving the problem than rl the derivations and the appendix skip steps of math. And it’s like, I tried for this book, like I redid the derivations and I’m like, what the heck is this log trick that they use to change the math? But doing it with language models, they’re like, this is the log trick. And I’m like, I don’t know if I like this, that the math is so commoditized. I think like some of the struggle in reading this appendix and following the math, I think is good for learning. (Time 2:20:54)
  • Continual Learning Is An Economics Problem
    • Continual learning is mainly a question of economics and personalization versus global model updates.
    • Sebastian Raschka notes weight updates per-user are costly; context and LoRA adapters are practical interim solutions. Transcript: Nathan Lambert Relates a lot to this kind of SF zeitgeist of what is AGI, which is artificial general intelligence, and what is ASI, artificial superintelligence, and what are the language models That we have today capable of doing. I think the language models can solve a lot of tasks, but a key milestone among the AI community is essentially when AI could replace any remote worker, taking in information and solving Digital tasks and doing them. And the limitation that’s highlighted by people is that a language model will not learn from feedback the same way that an employee is. So if you hire an editor, the editor will mess up, but you will tell them. And if you hired a good editor, they don’t do it again. But language models don’t have this ability to modify themselves and learn very quickly. So the idea is if we’re going to actually get to something that is a true general adaptable intelligence that can go into any remote work scenario, it needs to be able to learn quickly From feedback and on-job learning. I’m personally more bullish on language models by being able to just provide them with very good context. You said, like you maybe offline said that, like you can write extensive documents to models where you say, I have all this information. Here’s all the blog posts I’ve ever written. I like this type of writing. My voice is based on this, but a lot of people don’t provide this to models and the models weren’t designed to like take this amount of context previously, like the agentic models are Just starting. So it’s this kind of trade off of do we need to update the weights of this model with this continual learning thing to make them learn fast? Or the counter argument is we just need to provide them with more context and information and they will have the appearance of learning fast by just having a lot of context and being very Smart. Lex Fridman So we should mention the terminology here. So continual learning refers to changing the weights continuously so that the model adapts, adjusts based on the new incoming information, does so continually and rapidly and frequently And so on. And then the thing you mentioned on the other side of it is generally be referred to as in-context learning. As you learn stuff, there’s a huge context window. You can just keep loading it with extra information every time you prompt the system, which I think both are legitimately can be seen as learning. It’s just a different place where you’re doing the learning. Sebastian Raschka I think, to be honest with you, continual learning, the updating of weights, we already have that in different flavors. I mean, if you think about how, so I think the distinction here is, do you do that on a personalized custom model for each person or do you on a global model scale? I think we have that already with uh going from gpt5 to 5.1 and 5.2 it’s maybe not immediate but it is like a curated update a quick curated update where uh there was feedback by the things That couldn’t do feedback by the community they updated the weights next model and so forth so it is i mean kind of like a flavor of that um other even finer grade example a finer grained Example is like rlvr you run it it updates the problem is you can’t just do that for each person because it would be too expensive to update the weights for each person and i think that’s The problem so unless you get i mean even at open ai scale building data centers it would be too expensive i think that is only feasible once you have something on the device where the cost Is on the consumer like what apple tried to do with the apple foundation models putting them on the phone and then they learn from the experience a (Time 2:53:36)
  • Long Context Is A Multi-Tool Problem
    • Long context scaling combines system, data, and architecture work; incremental gains likely continue but huge leaps remain hard.
    • Teams will mix attention tweaks, state-space layers, and recursive decomposition to extend context affordably. Transcript: Nathan Lambert I think the colloquially accepted thing is that it’s a compute and data problem where you can and sometimes like small architecture things which are like attention variants so if you Have we talked about like hybrid attention models which is essentially if you have what looks like a state space model within your transformer and like those are better suited because You have to spend less compute to model the furthest along token and i think that but those aren’t free because they have to be accompanied by a lot of compute or um the right data so how Many sequences of 100 000 tokens do you have in the world and where do you get these and i think it just ends up being pretty expensive to scale them so we’ve like gotten to pretty quickly To like a million tokens of input context length and i would expect it to keep increasing and like get to like 2 million or 5 million this year but i don’t expect it to go to like 100 million That would be like a true breakthrough and i think those breakthroughs are possible like the continual learning thing i think of as a research problem where you could there could be A breakthrough that just makes transformers work way better at this and it’s cheap like these things could happen with so much scientific attention but turning the crank it’ll be consistent Sebastian Raschka Increases over time i think also looking at the extremes i think there’s again no free lunch so the one extreme to make it cheap you have a let’s say an rnn that has a single state a state Where you save everything from the previous stuff it’s like a specific uh fixed size thing so you never really grow the memory because it’s you are stuffing everything into one state But then the longer the context gets the more information you forget because you can’t you can’t keep i mean compress everything into one state then on the other hand you have the transformers Which try to remember every token which is great sometimes if you want to look up specific information but very expensive because you have the kv cache that grows the dot product that Grows but then yeah like you said the mamba layers i mean they kind of have the same problem i would say like an rnn you try to compress everything into one state you’re a bit more selective There but then i think it’s like this goldilocks zone again with nemotron 3 they found like a good ratio of how many attention layers do you need for the global information where everything Is accessible compared to having these compressed states. And I think that’s how I think we will scale more by finding better, let’s say, ratios in Goldilocks zone, like between computing, making it cheap enough to run, but then also making It powerful enough to be useful. And one more plug here, the recursive language model paper, that is one of the papers that tries to kind of address the long context thing. So what they found is essentially instead of stuffing everything into this long context, if you break it up into the smaller, multiple smaller tasks, so you save memory by having multiple Smaller calls, you can get actually better accuracy than having the LLM try everything all at once. I mean, it’s a new paradigm. We’ll see, you know, there might be other flavors of that. So I think with that, we will still make improvement on long context. But then also, like Nathan said, I think the problem is for pre-training itself, we don’t have as many long context documents as other documents. So it’s harder to study basically how LMs behave and stuff like that on that level. Nathan Lambert There are some rules of thumb where essentially you pre-train a language model. Like, oh no, we pre-trained like 8K context length and then extended to 32K with training. And there’s some rules of thumb where you’re just like essentially doubling the training context length takes like 2x compute and then you can normally like 2 to 4x the context length Again so i think a lot of it ends up being kind of compute bound at pre-training which is in this like we talked about this everyone talks about this big increase in compute for the top labs This year and that should reflect in some longer context windows but i think on the post trading side there’s some more interesting things which is as we have agents the agents are going To manage this context on their own where now people that use cloud code a lot dread the compaction which is when cloud takes its entire full 100 000 tokens of work and compacts it into Bulleted list but what the next models will do i’m just not a novel i’m sure people are already working on this is essentially the model can control when it compacts and how so you can essentially Like train your rl algorithm where compaction is an action where it shortens the history and then the problem formulation will be i want to keep the maximum evaluation scores that i Have gotten while the model compacts its history to the minimum length because then you have the minimum amount of tokens that you need to do this kind of compounding auto regressive Prediction so there’s actually pretty nice problem setups in this where the like these agentic models learn to use their context in a different way than just plow forward one (Time 2:58:45)
  • Data Rights Are A Civilizational Issue
    • Widespread LLM adoption raises legal, ethical, and societal tensions around training data, copyright, and monetization.
    • Nathan Lambert highlights high-stakes lawsuits and the need for new compensation models for authors and creators. Transcript: Nathan Lambert It’s like, Claude, you tried to use the CLI command that don’t have installed 14 times, and then I sent you the command to run. It’s like that thing from a modeling perspective is pretty fixable. I agree with you. Lex Fridman I’ve been becoming more and more bullish in general. Speaking to what you’re articulating, I think it is a human skill issue so anthropic is leading the way in or other companies in understanding how to best use the models for programming Therefore they’re effectively using them i think there’s a lot of programmers on the outskirts they’re like they don’t i mean there’s not a really good guide on how to use them people Trying to figure it out exactly it might be very expensive like it might be that the entry point for that is two thousand dollars a month which is only tech companies and rich people just Like like that could be it but it might be worth it i mean if if the final result is a working software system well they’d be worth it but by the way it’s funny how we converge from the discussion Of timeline to AGI to something more pragmatic and useful. Is there anything concrete and interesting and useful and profound to be said about timeline to AGI and ASI? Or are these discussions a bit too detached from the day-to There’s interesting bets. Nathan Lambert So there’s a lot of people trying to do reinforcement learning with verifiable rewards, but in real scientific domains where there’s startups that are spending, like they have hundreds Of millions of dollars of funding and they have wet labs where they’re having language models propose hypotheses that are tested in the real world. And I would say that I think they’re very early or they’re early, but with the pace of progress, it’s like maybe they’re early by six months and they make it because they were there first Or maybe they’re early by eight years you don’t really know so i think that that type of moonshot to um branch this momentum into other other sciences is like okay like that would be very Transformative if like alpha fold moments happen in all sorts of other scientific domains by like a startup solving this i think there are startups i think maybe harmonic is one where They’re going all in on language models plus lean for math i think you had another podcast guest talked about this recently and it’s like we don’t know exactly what’s going to fall out Of spending 100 million dollars on that model and most of them will fail but a couple of them might be big breakthroughs that are very different than chat gpt or cloud code type software Experiences like a tool that’s only good for a phd mathematician but makes them 100x effective like okay well i agree i think this will happen in a lot of domains especially also like Sebastian Raschka Domains that have a lot of um you know resources like finance and legal and pharmaceutical companies but then again is it really agi again because we are now specializing it again and Then again is it really that much different from back in the day how we had specialized algorithms i think it’s just the same thing more way more sophisticated but i don’t know is there A threshold when we call it agi i guess i think the the real cool thing is here that we have like the foundation models that we can specialize i think that that’s like the breakthrough at Some point right now i think we are not there yet because well first it’s too expensive but also you know like chatgbd doesn’t just give away that chat gbd to customize it i think once that’s Going to be true in some way and i think i can imagine this as a business model that uh chat gbd open i says at some point like hey you know bank of america 400 million we will do your custom Model or something like that and i think that will be the huge economic value add the other thing though is also companies i mean right now what is the differentiating factor i mean if Everyone uses the same llm if everyone uses chachupiti they will all do the same thing again i mean then well it’s everyone is moving in lockstep but usually companies they want to have A competitive advantage and i think there’s no way around using some of their private data and experimenting and maybe specializing. It’s going to be interesting. Yeah. Nathan Lambert Sitting in the pace of progress, it does just feel like things are coming. I don’t think the AGI and ASI thresholds are particularly useful. I (Time 3:26:56)