Skip to content

Podcast

#452 – Dario Amodei- Anthropic CEO on Claude, AGI & the Future of AI & Humanity

Lex Fridman Podcast

Source ↗ ← All highlights
  • Scaling Laws
    • Scaling laws suggest linearly scaling network size, training time, and data improves model performance.
    • This observation has held true across various AI domains, like speech and language, despite initial skepticism. Transcript: Lex Fridman Start with the big idea of scaling laws and the scaling hypothesis. What is it? What is its history? And where do we stand today? Dario Amodei So I can only describe it as it relates to kind of my own experience, but I’ve been in the AI field for about 10 years. And it was something I noticed very early on. So I first joined the AI world when I was working at Baidu with Andrew Ng in late 2014, which is almost exactly 10 years ago now. And the first thing we worked on was speech recognition systems. And in those days, I think deep learning was a new thing. It had made lots of progress, but everyone was always saying, we don’t have the algorithms we need to succeed. We’re only matching a tiny, tiny fraction. There’s so much we need to kind of discover algorithmically. We haven’t found the picture of how to match the human brain. And in some ways, it was fortunate. I was kind of, you can have almost beginner’s luck, right? I was like a newcomer to the field. And I looked at the neural net that we were using for speech, the recurrent neural networks. And I said, I don’t know, what if you make them bigger and give them more layers? And what if you scale up the data along with this, right? I just saw these as like independent dials that you could turn. And I noticed that the model started to do better and better as you gave them more data, as you made the models larger, as you trained them for longer. And I didn’t measure things precisely in those days, but along with colleagues, we very much got the informal sense that the more data and the more compute and the more training you put Into these models, the better they perform. And so initially my thinking was, hey, maybe that is just true for speech recognition systems, right? Maybe that’s just one particular quirk, one particular area. I think it wasn’t until 2017 when I first saw the results from GPT-1 that it clicked for me that language is probably the area in which we can do this. We can get trillions of words of language data. We can train on them. And the models we were trained in those days were tiny. You could train them on one to eight GPUs, whereas now we train jobs on tens of thousands, soon going to hundreds of thousands of GPUs. And so when I saw those two things together, and there were a few people like Ilya Suitskiver, who you’ve interviewed, who had somewhat similar views, right? He might’ve been the first one, although I think a few people came to similar views around the same time, right? There was, you know, Rich Sutton’s Bitter Lesson. There was Gorin wrote about the scaling hypothesis. But I think somewhere between 2014 and 2017 was when it really clicked for me, when I really got conviction that, hey, we’re going to be able to do these incredibly wide cognitive tasks If we just scale up the models. And at every stage of scaling, there are always arguments. And when I first heard them, honestly, I thought probably I’m the one who’s wrong. And all these experts in the field are right. They know the situation better than I do, right? There’s the Chomsky argument about like, you can get syntactics, but you can’t get semantics. There was this idea, oh, you can make a sentence make sense, but you can’t make a paragraph make sense. The latest one we have today is, you know, we’re going to run out of data or the data isn’t high quality enough or models can’t reason. And each time, every time we manage to either find a way around or scaling just is the way around. Sometimes it’s one, sometimes it’s the other. And so I’m now at this point, I still think, you know, it’s always quite uncertain. We have nothing but inductive inference to tell us that the next few years are going to be like the last 10 years. But I’ve seen the movie enough times. I’ve seen the story happen for enough times to really believe that probably the scaling is going to continue and that there’s some magic to it that we haven’t really explained on a theoretical Basis yet. (Time 0:10:20)
  • Why Bigger Networks Are Better
    • Larger networks capture simpler patterns first, then progressively learn more complex ones with increased capacity.
    • This is analogous to 1/f noise, where larger scales reveal finer details in a distribution. Transcript: Lex Fridman Of course, the scaling here is bigger networks, bigger data, bigger compute. Yes. Dario Amodei In particular, linear scaling up of bigger networks, bigger training times, and more data. So all of these things, almost like a chemical reaction, you know, you have three ingredients in the chemical reaction, and you need to linearly scale up the three ingredients. If you scale up one, not the others, you run out of the other reagents and the reaction stops. But if you scale up everything in series, then the reaction can proceed. (Time 0:14:16)
  • Limits of Scaling
    • AI progress may be limited by data quality and availability, as internet data has limitations.
    • Synthetic data generation or new training methods like chain-of-thought reasoning may overcome this. Transcript: Lex Fridman What do you think would be the reason? Is it compute limited, data limited? Is it something else, idea limited? Dario Amodei So a few things. Now we’re talking about hitting the limit before we get to the level of humans and the skill of humans. So I think one that’s popular today, and I think could be a limit that we run into, like most of the limits, I would bet against it, but it’s definitely possible, is we simply run out of data. There’s only so much data on the internet. And there’s issues with the quality of the data, right? You can get hundreds of trillions of words on the internet, but a lot of it is repetitive or it’s search engine, you know, search engine optimization drivel, or maybe in the future it’ll Even be text generated by AIs itself. And so I think there are limits to what can be produced in this way. That said, we, and I would guess other companies are working on ways to make data synthetic where you can, you know, you can use the model to generate more data of the type that you have, That you have already, or even generate data from scratch. If you think about what was done with DeepMind’s AlphaGo Zero, they managed to get a bot all the way from, you know, no ability to play Go whatsoever to above human level just by playing Against itself. There was no example data from humans required in the AlphaGo Zero version of it. The other direction, of course, is these reasoning models that do chain of thought and stop to think and reflect on their own thinking. In a way, that’s another kind of synthetic data coupled with reinforcement learning. So my guess is with one of those methods, we’ll get around the data limitation, or there may be other sources of data that are available. We could just observe that even if there’s no problem with data, as we start to scale models up, they just stop getting better. It seemed to be our reliable observation that they’ve gotten better. That could just stop at some point for a reason we don’t understand. The answer could be that we need to, you know, we need to invent some new architecture. It’s been there have been problems in the past with, say, numerical stability of models where it looked like things were were leveling off. But but actually, you know, when we found the right unblocker, they didn’t end up doing so. So perhaps there’s some new optimization method or some new technique we need to unblock things. I’ve seen no evidence of that so far, but if things were to slow down, that perhaps could (Time 0:22:50)
  • Golden Gate Claude
    • Anthropic released a demo model fixated on the Golden Gate Bridge due to an activated feature.
    • This demonstrated how specific activations influence model behavior and create unique personalities. Transcript: Lex Fridman Golden Gate Bridge. Or, you know. It would masterfully change topic to the Golden Gate Bridge and integrate it. There was also a sadness to it, to the focus it had on the Golden Gate Bridge. I think people quickly fell in love with it, I think. So people already miss it because it was taken down, I think, after a day. Dario Amodei Somehow these interventions on the model where you kind of adjust its behavior somehow emotionally made it seem more human than any other version of the model would seem. It’s a strong personality, strong identity. It has a strong personality. It has these kind of like obsessive interests. You know, we can all think of someone who’s like obsessed (Time 0:32:34)
  • Claude’s Perceived Intelligence
    • User complaints about Claude getting dumber are likely due to unchanging model weights and psychological effects.
    • Subtle prompt changes or A/B testing might cause perceived differences, but the core model remains static. Transcript: Lex Fridman Know, there’s just this fascinating, to me at least, it’s a psychological social phenomenon where people report that Claude has gotten dumber for them over time. And so the question is, does the user complaint about the dumbing down of Claude 3-5 Sonnet hold any water? So are these anecdotal reports a kind of social phenomena or is there any cases where Claude would get dumber? Dario Amodei So this actually doesn’t apply. This isn’t just about Claude. I believe I’ve seen these complaints for every foundation model produced by a major company. People said this about GPT-4. They said it about GPT-4 Turbo. So a couple things. One, the actual weights of the model, right? The actual brain of the model, that does not change unless we introduce a new model. There are just a number of reasons why it would not make sense practically to be randomly substituting in new versions of the model. It’s difficult from an inference perspective, and it’s actually hard to control all the consequences of changing the weights of the model. Let’s say you wanted to fine-tune the model to be like, I don’t know, to say certainly less, which an old version of Sonnet used to do. You actually end up changing a hundred things as well. So we have a whole process for it, and we have a whole process for modifying the model. We do a bunch of testing on it. We do a bunch of user testing in early customers. So we both have never changed the weights of the model without telling anyone. And certainly in the current setup, it would not make sense to do that. Now, there are a couple of things that we do occasionally do. One is sometimes we run A-B tests, but those are typically very close to when a model is being released and for a very small fraction of time. So, you know, like the day before the new Sonnet 3.5, I agree, we should have had a better name. It’s clunky to refer to it. There were some comments from people that like it’s gotten a lot better and that’s because, you know, a fraction were exposed to an A-B test for those one or two days. The other is that occasionally the system prompt will change. The system prompt can have some effects, although it’s unlikely to dumb down models. It’s unlikely to make them dumber. And we’ve seen that while these two things, which I’m listing to be very complete, happen relatively, happen quite infrequently. The complaints about, for us and for other model companies about the model change, the model isn’t good at this, the model got more censored, the model was dumbed down, those complaints Are constant. And so I don’t want to say like people are imagining it or anything, but like the models are for the most part, not changing. Um, if I were to offer a theory, um, I, I think it actually relates to one of the things I said before, which is that models have many are very complex and have many aspects to them. And so often, you know, if I, if I, if I, if I asked the model a question, you know, if I’m like, if I’m like, do X versus can you do Task X, the model might respond in different ways. And so there are all kinds of subtle things that you can change about the way you interact with the model that can give you very different results. To be clear, this itself is like a failing by us and by the other model providers that the models are just often sensitive to like small changes in wording. It’s yet another way in which the science of how these models work is very poorly developed. And so, you know, if I go to sleep one night and I was like talking to the model in a certain way and I like slightly change the phrasing of how I talk to the model, you know, I could get different Results. So that’s one possible way. The other thing is, man, it’s just hard to quantify this stuff. It’s hard to quantify this stuff. I think people are very excited by new models when they come out. And then as time goes on, they become very aware of the limitations. So that may be another effect. But that’s all a very long-winded way of saying, for the most part, with some fairly narrow exceptions, the models are not changing. I think there is a psychological effect. (Time 0:49:12)
  • Using Claude for Computer Tasks
    • Give Claude tasks by providing screenshots and instructions for clicks and key presses.
    • This simple loop allows interaction with computer interfaces, enabling spreadsheet work and website navigation. Transcript: Lex Fridman Of the ways that cloud has been getting more and more powerful is it’s now able to do some agentic stuff, um, computer use. Uh, there’s also an analysis within the sandbox of cloud.ai itself, but let’s talk about computer use. That’s seems to me super exciting that you can just give cloud a task and it takes a bunch of actions, figures it out, and has access to your computer through screenshots. So can you explain how that works and where that’s headed? Dario Amodei Yeah, it’s actually relatively simple. So Claude has had for a long time since Claude 3 back in March the ability to analyze images and respond to them with text. The only new thing we added is those images can be screenshots of a computer. And in response, we train the model to give a location on the screen where you can click and or buttons on the keyboard you can press in order to take action. And it turns out that with actually not all that much additional training, the models can get quite good at that task. It’s a good example of generalization. You know, people sometimes say if you get to low Earth orbit, you’re like halfway to anywhere, right, because of how much it takes to escape the gravity well. If you have a strong pre-trained model, I feel like you’re halfway to anywhere in terms of the intelligence space. And so actually, it didn’t take all that much to get Claude to do this. And you can just set that in a loop, give the model a screenshot, tell it what to click on, give it the next screenshot, tell it what to click on. And that turns into a full kind of almost 3D video interaction of the model. (Time 1:16:46)
  • AI Regulation
    • Dario advocates for AI regulation, like California’s SB 1047, to ensure industry-wide safety standards.
    • He emphasizes the need for surgical and targeted regulations to avoid hindering innovation. Transcript: Lex Fridman Having good models. Let me ask about regulation. What’s the role of regulation in keeping AI safe? So for example, can you describe California AI regulation bill SB 1047 that was ultimately vetoed by the governor? What are the pros and cons of this bill? Yes. Dario Amodei We ended up making some suggestions to the bill and then some of those were adopted and we felt, I think, quite positively about the bill by the end of that. It did still have some downsides. And, you know, of course, it got vetoed. I think at a high level, I think some of the key ideas behind the bill are, you know, I would say similar to ideas behind our RSPs. And I think it’s very important that some jurisdiction, whether it’s California or the federal government and or other countries and other states, passes some regulation like this. And I can talk through why I think that’s so important. So I feel good about our RSP. It’s not perfect. It needs to be iterated on a lot, but it’s been a good forcing function for getting the company to take these risks seriously, to put them into product planning, to really make them a central Part of work at Anthropic and to make sure that all of a thousand people, and it’s almost a thousand people now at Anthropic, understand that this is one of the highest priorities of the Company, if not the highest priority. But one, there are still some companies that don’t have RSP-like mechanisms. Like OpenAI, Google did adopt these mechanisms a couple months after Anthropic did. But there are other companies out there that don’t have these mechanisms at all. And so if some companies adopt these mechanisms and others don’t, it’s really going to create a situation where some of these dangers have the property that it doesn’t matter if three Out of five of the companies are being safe. If the other two are being unsafe, it creates this negative externality. And I think the lack of uniformity is not fair to those of us who have put a lot of effort into being very thoughtful about these procedures. The second thing is I don’t think you can trust these companies to adhere to these voluntary plans in their own, right? I like to think that Anthropic will. We do everything we can that we will. Our RSP is checked by our long-term benefit trust. So we do everything we can to adhere to our own RSP. But you hear lots of things about various companies saying, oh, they said they would give this much compute and they didn’t. They said they would do this thing and they didn’t. I don’t think it makes sense to litigate particular things that companies have done. But I think this broad principle that if there’s nothing watching over them, there’s nothing watching over us as an industry, there’s no guarantee that we’ll do the right thing, and The stakes are very high. And so I think it’s important to have a uniform standard that everyone follows and to make sure that simply that the industry does what a majority of the industry has already said is important And has already said that they definitely will do. I think there’s a class of people who are against regulation on principle. I understand where that comes from. If you go to Europe and, you know, you see something like GDPR, you see some of the other stuff that they’ve done, you know, some of it’s good, but some of it is really unnecessarily burdensome. And I think it’s fair to say really has slowed, really has slowed innovation. And so I understand where people are coming from on priors. I understand why people come from, start from that position. But again, I think AI is different. If we go to the very serious risks of autonomy and misuse that I talked about just a few minutes ago, I think that those are unusual and they weren’t an unusually strong response. And so I think it’s very important. Again, we need something that everyone can get behind. You know, I think one of the issues with SB 1047, especially the original version of it, was it had a bunch of the structure of RSPs, but it also had a bunch of stuff that was either clunky Or that just would have created a bunch of burdens, a bunch of hassle, and might even have missed the target in terms of addressing the risks. You don’t really hear about it on Twitter. You just hear about kind of, you know, people are people are cheering for any regulation. And then the folks who are against make up these often quite intellectually dishonest arguments about how, you know, you know, it’ll make us move away from California. Bill doesn’t apply if you’re headquartered in California. Bill only applies if you do business in California or that it would damage the open source ecosystem, or that it would cause all of these things. I think those were mostly nonsense, but there are better arguments against regulation. There’s one guy, Dean Ball, who’s really, I think, a very scholarly analyst who looks at what happens when a regulation is put in place in ways that they can kind of get a life of their own Or how they can be poorly designed. And so our interest has always been, we do think there should be regulation in this space, but we want to be an actor who makes sure that that regulation is something that’s surgical, That’s targeted at the serious risks, and is something people can actually comply with. Because something I think the advocates of regulation don’t understand as well as they could is if we get something in place that’s poorly targeted, that wastes a bunch of people’s Time, what’s going to happen is people are going to say, see these safety risks there. You know, this is this is nonsense. I just you know, I just had to hire 10 lawyers to, you know, to fill out all these forms. I had to run all these tests for something that was clearly not dangerous. And after six months of that, there will be there will be a groundswell and we’ll we’ll we’ll end up with a durable consensus against regulation. And so I think the worst enemy of those who want real accountability is badly designed regulation. We need to actually get it right. And if there’s one thing I could say to the advocates, it would be that I want them to understand this dynamic better. And we need to be really careful and we need to talk to people who actually have experience seeing how regulations play out in practice. And the people who have seen that understand to be very careful. If this was some lesser issue, I might be against regulation at all. But what I want the opponents to (Time 1:26:40)
  • Advice for Aspiring AI Researchers
    • Start playing with AI models and APIs to gain practical experience.
    • Explore less crowded research areas like mechanistic interpretability, long-horizon learning, or evaluations. Transcript: Lex Fridman Said what it takes to be a great AI researcher. Can we rewind the clock back? What advice would you give to people interested in AI? Dario Amodei If they’re young, looking forward, how can I make an impact on the world? I think my number one piece of advice is to just start playing with the models. This was actually, I worry a little, this seems like obvious advice now. I think three years ago, it wasn’t obvious and people started by, oh, let me read the latest reinforcement learning paper. Let me, you know, let me, let me kind of, no, I mean, that was really the, that was really the, the, and I mean, you should do that as well. But now, you know, with wider availability of models and APIs, people are doing this more, but I think, I think just experiential knowledge. These models are new artifacts that no one really understands. And so getting experience playing with them. I would also say, again, in line with the like, do something new, think in some new direction. Like, there are all these things that haven’t been explored. Like, for example, mechanistic interpretability is still very new. It’s probably better to work on that than it is to work on new model architectures because it’s more popular than it was before. There are probably like 100 people working on it, but there aren’t like 10,000 people working on it. And it’s just this fertile area for study. There’s so much low-hanging fruit. You can just walk by and, you know, you can just walk by and you can pick things. Um, and, and the, the only reason for whatever reason people aren’t, people aren’t interested in it enough. I think there are some things around long, long horizon learning and long horizon tasks where there’s a lot to be done. I think evaluations are still, we’re still very early in our ability to study evaluations, particularly for dynamic systems acting in the world. I think there’s some stuff around multi-agent, um, skate where the puck is going is my, is my advice. And you don’t have to be brilliant to think of it. Like all the things that are going to be exciting in five years, like in people even mentioned them as like, you know, conventional wisdom, but like, it’s there’s this barrier that people Don’t double down as much as they could, or they’re afraid to do something that’s not the popular thing. I don’t know why it happens, but getting over that barrier, that’s my number one piece of advice. Let’s (Time 1:52:01)
  • Constitutional AI
    • Constitutional AI uses a set of human-readable principles to guide model responses, enabling self-play.
    • The model evaluates its responses against these principles, reducing reliance on human feedback. Transcript: Lex Fridman What is it? Dario Amodei Yes. So this was from two years ago. The basic idea is, so we describe what RLHF is. You have a model and it, you know, spits out two, you know, like you just sample from it twice. It spits out two possible responses and you’re like human, which response do like better? Or another variant of it is rate this response on a scale of one to seven. So that’s hard because you need to scale up human interaction. And it’s very implicit, right? I don’t have a sense of what I want the model to do. I just have a sense of like what this average of a thousand humans wants the model to do. So two ideas. One is, could the AI system itself decide which response is better, right? Could you show the AI system these two responses and ask which response is better? And then second, well, what criterion should the AI use? And so then there’s this idea, could you have a single document, a constitution, if you will, that says these are the principles the model should be using to respond? And the AI system reads those principles as well as reading the environment and the response. And it says, well, how good did the AI model do? It’s basically a form of self-play. You’re kind of training the model against itself. And so the AI gives the response, and then you feed that back into what’s called the preference model, which in turn feeds the model to make it better. So you have this triangle of like the AI, the preference model, and the improvement of the AI itself. Lex Fridman And we should say that in the constitution, the set of principles are like human interpretable. They’re like… Dario Amodei Yeah, yeah. It’s something both the human and the AI system can read. So it has this nice kind of translatability or symmetry. In practice, we both use a model constitution and we use RLHF and we use some of these other methods. So it’s turned into one tool in a toolkit that both reduces the need for RLHF and increases the value we get from using each data point of RLHF. It also interacts in interesting ways with kind of future reasoning type RL methods. So it’s one tool in the toolkit, but I think it is a very important tool. (Time 1:59:58)
  • Machines of Loving Grace
    • Dario’s “Machines of Loving Grace” essay explores the potential positive impacts of powerful AI.
    • It emphasizes AI’s ability to accelerate breakthroughs in fields like biology, curing diseases, and extending lifespans. Transcript: Lex Fridman And you took sort of a bold stance because it’s very possible that you might be wrong on the dates or the specific applications. Dario Amodei Oh, yeah. I’m fully expecting to definitely be wrong about all the details. I might be just spectacularly wrong about the whole thing and people will laugh at me for years. That’s just how the future works. Lex Fridman So you provided a bunch of concrete positive impacts of AI and how, you know, exactly a super intelligent AI might accelerate the rate of breakthroughs in, for example, biology and Chemistry that would then lead to things like we cure most cancers, prevent all infectious disease, double the human lifespan, and so on. So let’s talk about this essay first. Can you give a high-level vision of this essay and what key takeaways that people have? Dario Amodei Yeah, I have spent a lot of time, and Anthropic has spent a lot of effort on like, you know, how do we address the risks of AI, right? How do we think about those risks? Like, we’re trying to do a race to the top, you know, that requires us to build all these capabilities and the capabilities are cool, but, you know, we’re like a big part of what we’re trying To do is like address the risks. And the justification for that is like, well, you know, all these positive things, you know, the market is this very healthy organism, right? It’s going to produce all the positive things. The risks, I don’t know. We might mitigate them. We might not. And so we can have more impact by trying to mitigate the risks. But I noticed that one flaw in that way of thinking, and it’s not a change in how seriously I take the risks. It’s maybe a change in how I talk about them, is that, you know, no matter how kind of logical or rational that line of reasoning that I just gave might be, if you kind of only talk about risks, Your brain only thinks about risks. And so I think it’s actually very important to understand, what if things do go well? And the whole reason we’re trying to prevent these risks is not because we’re afraid of technology, not because we want to slow it down. It’s because if we can get to the other side of these risks, right, if we can run the gauntlet successfully, to put it in stark terms, then on the other side of the gauntlet are all these Great things. And these things are worth fighting for. And these things can really inspire people. And I think I imagine because, look, you have all these investors, all these VCs, all these AI companies talking about all the positive benefits of AI. But as you point out, it’s weird. There’s actually a dearth of really getting specific about it. There’s a lot of random people on Twitter, like posting these kind of like gleaming cities and this just kind of like vibe of like, grind, accelerate harder, like kick out the decel. You know, it’s just this very like aggressive ideological, but then you’re like, well, what are you, what are you actually excited about? And so I figured that, you know, I think it would be interesting and valuable for someone who’s actually coming from the risk side to try and really make a try at explaining, explaining, Explain what the benefits are. Because I think it’s something we can all get behind. And I want people to understand, I want them to really understand that this isn’t doomers versus accelerationists. This is that if you have a true understanding of where things are going with AI, and maybe that’s the more important axis, AI is moving fast versus is not moving fast, then you really appreciate The benefits and you really, you want humanity, our civilization to seize those benefits, but you also get very serious about anything that could derail them. So I think the starting point is to talk about what this powerful AI, which is the term you like to (Time 2:05:24)
  • Claude’s Character Development
    • Amanda Askell focuses on Claude’s character, aiming for nuanced, ethical, and respectful interactions.
    • Her goal is to make Claude behave ideally, like a person aware of their impact on millions. Transcript: Lex Fridman I guess there’s like a Slack channel where the legend goes, you just talk to it nonstop. So what’s the goal of creating and crafting Claude’s character and personality? Amanda Askell It’s also funny if people think that about the Slack channel, because I’m like, that’s one of like five or six different methods that I have for talking with Claude. And I’m like, yes, this is a tiny percentage of how much I talk with Claude. I think the goal, like one thing I really like about the character work is from the outset, it was seen as an alignment piece of work and not something like a product consideration um which Isn’t to say I don’t think it makes Claude I think it actually does make Claude like enjoyable to talk with at least I hope so um but I guess like my main thought with it has always been trying To get Claude to behave the way you would kind of ideally want anyone to behave if they were in Claude’s position. So imagine that I take someone and they know that they’re going to be talking with potentially millions of people so that what they’re saying can have a huge impact. And you want them to behave well in this like really rich sense. So I think that doesn’t just mean like being say ethical, though it does include that and not being harmful, but also being kind of nuanced, you know, like thinking through what a person Means, trying to be charitable with them, being a good conversationalist, like really in this kind of like rich sort of Aristotelian notion of what it is to be a good person and not in This kind of like thin like ethics as a more comprehensive notion of what it is to be so that includes things like when should you be humorous when should you be caring how much should you Like respect autonomy and people’s like ability to form opinions themselves and how should you do how should you I think that’s the kind of like rich sense of character that I wanted To and still do want Claude to have. (Time 2:56:31)
  • Prompt Engineering
    • Write clear, specific prompts for language models, defining terms like a philosopher to ensure clarity.
    • Iteratively refine prompts by testing edge cases and incorporating examples for improved performance. Transcript: Lex Fridman Yeah, that’s interesting. That’s really interesting. So on that topic, so the way to produce creativity or something special, you mentioned writing prompts and I’ve heard you talk about, I mean, the science and the art of prompt engineering. Could you just speak to what it takes to write great prompts? Amanda Askell I really do think that philosophy has been weirdly helpful for me here more than in many other respects. So in philosophy, what you’re trying to do is convey these very hard concepts. One of the things you are taught is like and and i think it is because it is i think it is an anti-bullshit device in philosophy philosophy is an area where you could have people bullshitting And you don’t want that um and so it’s like this like desire for like extreme clarity so it’s like anyone could just pick up your paper read it and know exactly what you’re talking about It’s why it can almost be kind of dry like all of the terms are defined, every objection’s kind of gone through methodically. And it makes sense to me because I’m like, when you’re in such an a priori domain, like you just, clarity is sort of this way that you can, you know, prevent people from just kind of making Stuff up. And I think that’s sort of what you have to do with language models. Like very often I actually find myself doing sort of mini versions of philosophy. You know, so I’m like, suppose that you give me a task, I have a task for the model and I want it to like pick out a certain kind of question or identify whether an answer has a certain property. Like I’ll actually sit and be like, let’s just give this a name, this property. So like, you know, suppose I’m trying to tell it like, oh, I want you to identify whether this response was rude or polite. I’m like, that’s a whole philosophical question in and of itself. So I have to do as much like philosophy as I can in the moment to be like, here’s what I mean by rudeness. And here’s what I mean by politeness. And then there’s a like, there’s another element that’s a bit more, I guess i don’t know if this is scientific or empirical i think it’s empirical so like i take that description and then What i want to do is is again probe the model like many times like this is very prompting is very iterative like i think a lot of people where they’re if a prompt is important they’ll iterate On it hundreds or thousands of times um and so you give it the instructions and then I’m like, what are the edge cases? So if I looked at this, so I try and like almost like, you know, see myself from the position of the model and be like, what is the exact case that I would misunderstand or where I would just Be like, I don’t know what to do in this case. And then I give that case to the model and I see how it responds. And if I think I got it wrong, I add more instructions or I add that in as an example. So these very like taking the examples that are right at the edge of what you want and don’t want and putting those into your prompt as like an additional kind of way of describing the thing. Of like, it’s really just trying to do clear exposition. And I think I do that because that’s how I get clear on things myself. So in many ways, like clear prompting for me is often just me understanding what I want. It’s like half the task. (Time 3:12:44)
  • Optimal Rate of Failure
    • An optimal rate of failure exists in various domains; too little failure can indicate insufficient risk-taking.
    • The cost of failure influences the optimal rate, with higher costs demanding lower failure rates. Transcript: Lex Fridman Take a tangent on that, since it reminded me of a blog post you wrote on optimal rate of failure. Oh, yeah. Can you explain the key idea there? How do we compute the optimal rate of failure in the various domains of life? Amanda Askell Yeah, I mean, it’s a hard one because it’s like, what is the cost of failure is a big part of it. Yeah, so the idea here is, I think in a lot of domains, people are very punitive about failure. And I’m like, there are some domains where, especially cases, you know, I’ve thought about this with like social issues. I’m like, it feels like you should probably be experimenting a lot. Cause I’m like, we don’t know how to solve a lot of social issues, but if you have an experimental mindset about these things, you should expect a lot of social programs to like fail. And for you to be like, well, we tried that. It didn’t quite work, but we got a lot of information that was really useful. And yet people are like, if, if a social program doesn’t work, I feel like there’s a lot of like, this is just something must have gone wrong. And I’m like, or correct decisions were made. Like maybe someone just decided like it’s worth a try. It’s worth trying this out. And so seeing failure in a given instance doesn’t actually mean that any bad decisions were made. And in fact, if you don’t see enough failure, sometimes that’s more concerning. And so like in life, you know, I’m like, if I don’t fail occasionally, I’m like, am I trying hard enough? Like, like, surely there’s harder things that I could try or bigger things that I could take on if I’m literally never failing. And so in and of itself, I think like not failing is often actually kind of a failure. Now, this varies because I’m like, well, you know, if this is easy to say when, especially as failure is like less costly, you know, so at the same time, I’m not going to go to someone who Is like, I don’t know, like living month to month and then be like, why don’t you just try to do a startup? Like, I’m just not I’m not going to say that to that person because I’m like, well, that’s a huge risk. You might like you maybe have a family depending on you. You might lose your house. Like then I’m like, actually, your optimal rate of failure is quite low and you should probably play it safe. Because like right now, you’re just not in a circumstance where you can afford to just like fail and it not be costly. And yeah, in cases with AI, I guess, I think similarly, where I’m like, if the failures are small, and the costs are kind of like low, then I’m like, then, you know, you’re just going to See that, like, when you do the system prompt, you can’t iterate on it forever. But the failures are probably, hopefully going to be kind of small, and you can like fix them. Really big failures, like things that you can’t recover from, I’m like, those are the things that actually I think we tend to underestimate the badness of um I’ve thought about this Strangely in my own life where I’m like I just think I don’t think enough about things like car accidents or like or like I’ve thought this before but like how much I depend on my hands for My work then I’m like things that just injure my hands I’m like you, you know, I don’t know. It’s like, there’s, these are like, there’s lots of areas where I’m like, the cost of failure there is really high. And in that case, it should be like close to zero. Like I probably just wouldn’t do a sport if they were like, by the way, lots of people just like break their fingers a whole bunch doing this. I’d be like, that’s not for me. (Time 3:54:38)
  • Mechanistic Interpretability
    • Mechanistic interpretability aims to reverse engineer neural networks to understand their internal algorithms.
    • Unlike saliency maps, it focuses on the underlying processes rather than just identifying important inputs. Transcript: Chris Olah I think one useful way to think about neural networks is that we don’t program them, we don’t make them, we kind of, we grow them. You know, we have these neural network architectures that we design and we have these loss objectives that we create. And the neural network architecture, it’s kind of like a scaffold that the circuits grow on. And they sort of, you know, it starts off with some kind of random, you know, random things and it grows. And it’s almost like the objective that we train for is this light. And so we create the scaffold that it grows on and we create the, you know, the light that it grows towards. But the thing that we actually create, it’s this almost biological, you know, entity or organism that we’re studying. And so it’s very, very different from any kind of regular software engineering. Because at the end of the day, we end up with this artifact that can do all these amazing things. It can, you know, write essays and translate and, you know, understand images. It can do all these things that we have no idea how to directly create a computer program to do. And it can do that because we grew it. We didn’t write it. We didn’t create it. And so then that leaves open this question at the end, which is what the hell is going on inside these systems? And that, you know, is, you know, to me, a really deep and exciting question. It’s, you know, a really exciting scientific question to me. It sort of is like the question that is just screaming out. It’s calling out for us to go and answer it when we talk about neural networks. And I think it’s also a very deep question for safety reasons. Lex Fridman So mechanistic interpretability, I guess, is closer to maybe neurobiology? Yeah, yeah, I think that’s right. Chris Olah So maybe to give an example of the kind of thing that has been done that I wouldn’t consider to be mechanistic interpretability, there was for a long time a lot of work on saliency maps, Where you would take an image and you try to say, you know, the model thinks this image is a dog. What part of the image made it think that it’s a dog? And, you know, that tells you maybe something about the model, if you can come up with a principled version of that. But it doesn’t really tell you, like, what algorithms are running in the model. How was the model actually making that decision? Maybe it’s telling you something about what was important to it, if you can make that method work. But it isn’t telling you, you know, what are the algorithms that are running? How is it that the system is able to do this thing that no one knew how to do? And so I guess we started using the term mechanistic interoperability to try to sort of draw that divide or to distinguish ourselves in the work that we’re doing in some ways from some Of these other things. And I think since then, it’s become this sort of umbrella term for, you know, a pretty wide variety of work. But I’d say that the things that are kind of distinctive are, I think, A, this focus on we really want to get at, you know, the mechanisms, we want to get at the algorithms. You know, if you think of neural networks as being like a computer program, then the weights are kind of like a binary computer program. And we’d like to reverse engineer those weights and figure out what algorithms are running. So, okay, I think one way you might think of trying to understand a neural network is that it’s kind of like we have this compiled computer program. And the weights of the neural network are the binary. And when the neural network runs, that’s the activations. And our goal is ultimately to go and understand these weights. And so, you know, the project of mechanistic interpolability is to figure out how do these weights correspond to algorithms. And in order to do that, you also have to understand the activations because it’s sort of the activations are like the memory. And if you imagine reverse engineering a computer program and you have the binary instructions, you order to understand what a particular instruction means, you need to know what Is stored in the memory that it’s operating on. And so those two things are very intertwined. So mechanistic interoperability tends to be interested in both of those things. That’s interested in those things, especially all this work on probing, which you might see as part of being mechanistic interpolability, although it’s, again, it’s just a broad Term and not everyone who does that work would identify as doing mechanterp. I think a thing that is maybe a little bit distinctive to the vibe of mechanterp is, I think people working in this space tend to think of neural networks as, well, maybe one way to say is That gradient descent is smarter than you, that gradient descent is actually really great. The whole reason that we’re understanding these models is because we didn’t know how to write them in the first place. The gradient descent comes up with better solutions than us. And so I think that maybe another thing about Mechantorp is sort of having almost a kind of humility that we won’t guess a priori what’s going on inside the models. We have to have the sort of bottom up approach where we don’t really assume, you know, we don’t assume that we should look for a particular thing and that that will be there and that’s how It works. But instead, we look for the bottom-up and discover what happens to exist in these models and study them that way. (Time 4:25:08)
  • Superposition Hypothesis
    • The superposition hypothesis suggests neural networks represent many concepts by combining neuron activations.
    • This explains polysemanticity, where neurons respond to multiple unrelated things. Transcript: Chris Olah Can you describe what superposition is? Yeah. So earlier we were talking about word to back, right? And we were talking about how, you know, maybe you have one direction that corresponds to gender and maybe another that corresponds to royalty and another one that corresponds to Italy And another one that corresponds to, you know, food and all of these things. Well, you know, oftentimes maybe these word embeddings, they might be 500 dimensions, a thousand dimensions. And so if you believe that all of those directions were orthogonal, then you could only have, you know, 500 concepts. And, you know, I love pizza. But like, if I was going to go and like give the like 500 most important concepts in, you know, the English language, probably Italy wouldn’t be, it’s not obvious at least that Italy would Be one of them, right? Because you have to have things like plural and singular and verb and noun and adjective. And, you know, there’s a lot of things we have to get to before we get to Italy and Japan. And, you know, there’s a lot of countries in the world. And so how might it be that models could, you know, simultaneously have the linear representation hypothesis be true and also represent more things than they have directions. So what does that mean? Well, okay, so if linear representation hypothesis is true, something interesting has to be going on. Now, I’ll tell you one more interesting thing before we go and we do that, which is, you know, earlier we were talking about all these polysomatic neurons, right? These neurons that, you know, when we’re looking at Inception V1, there’s these nice neurons that like the car detector and the curve detector and so on that respond to lots of, you know, To to very coherent things. But it’s lots of neurons that respond to a bunch of unrelated things. And that’s also an interesting phenomenon. And it turns out as well that even these neurons that are really, really clean, if you look at the weak activations, right? So if you look at like, you know, the activations where it’s like activating 5% of the, you know, of the maximum activation, it’s really not the core thing that it’s expecting, right? So if you look at a curve detector, for instance, and you look at the places where it’s 5% active, you know, you could interpret it just as noise, or it could be that it’s doing something Else there. Okay, so how could that be? Well, there’s this amazing thing in mathematics compressed sensing. And it’s actually this very surprising fact where if you have a high dimensional space and you project it into a low dimensional space, ordinarily you can’t go and sort of unproject It and get back your high dimensional vector, right? You threw information away. This is like, you know, you can’t invert a rectangular matrix. You can only invert square matrices. But it turns out that that’s actually not quite true. If I tell you that the high dimensional vector was sparse, so it’s mostly zeros, then it turns out that you can often go and find back the high dimensional vector with very high probability. So that’s a surprising fact, right? It says that, you know, you can have this high-dimensional vector space, and as long as things are sparse, you can project it down, you can have a lower-dimensional projection of it, And that works. So the Suitsich hypothesis is saying that that’s what’s going on in neural networks. For instance, that’s what’s going on in word embeddings. The word embeddings are able to simultaneously have directions be the meaningful thing. And by exploiting the fact that they’re operating on a fairly high dimensional space, they’re actually, and the fact that these concepts are sparse, right? Like, you know, you usually aren’t talking about Japan and Italy at the same time. You know, most of those concepts, you know, in most sentences, Japan and Italy are both zero. They’re not present at all. And if that’s true, then you can go and have it be the case that you can have many more of these sort of directions that are meaningful, these features, than you have dimensions. And similarly, when we’re talking about neurons, you can have many more concepts than you have neurons.. So that’s the, at a high level of superstition hypothesis. Now it has this even wilder implication, which is to go and say that neural networks are, it may not just be the case that the representations are like this, but the computation may also Be like this, you know, the connections between all of them. And so in some sense, neural networks may be shadows of much larger, sparser neural networks. And what we see are these projections. And the strongest version of the superstition hypothesis would be to take that really seriously and sort of say, you know, there actually is in some sense this upstairs model, this, You know, where the neurons are really sparse and all-interpol, and there’s, you know, the weights between them are these really sparse circuits. And that’s what we’re studying. And the thing that we’re observing is the shadow of it, and so we need to find the original object. Lex Fridman And the process of learning is trying to construct a compression of the upstairs that doesn’t lose too much information in the projection. Chris Olah Yeah, it’s finding how to fit it efficiently or something like this. The gradient descent is doing this. And in fact, so this sort of says that gradient descent, you know, it could just represent a dense neural network, but it sort of says that gradient descent is implicitly searching over The space of extremely sparse models that could be projected into this low dimensional space. And this large body of work of people going and trying to study sparse neural networks, right? Where you go and you have, you could design neural networks, right? Where the edges are sparse and the activations are sparse. And, you know, my sense is that work has generally, it feels very principled, right? It makes so much sense. And yet that work hasn’t really panned out that well, is my impression broadly. And I think that a potential answer for that is that actually the neural network is already sparse in some sense. Gradient descent was the whole time gradient, you were trying to go and do this. Gradient descent was actually in the behind the scenes going and searching more efficiently than you could through the space of sparse models and going and learning whatever sparse Model was most efficient. (Time 4:47:26)
  • Features in Claude 3 Sonnet
    • Features extracted from Claude 3 Sonnet include security vulnerabilities and backdoors in code.
    • The backdoor feature, when activated, makes Claude write backdoors and is associated with hidden camera images. Transcript: Chris Olah Even these models seem to be substantially explained, at least, by linear features. And doing dictionary running on them works. And as you learn more features, you go and you explain more and more. So that’s, I think, quite a promising sign. And you find, now, really fascinating abstract features. And the features are also multimodal. They respond to images and text for the same concept, which is fun. Lex Fridman Yeah, can you explain that? I mean, like, you know, backdoor. There’s just a lot of examples that you can… Yeah, so maybe let’s start with one example to start, which is we found some features around sort of security vulnerabilities and backdoors in code. Chris Olah So it turns out those are actually two different features. So there’s a security vulnerability feature. And if you force it active, Claude will start to go and write security vulnerabilities like buffer overflows into code. And it also fires for all kinds of things. Like, you know, some of the top dataset examples for it were things like, you know, dash, dash, disable, you know, SSL or something like this, which are sort of obviously really, really Insecure. Lex Fridman So at this point, it’s kind of like, maybe it’s just because the examples are presented that way. It’s kind of like surface, a little bit more obvious examples, right? I guess the idea is that down the line, it might be able to detect more nuanced, like deception or bugs or that kind of stuff? Chris Olah Yeah, well, maybe I want to distinguish two things. So one is the complexity of the feature or the concept, right? And the other is the nuance of how subtle the examples we’re looking at, right? So when we show the top dataset examples, those are the most extreme examples that cause that feature to activate. And so it doesn’t mean that it doesn’t fire for more subtle things. So the insecure code feature, the stuff that it fires most strongly for are these really obvious, disable the security type things. But, you know, it also fires for, you know, buffer overflows and more subtle security vulnerabilities in code. You know, these features are all multimodal. So you could ask like, what images activate this feature? And it turns out that the security vulnerability feature activates for images of like people clicking on Chrome to like go this website, the SSL certificate might be wrong or something Like this. Another thing that’s very entertaining is there’s backdoors and code feature. You activate it, it goes on. Claude writes a backdoor that will go and dump your data to port or something. But you can ask, okay, what images activate the backdoor feature? It was devices with hidden cameras in them. So there’s a whole, apparently, genre of people going and selling devices that look innocuous that have hidden cameras and they have ads about how there’s a hidden camera in it. And I guess that is the physical version of a backdoor. And so it sort of shows you how abstract these concepts are, right? And I just thought that was, I’m sort of sad that there’s a whole market of people selling devices like that. But I was kind of delighted that that was the thing that it came up with as the top image examples for the feature. Yeah, (Time 5:07:34)