Podcast

OpenAI Researcher on Why Soft Skills Are the Future of Work | Karina Nguyen

Career | Growth

Last highlighted February 16, 2025 Source ↗ ← All highlights

Model Training as Art
- Model training is more of an art than a science, focusing heavily on data quality and debugging similar to software.
- Models learn by compressing knowledge and modeling the world, not just predicting the next word. Transcript: Lenny Rachitsky It. Okay. I’m going to definitely ask you about the differences between those, but I want to start more technical and just dive right in. I want to talk about model training. People always hear about models being trained, these big models, how much data takes, how long it takes, how much money it takes, how we’re running out of data, which I want to talk about. Let me just ask you this question. What do you think people most misunderstand about how models are created? Karina Nguyen Model training is more an art than a science. And in a lot of ways, we as model trainers think a lot about data quality. It’s one of the most important things in model training. It’s like, how do you ensure the highest quality data for certain interaction model behavior that you want to create? But the way you debug models is actually very similar the way you would debug software. So one of the things that I’ve learned early days at Anthropik was we’ve discovered, especially with like cloud training, when you taught the model some of the self-knowledge of like, Hey, like you actually don’t have a physical body to operate like in the physical world. But then at the same time, we had data that kind of taught the model some of the function calls, which is like, this is how you set the alarm. And so the model would get extremely confused about whether it can set an alarm, but it doesn’t have a body in the physical world. So it’s like the model gets confused and sometimes it’s like over refused. So sometimes it’s like, I don’t know, like, sorry, I cannot help you. And so there is always like a balanced trade off between how do you make the model to be more helpful for users, but also not being harmful in other scenarios. So it’s always about like, how do you make the model like more robust and like operate across like variety of diverse scenarios. Lenny Rachitsky That is so funny. I never thought about that. Most of the data that’s trained on is kind of like assuming it’s like a human describing the world and how they operate. And there’s, it assumes there’s a body and you could do things and the model told you don’t have a body. Yeah. Okay. I want to talk a little bit about data while we’re on this topic. I know you have strong opinions here. There’s kind of this meme that models are going to stop getting smarter because they’re running out of data. They’re trained in a large part on the internet, and there’s only one internet, and they’ve already been trained on it. What more can you show them about the world? And there’s this trend of synthetic data, this term synthetic data. (Time 0:06:11)
Models’ Physical Body Confusion
- Karina Nguyen discovered that models get confused when taught they lack a physical body while also learning physical world actions.
- This confusion arises from conflicting data, like instructions on setting alarms despite not having a body to do so. Transcript: Karina Nguyen But the way you debug models is actually very similar the way you would debug software. So one of the things that I’ve learned early days at Anthropik was we’ve discovered, especially with like cloud training, when you taught the model some of the self-knowledge of like, Hey, like you actually don’t have a physical body to operate like in the physical world. But then at the same time, we had data that kind of taught the model some of the function calls, which is like, this is how you set the alarm. And so the model would get extremely confused about whether it can set an alarm, but it doesn’t have a body in the physical world. So it’s like the model gets confused and sometimes it’s like over refused. So sometimes it’s like, I don’t know, like, sorry, I cannot help you. And so there is always like a balanced trade off between how do you make the model to be more helpful for users, but also not being harmful in other scenarios. So it’s always about like, how do you make the model like more robust and like operate across like variety of diverse scenarios. Lenny Rachitsky That is so funny. I never thought about that. Most of the data that’s trained on is kind of like assuming it’s like a human describing the world and how they operate. And there’s, it assumes there’s a body and you could do things and the model told you don’t have a body. (Time 0:06:59)
Beyond the Data Wall
- The perceived data wall for large language models is a misconception, as post-training with reinforcement learning uses infinite tasks.
- The bottleneck lies in evaluation metrics, not data, as models saturate existing benchmarks. Transcript: Lenny Rachitsky Yeah. Okay. I want to talk a little bit about data while we’re on this topic. I know you have strong opinions here. There’s kind of this meme that models are going to stop getting smarter because they’re running out of data. They’re trained in a large part on the internet, and there’s only one internet, and they’ve already been trained on it. What more can you show them about the world? And there’s this trend of synthetic data, this term synthetic data. What is synthetic data? Why do you think this is important? Do you think it’s going to work? Karina Nguyen I think there are two questions here. We can unpack one at a time. But people say we’re hitting the data wall. I think people think more in the terms of like pre-trained large models that are trained on the entire internet to predict the next token. But what actually the model is learning during that process is actually, how do you compress the compression algorithm here? The model learns to compress a lot of knowledge and it learns how to model the world. So the next prediction of the word, like, teach me how to drive, basically, and you only have like a few words that will match that a car so the model actually learns um about the world in Itself so it’s like it’s modeling human behavior sometimes it’s modeling and when you talk to like pre-tuned models which are very very large they’re actually extremely diverse and Extremely creative because you can talk to almost any Reddit user through Pre-Chain Model. But I think what’s happening right now with new paradigm of L1 series is that the scaling in post-chaining itself is not hitting the wall. And that’s because basically we went from like raw data sets from pre-trained models to infinite amount of tasks that you can teach the model in the post-training world via reinforcement Learning. So any task, for example, like how to search the web, how to use the computer, how to write, well, like all sorts of tasks that you like trying to teach the model, all the different skills. And that’s why we’re saying like there’s no data wall or whatever, because there will be infinite amount of tasks. And that’s how the model becomes extremely super diligent. And we are actually getting saturated in all benchmarks. So I think the bottleneck is actually in evaluations. That we don’t have all the frontier, like evals, like, I don’t know, GPQA, which is, like, a Google-proof question answering, like, PhD-level. And that’s just a better work is like getting to like, I don’t know, more than like 60, 70%, which is what PhD gets. So it’s like literally hitting the wall in like evals. (Time 0:08:20)
Building Canvas with Synthetic Data
- Karina Nguyen’s team at OpenAI uses synthetic data training to rapidly iterate on product features like Canvas.
- They focus on core behaviors, such as triggering Canvas for long documents or code, teaching edits, and generating comments. Transcript: Lenny Rachitsky Actually get into that. That’s really interesting. I want to talk about evals, but let’s follow that thread. So talk about how this helped you create Canvas. Karina Nguyen So when I first came to OpenAI, I really had this idea of like, okay, like it would be really cool for Chattapiti to actually like change the visual interface, but also change the way it Is with people. So going from being a chatbot to more of a collaborative agent and a collaborator is a step towards more agent existence that become innovators, ultimately. And so the entire team of applied engineers, designers, products, research, kind of got formed in the air, almost out of nothing. It’s just a collection of people who just got together, and we rapidly started iterating with each other. Actually, Canvas is one of the… I would say the first project at OpenAI where researchers and applied engineers started working together from the very beginning of the product development cycle. And I think there’s a lot of things that we have learned on the way, but I definitely came with the mindset of we need to do a really rapid model iteration such that, like, it would be much Easier for engineers to, you know, work with the latest model possible, but also learn from, like, user feedback or, like, early, like, internal dog food, how do we improve the model Very rapidly. And, you know, it’s really hard to, like, kind of, like, figure out, like, how people, when you deploy a product, how people would be able to, like, use it. And so, like, the way you synthetically train the model is basically figuring out, like, what are the most core behaviors that you want this product feature to do? And for Canvas, for example, it was, it came down to like three main behaviors. It was, how do you trigger Canvas for prompts like, write me a long essay, when the user intention is mostly like, iterating over long documents, or write me a piece of code. Or when to not trigger Canvas for prompts like, can you tell me more about precedent? I don’t know. Some of the general questions. So you don’t want to let trigger Canvas because the user intention is mostly getting answered and not necessarily iterate over the long document. The second behavior is how do we teach the model to update the document when the user asks? So one of the behaviors that we taught the model is actually have some agency and autonomy to literally go to the document and select specific sections and either delete it or edit. So highlight it and rewrite certain sections. So sometimes the model, sometimes the user would just say, change the second paragraph to be something friendlier. And you would have to teach the model to literally find the second paragraph in the document and change it to a friendly tool. So basically you teach both like how to trigger like edit itself, but also how do you teach the model to get higher quality edit for the document. In case of like coding, for example, there’s also like the question of like how good the model is at like completely rewriting the document versus like having like specific targeted Edits. So that’s another layer of decision boundary within edit itself. It’s like select the entire document and rewrite completely, or you want to have very targeted custom behavior. And when we first launched the model, we would bias the model towards more rewrites because we thought the quality of the rewrites were much higher. But over time, you’re kind of shifting based on user feedback and what you’re learning from iterative deployment. Lastly, the third behavior that we taught genetically, the model is how to make comments on any document. So the way we used it is like, we would use a one model to produce, to like simulate, like use a conversation. Let’s say like, write me a document about XYZ. But then we used a one to like produce the document. And then we kind of injected like user prompt to be like, oh, make some comments, critique my piece of writing or critique this piece of writing that you just made. And then we taught the model to make comments on the document on very specific document. Also, what kind of comments you want the model to make? Do they make sense or not? Like, how do you teach the quality of that? And it all came down to like measuring progress via very robust evals. But yeah, this is how you like used like a one, like kind of like synthetic data generation for like the screening. (Time 0:12:49)
Prototyping with Prompts
- Product development is shifting towards AI-driven prototyping, where product managers define what “correct” looks like via evaluations.
- Karina Nguyen suggests prompting models for rapid prototyping, citing her work on personalized starter prompts and title generation. Transcript: Lenny Rachitsky Interesting. Like basically what I’m hearing and there’s something I’m learning about as I talk to people is product development might move from this, like, here’s a spec PRD, let’s build it together. And then cool, let’s review it. Are we happy with this to from that to, hey, AI, build this thing for me. And here’s what correct looks like. And I’m spending all my time on what is correct look like. Karina Nguyen And evals, essentially, you definitely want to like, measure progress. It’s the model. And this is where evals is because like you can have prompted model as a baseline already and the most robust evals is the one where prompted baselines get the lowest score or something And then because then you know like if you’re trained a good model then it should like just like hill climb on that eval all the time while not like also like regressing on like other intelligence Evals so it’s like i think it’s more what that’s that’s what i’m saying like it’s more than science it’s like okay like if you optimize the model for this behavior like you kind of don’t Want to like brain damage in like other areas of intelligence or and this is happening like all the time in every lab and every like research team um i would say like prompting is like also A way to like prototype like new product ideas like early days at andara glenay was working on like file uploads feature i remember i was just like you know prompting the model um to just Like I mean when we were like launching like 100 key contexts I was just like prototyping this in my local person which I did the demo like people really really loved it and they just like Wanted like API for like file uploads or something and then that’s when it clicked to me. I also wrote a blog post on February. It clicked to me that prompting is a new way of product development or prototyping for designers and for product managers. For example, one of the features that I wanted to do is have personalized starter prompts. So whenever you come to like Cloud, like it should like recommend you like starter prompts based on what your interests are. And so like you can literally do it like prompting for that. To experiment with that. Another feature was like generating titles for the conversations. It’s a very small like micro-hperience, but I’m really proud of. The way we did that was we took five latest conversations from the user and asked the model, what’s the style of the user? And then for the next new conversation, the generated title will be of the same style. It’s just really little micro-experiences like this. (Time 0:23:22)
Origin of Canvas and Tasks
- OpenAI’s Canvas and Tasks features began with individual prototypes and specs for model behavior.
- Despite not being groundbreaking, they leverage familiar interfaces like Google Docs and notifications, enhancing them with AI. Transcript: Lenny Rachitsky Okay. I want to spend a little more time on how you operate. So you talked about, you built this and launched this tasks feature. Is that the way you describe your tasks? Yeah. So talk about how that emerged and let’s better understand just how you collaborate with product teams and how OpenAI works in that way whatever you can share there I think canvas and Karina Nguyen Tasks are uh going into the bucket of projects where it’s like more like short or like medium terms and um actually the way canvas and tasks came about to be was like it started with like One person prototyping and creating like works a spec it’s kind of like prd is like creating a spec of like the behavior of the model i don’t think like tasks is like extremely like groundbreaking Groundbreaking feature necessarily. What makes it really cool is because the models are so general, models can now search. They can write sci-fi stories. They can search for stocks. They can summarize the news every day. Because the models are so general, giving something familiar to people, that notification is very familiar, having reminders is very familiar. So creating a form factor for the people who are very familiar, same as Canvas, Google Docs is very familiar. But then you add magical AI moment and it becomes very powerful. But the way it comes, usually operationally like yeah it says it’s like a prototype like literally prompted prototype of like how you would want like the model to behave for like tasks For example like you kind of like need to design a little bit like design design systems design thinking is like okay like well if the user says like um remind me to go to lunch, like, at 8 a.m. Tomorrow, okay, what kind of information does the model need to extract from that prompt in order to create a reminder? And so this is how you, like, design, like, a spec for a new feature, like, a tool. Canvas and tasks are all tools. So it’s, like, how do you, like, create the tool spec? And then it’s, like, like, like, developing JSON schema. I was like, okay, like, from this problem, maybe the model should extract like, the time that the user requested. And then you’re thinking about like, which format do you want the time to be? And then like, how do you want the model to (Time 0:26:56)
From Engineer to Researcher
- Karina Nguyen switched from engineering to research after realizing AI’s growing coding proficiency.
- She predicts AI will unblock tasks bottlenecked by intelligence, democratizing access to expertise in fields like healthcare. Transcript: Lenny Rachitsky And so I’m very curious to hear just your take on how you think things are going to change in the world and how people work based on where you see things are going. And I know it’s a broad question, but let’s say like in the next three years, how do you see the world changing? How do you see people’s way of working changing? Karina Nguyen It’s a very humbling experience to be in both labs, I guess. To me, when I first came to Endorbit, I was like, oh, no, I really love firm-end engineering. And then the reason why I switched to research is because I realized at that time, it’s like, oh, my God, cloud is getting better at front-end. Cloud is getting better at coding. I think cloud can develop new apps or something. And so it can develop new features for the thing that I’m working. So it was kind of like this meta-realization where it’s like, oh my god, the world is actually changing. And when we first launched 100k context at that time, obviously, I’m thinking about form factors that’s like yeah like file uploads were like very natural very familiar to people but You can imagine we could just like make like infinite chats in the cloud.ai app right like as if like it’s like in a 100k context but because like file uploads it’s like foreign follows Function it’s like the form factor the file uploads kind of enable people to just like literally upload anything the books are like any reports financial and like ask any task to the Model and then i remember it was like you know enterprise customers like um like financial customers are like really interested in that. It’s like, oh, wow, like actually they, it’s actually one of the very common tasks that people do in that setting. It was like kind of crazy to like see how some of the redundant tasks are getting like automated basically by these like smart models. And they’re entering the era where I actually don’t know, for example, sometimes if L1 gives me the correct answer or not because I’m not an expert in that field. And it’s like, I don’t even know how to verify the outputs of the models because only experts know, can like verify this. So yes so basically there are trends that are going on. The first trend is the cost of reasoning and intelligence is drastically going down. I had a blog post about this. Maybe I should update them like latest benchmarks because at that time like MMO everybody was like doing like um it’s not like like one benchmark and they’d be like quickly saturated The benchmarks and like now we need to like do the same plot but with another like frontier eval but the cost of intelligence is like going down because it’s it becomes like much cheaper Smart small models are becoming even smarter than like large models and that’s because of like the distillation research this happened with like klotty haiku i was like working like Pristine like klotty haiku and i realized it was much smarter than like klotty which was like way you know bigger let’s not like that um but like the power of like small models become very Intelligent and fast and cheap we are moving towards that world that has like multiple implications but that means that like people will have more access to ai and that’s really good Like builders and developers will have much better access to ai but also it means like all the work that has been like bottlenecked by the intelligence will be kind of like unblocked So anyone like i’m thinking about like healthcare right like if i have instead of going to a doctor i can like ask chai gpt or give chai a list of symptoms and ask me like, which like, would I have like a cold flu or like something else? Like I can literally get the access to like a doctor almost. And there’s like been some like research studies around that. (Time 0:35:48)
Building for the AI Future
- Karina Nguyen emphasizes creative thinking and building products immune to general model replacement.
- She advocates for rapid iteration, user feedback integration, and exploring novel interaction paradigms with AI. Transcript: Lenny Rachitsky Is awesome. Okay. I want to follow this thread for sure. And it’s funny that what you described is like, you were an engineer at Anthropic and you’re like, okay, Claude is going to be very good at engineering. This isn’t going to be a potentially career long-term. So I’m going to move into research and AI is going to need me for a long time to build it to make it smarter. Karina Nguyen I would say we still have like I think Canvas team has still have like really cool like front engineers that are really like you know people who like really care about like interaction Design like interacting experience like I don’t think like models are there yet like I think if but we can get the models to like this top one percent of like front end or something um for Lenny Rachitsky Sure so what i want to move on to next along these lines is just and this is just speculation but uh what skills do you think will be most valuable going forward for product teams in particular So folks are listening and they’re like, okay, this is scary. What should I be building now to help me stay ahead and not be in trouble down the road? What skills do you think are going to be most, more and more important to build? Karina Nguyen Yeah, I think like creative thinking, like you kind of want to like come up, like generate a bunch of ideas and like filter through them and not just like build the best product experience. Listening, you know, you want to like build something that like the most general model will not replace you. And oftentimes you, you build something and you make it really, really good like specific set of users and actually the mode is now in like your user feedback the mode is like more in like You whether you listen to them like whether you you can like rapidly iterate like the mode is like in here i don’t think like we are yet to like there are so many ideas i think there’s an abundance Of ideas that you can look around. I wouldn’t be worried. I feel like, in fact, I just think people in AI fields are like… I wish they were a little bit more creative and connecting dots across different fields or something like that to develop really cool new like generation and new paradigms of interactions With this ai like i don’t think we’ve cracked this problem at all um a couple years ago i was like telling some people i was like you know you kind of want to like build for the future so it’s Like it doesn’t necessarily matter whether the model is good or not good right now but you can build product ideas such that like by the time the models will be really good it will work Really well um and i think it just like happened naturally like for example like i don’t know like right like the cloud artifacts and i feel like early days of canvas was like back in like 2022 like before chachapiti like writing id was like i know chachapiti but i feel like cloud 1.3 model itself was like not there to like made like really extreme good like high quality Edits for example like coding um and i feel like i see like startups likecer and it’s doing super well. And that’s because they iterate so fast. They invent new ways of training models. They move really fast. They listen to users. (Time 0:41:34)
The Rise of Soft Skills
- Karina Nguyen believes soft skills like creative thinking, management, and empathy will become increasingly important in product development.
- AI struggles with aesthetics, visual design, creative writing, and emotional intelligence, making these human skills valuable. Transcript: Lenny Rachitsky Really helpful, actually. So what I’m hearing is that soft skills essentially are going to be more and more important and powerful. You talked about management, leading people, being creative and coming up with innovative insights, listening. There’s a post I wrote that I’ll link to where I try to analyze how AI will impact product management. And we’re actually very aligned. And my sense was the same thing, that soft skills are going to become more and more important. And the things that are going to be replaced is the hard skills, which is interesting because usually people value the hard skills like coding, design, writing really well. And it’s interesting that AI is actually really good at that because it’s taking a bunch of data, synthesizing it and writing, creating a versus all these fuzzy things around of what Influences convinces people to do things and aligning and listening like you said creativity anything along those lines come up as i say that i think it’s actually really really hard Karina Nguyen To teach the model how to be aesthetic or like uh do like visual really good like visual design or like how to be extremely creative in the way to write. I think like, I still think like Chai GP kind of sucks at like writing. And that’s because it’s like, it’s like bottlenecked by this like creative reasoning. I think like prioritization is like one of the most important, like I think like, um, for a manager, I feel like I actually like AI research progress is bottlenecked by, like, management, Like, research management, is because you have, like, constrained set of compute, and you need to, like, allocate the compute to the research path that you feel the most convinced About. It was, like, you need to, like, really, you need to have, like, a really high conviction in the research path to put the compute and like it’s more like return on investment um kind of Situation as like okay yeah like i’m thinking a lot about like like okay like how to across all my projects which projects are higher priorities like prioritization and also like on The lower level it’s like which experiments are really important to run right now and which are not and like cut through the line. So I was thinking like prioritization, communication, like management, people skills, like empathy, like understanding people, like kind of like collaboration. I think like Canvas wouldn’t be like an amazing launch if it wasn’t like about like people. And I think it’s wonderful group of people and like i get a chance to like work with like people like lee byron who’s like a co-creator like graphql and like some of the best like apple designers And it’s like so cool to like see and like how do you create this like collaboration between people it’s just like something that’s still humane i think let (Time 0:45:08)
Anthropic vs. OpenAI
- Karina Nguyen observes that Anthropic prioritizes careful model behavior and crafting, while OpenAI fosters innovation and risk-taking.
- These cultural nuances influence the respective models’ personalities and product development approaches. Transcript: Lenny Rachitsky Holy moly. Okay, so speaking of Anthropic and OpenAI, you’ve worked at both. Very few people have worked at both companies and seen how they operate. I’m curious just what you’ve noticed about the differences between these two, how they operate, how they think, how they approach stuff. What can you share along those lines? Karina Nguyen It’s more similar than different uh obviously there is a lot of like there are some like differences also comes to like nuances uh to culture i really love anthropic and i have a lot of Friends there and i also love open ai and i still have a lot of friends though so it’s like it’s not about like enemies i feel like there’s like in the eye it was all like yeah the competitors Those like enemies this is actually like one big community and like of people like doing the same thing i would say what i’ve learned from on topic is this like real care and craft towards Like model behavior model class, model training. And I’ve been thinking a lot about like, okay, like what makes Cloud Cloud and what makes Chachapi Chachapi. And it’s like, I tell you, it comes down to like operational processes that kind of leads to the outputs to the model, is the output and model. And it’s like the reason why Claude has so much more personality and is more like a librarian, I don’t know, I’m visualizing Claude being like a librarian, very nerdy or something, is Because I feel like it’s a reflection of the creators who are making this model and a lot of details around the character and the personality and whether the model should follow up on This question or not. Like, what’s the correct ethical behavior for the model in this scenario? It’s like a lot of craft and created data sets. And this is where I learned that part of like art i guess at antarctic i would say like antarctic is like much smaller like when i joined it was like what like 70 people when i left it was 700 People and like obviously the culture changed so much i really enjoyed being like early days startup like wives and like people knew each other as a family but like the culture shifted I would say like under i learned from andarvik that like they’re much better at like focusing and like prioritization or like very very hard like very hardcore prioritization i guess And they need to do it like but i think like open ai is like much more innovative and much more like risk takers in terms of like product or like research actually you know like I don’t know You can like your full-time job can be just like teaching the model how to be like creative writers and it’s like there’s some luxury in this like research freedom that that comes to scale Maybe I don’t um but it gives you it’s like you’ll have i feel like i have much more creative like product freedom to do almost anything i guess within like open ai like i’ve lost chatty Into like the version that you want it’s like more like yeah probably bottoms up i guess yeah (Time 0:53:33)