Skip to content

Podcast

OpenAI's Codex- This Model Is So Fast It Changes How You Code

AI & I

Source ↗ ← All highlights
  • Super Bowl Sparked Massive Interest
    • OpenAI ran a Super Bowl ad for Codex and saw a massive surge in app downloads and traffic immediately after it aired.
    • The team used the ad to inspire builders and show Codex as an inviting, powerful experience for many users. Transcript: Thibault Sottiaux Can you tell me about that yeah i think especially like in you know we can talk about last week right so like last week on monday we released the codex app uh immediately we saw like a ton Of downloads like more than a million downloads in the first week and then we knew that we were releasing like an extremely strong model, like, you know, 5.3 Codex on Thursday. That just made, I think this, it very visible that, you know, we’re here to, you know, put incredible experiences out there. We’re very committed to Codex. And like, also agents are really starting to work and be able to create these things, you know, even if you’re like a little bit less technical. I think like the app really showed that, you know, it’s like it’s much more inviting for people to just try it and like, you know, run multiple agents, you know, with our models being like Very, very good at sort of like allowing for multitasking and being reliable for long running, long running sessions. So it like allows you to create a lot more. So it just felt that, you know maybe we can inspire more people to build and then show that agents are here right it’s like it’s not it’s coming it’s going to be mainstream um you know why Don’t you try and like create something new and you know inspire people i felt like the right thing that we wanted to reinforce yeah what while we were designing and developing the app Andrew Ambrosino One of our internal mandates to ourselves the whole time was that we had to make something that we love to use and that we used for all of our work. And if we couldn’t do that, then we weren’t going to put this out. And this was back when we started. And I think that we surprised ourselves a lot with how fun it was. And especially as we started to build this app before we started to build agent skills. And then once we kind of paired them together, it became this really rich interactive experience where you could open the browser or you could connect to these various services. And so all of a sudden we started to feel this like really connected interactive experience and wanted to share, like, I kind of see the ad as like a love letter to builders, right? I have never seen a Linux CD in a Super Bowl ad. Thibault Sottiaux And so, you know, like that was really cool to watch. Dan Shipper What was the impact of the ad? Thibault Sottiaux We’re still to measure that. We’ll see like, you know, how it plays out over the long term. But we saw a giant surge of traffic, actually, like, remarkably, like, you know, very, very quickly after 4 p.m., like, PST when it aired, like, the surge and, like, our systems were, Like, under heavy load. So it felt kind of weird to me that, you know, people are watching the Super Bowl and then going and, like, you know, installing the app and they’re just trying it out right there and then. But it happened. And a lot of people reached out and saying they were really inspired by it and just wanted to build afterwards, which is what we’re aiming for. Dan Shipper Timmy back, I still want to talk a little bit about the strategic (Time 0:02:34)
  • A New GUI Class For Programming
    • The Codex app is a different GUI class from an IDE and can act as a daily command center for agents.
    • GUIs enable persistent panes, voice, images, and dynamic affordances that TUIs/IDEs struggle to provide. Transcript: Dan Shipper You tell me about the decision to invest in a GUI over a TUI? I feel like TUIs are so hot right now. And obviously, you have one for Codex already. And you could have said, okay, we’re going to double down and just make the terminal experience even better than it is now. And really invest in that versus, okay, we’re going to go, you know, I think, yeah, making a GUI is a little bit of like a counterintuitive or like, counter narrative thing to do. So tell me about that decision process. Thibault Sottiaux I think it wasn’t counterintuitive. It’s more maybe it’s not mainstream. And so we experiment with a lot of different approaches. Like I very much consider that we’re still in the experimentation phase. And, you know, we’re responsible primarily for two things. It’s like, you know, building the most powerful entity out there, you know, that’s capable of coding. And then, you know, increasingly, this will become like a multi-agent system, and it will become like more and more capable. And, you know, you will have to figure out like how to steer and supervise like its outcome and its behavior. You know, that’s like one thing that we’re building. And then we’re also building like, how do you even, you know, interact with this? It’s like, you know, what is the optimal way to have visibility into what this like very capable entity or like system of entities is doing? How do you steer them? How do you supervise them? And so we, we’re very much still experimenting with what that is. It’s like, you know, sure, you can do it in the TUI. It’s like, at some point it starts to feel like very limiting, you know, especially on like multimodal, like, you know, actually like the models can like draw little diagrams and generate Images and, you know, or, you know, you can talk over it, you know, using voice. Maybe you have like many of them going in parallel and sort of you start to lose track. So we felt like we needed to start experimenting with something else. And it is only when, you know, we started to become like super, super popular internally, we were like, we have to ship this externally. Like, this is kind of like, this has come to a point where it’s like too good to sort of like, just keep it to ourselves. I mean, that was like the journey that you went, you know, you were not building in the app. But although like, when did you start building in the app? That was actually like fairly quickly, like when the app was building itself. That was, yeah, that was pretty quickly. Andrew Ambrosino And yeah, because I was starting with the TUI and with the IDE extension. And I think that my goal personally was how can I get to fully building the app on the app as fast as possible, right? It’s like, it’s really easy when building this stuff to slip into the mode of like, oh, this will be good for somebody. Like somebody will love this, a certain type of, like, they will love this, right? So we really wanted to get quickly to like, I want to be able to build the app on the app. I want it to be able to run itself with skills. I want it to click around on the app that it spawned. And I want this to be like part of my workflow as soon as possible. And there, like, I still use the Tui sometimes when I want to fire something quick, but I think that like there is something about the flexibility of controlling UI and being able to have Some panes be persistent and others be ephemeral and be, you know, we shipped voice with the app so you can prompt with voice. (Time 0:09:42)
  • Ship What Your Team Uses
    • Build product experiences you and your team love and use internally before shipping externally.
    • Use that internal adoption as a signal that your app is ready for wider release. Transcript: Andrew Ambrosino And yeah, because I was starting with the TUI and with the IDE extension. And I think that my goal personally was how can I get to fully building the app on the app as fast as possible, right? It’s like, it’s really easy when building this stuff to slip into the mode of like, oh, this will be good for somebody. Like somebody will love this, a certain type of, like, they will love this, right? So we really wanted to get quickly to like, I want to be able to build the app on the app. I want it to be able to run itself with skills. I want it to click around on the app that it spawned. And I want this to be like part of my workflow as soon as possible. And there, like, I still use the Tui sometimes when I want to fire something quick, but I think that like there is something about the flexibility of controlling UI and being able to have Some panes be persistent and others be ephemeral and be, you know, we shipped voice with the app so you can prompt with voice. We have mermaid diagrams in the app. We have full image rendering. So all of those things, I think, are like the tip of the iceberg and what we want to do with (Time 0:11:49)
  • Automations Power Daily Workflows
    • The team builds automations that run on schedules or triggers to keep PRs mergeable and fix build issues automatically.
    • Andrew used the app to generate a custom children’s book combining script, image-gen, and PDF skills. Transcript: Thibault Sottiaux Andrew came up with automations. And I think that sort of like shifts the way that, you know, you’re thinking about these things when they can just like, sort of like hop it into background, you know, on a specific trigger At a specific time. And then, you know, just, you can sort of like program it yourself. Yeah. You’re using that a lot. Andrew Ambrosino There are a lot of things that I use the app for that are a little bit outside of just like coding features. I keep it to, I use it to keep my PRs mergeable with automations. And so it’ll resolve merge conflicts. It’ll keep them updated. It will fix like build issues so that basically like as soon as they’re ready to go like they’re ready to go there’s no like oh hey somebody merged a big thing and there’s a conflict now Dan Shipper Um so i do that so you said like so at what point is the is the automation trigger because i thought the automation triggers like at a certain time schedule but it sounds like there are other Andrew Ambrosino Triggers i didn’t know about um i yeah i we’re looking a lot of things I have it right now just on a time schedule and I use our github skill and um some internal skills for our ci and that that Dan Shipper Runs hourly or every two hours and kind of just cleans everything up I see so it’s like through all you know if there are any changes on main and it just looks through any prs and just like Make sure that they’re all up to date so that whenever you’re ready to go it’s never like that section that’s good i like that yeah it’s it’s actually really helpful it’s surprisingly Andrew Ambrosino Helpful um i have one that every day at like 9 a.m uh i get sent all of the contributions that have merged to the codex app over the last day and so it’ll do like a nice report of who merged what. And it will, I have a group bit by theme. So I can be like, all right, like three people worked on this part of the composer, two people worked on automations, like here’s what happened so that I can at least be like knowledgeable Of what’s happening because things, things get chaotic right before launch. Thibault Sottiaux One automation I have is I run it multiple times a day, and it’s like pick a random file and find and fix a subtle bug. And then it’s kind of funny because it actually does pick a random file. So it will run Python, like RAND, and then it will find a random file, and it will start from there. And so it’s like, every time it sort of like explores like a new one. Dan Shipper Has it caught anything? Thibault Sottiaux Oh, yeah, yeah. It’s like, we catch, like, it’s often latent bugs that, you know, are not triggering actually, like on the critical path, but you know, they’re actually bugs. And then, you know just it’s like trivial to fix it like merge it takes very little time and it’s a thing that you know I would have never found myself uh found like an issue in like constraint Dan Shipper Sampling like the other day yeah that’s really cool do you have other other automations that are worth sharing let’s see I feel like I have 60 that are running at all times. Andrew Ambrosino Some for testing and some for real. Some of the members on the team really like this one that looks at the PRs that you’ve done in past day or so and quietly cleans up any bugs you shipped. And kind of like looks at a few of the observability platforms to see and like tries to basically ship a fix before anyone’s noticed that you shipped a bug. That’s cool. Thibault Sottiaux It’s not coding related, which is like marketing research. It runs daily. And it’s just sort of like, it’s probably with like a specific skill to do like deep marketing research, which I’ve like sort of like tuned over time. And then that just goes and like searches the web on, you know, any sort of like new things that sort of like came up in terms of like how, you know, just like how users are like perceiving, Talking about codecs. And then I just received that little report. And it always makes for like an interesting read. Yeah. We can just go on. It’s like, these are just examples that we do rely on, like. You know, they run. Yeah. Dan Shipper Yeah. Do you have any particular skills that you guys like that are beyond the normal kind of, you know, I have a GitHub skill and that kind of stuff? Thibault Sottiaux I love Andrew’s yeet skill, which it just like takes like the change. And then, you know, does the commit, does the PR, writes like the, puts it in draft and publishes a PR with a PR title and body. Andrew Ambrosino Yeah, it’s very satisfying. Thibault Sottiaux Yeah, it just does everything. That one definitely makes people productive. Andrew Ambrosino What are the top-used ones for you? Image Gen is a cool one. Yeah. For both like silly automation purposes, like, hey, make me an image that characterizes my last day of work. Not my last day of work, my previous day of work. Yes, yes, yes, Andrew. I, you know, the Image gen skill was actually really cool for, I use the Codex app to make a book for my daughters. And so I had, I like, you know, put together this prompt for teaching it about like a script that I wanted written. So like 24 pages, here are my daughter’s ages. Here’s like where we’ve lived in the past. Like we were in Boston and moved to New York and then moved over here. And then I said, like, after that, we went through that, I agreed on the script and then we went through and I said like, all right, now it’s time to use the image gen skill. And it made like, it prompted for every page in the book based on the script, it prompted for the image and then it kind of put them all together and use the PDF skill to put together the book’s Pdf and then i printed it um and so we’ve got like a super custom book that you know i read to my kids and it’s it’s really cool it’s just this awesome thing when you can combine like the intelligence Thibault Sottiaux Of like the agent and then it’s like like works in a programmatic way like you know by using skills and then can just combine them in like novel ways. And like, yeah, I think that the PDF and image 10 one is like, it’s a common combo that we see. (Time 0:20:38)
  • Personality Shapes Model Behavior
    • The team prioritizes personality and efficiency in the model, offering pragmatic and friendly modes.
    • Modulating personality helps reduce literal, brittle instruction-following mistakes like copying typos into code. Transcript: Thibault Sottiaux And then also like what we now refer to as like personalities, like how supportive is it? And then we understand that not everybody has the same preferences there. Like the previous default was definitely like super blunt, like pragmatic personality. Now, we’ve also introduced like a more supportive, like friendly personality, and you can just pick between those. And I think for things that don’t have like sort of like a universal, like, um, accepted, you know, thing that, you know, everybody that, you know, should just use is like, you know, we’re Probably going to introduce like some way for you to just make it your own, right. You know, you should feel like you have your own little personal products, um, that, you know, works in exactly the way that you want it to work. Um, do you use the friendly or the pragmatic one? Dan Shipper Pragmatic yeah okay i’ll say it’s pragmatic yeah um interesting i think um you guys recently put out a model that is so fucking fast i was testing it before it came out and i was just like Uh i can’t really keep up with this thing. So I’m curious how that changes how you think about what is now possible with coding with a model like this and also the affordances that you need in order to manage models that are so quick Effectively. Andrew Ambrosino Yeah. The first time we used this model in the app, we had kind of that same thing happen where all of a sudden, there was just like this wall (Time 0:27:54)
  • Speed Changes The Coding Experience
    • Extremely low latency changes how people code by keeping them in flow and enabling real-time interaction.
    • The team expects to further speed the model 2–3x and layer optimizations across infra and serving. Transcript: Thibault Sottiaux The first time I showed the very first prototype, when we hooked everything up and obviously the model is powered by Cerebrus. And it’s like, you know, we’ve talked about the partnership there. And like, we’re very excited to put like, you know, the first model that we’re serving through that, you know, out there. Like it’s, you know, obviously like still like very early. It’s literally the first time we hook it all up and we’re just like so excited that we want to share it. But the first time I showed it to someone, they were like, no way, this is like a fake, a fake demo. It’s like, you know, this is not real. Like this can not be this fast. And then they tried like a few prompts. They were just like, it’s just like, oh, I just, I literally cannot keep up. It’s like, this is insane. And yeah, I think this will change everything, especially because it’s not yet the fastest that we can actually get it to be. Like with the preview, we’re putting it out like, you know, quite early. We’re actually going to layer a number of optimizations on top of it, which should be able to like make it, you know, maybe two to three X faster than the experience that you have experienced. (Time 0:30:06)
  • Tune Infra For Low Latency
    • Optimize the entire serving stack (persistent connections, incremental state) when reducing model latency.
    • Apply these infra changes across models to lower overall turn latency by 30–40%. Transcript: Thibault Sottiaux And that decreases like the overall latency, like, you know, across all models. Like we haven’t shipped it by default yet, but it’s, you know, it is something that, you know, we are making the default for this new, like super fast model. (Time 0:32:40)
  • Verification Becomes The New Bottleneck
    • With generation no longer the slow step, verification and review become the bottleneck.
    • Teams must invent faster verification (screenshots, click-path evidence) and review tools to keep up. Transcript: Thibault Sottiaux The bottleneck that is very apparent is how fast can you verify that things are correct? So we can generate code faster than ever before. We can implement entire features. And, you know, we, like, I saw, like, someone just based on a description of, you know, the Codex app, if you sort of like synthesize that into a plan, just based on screenshots, like, The models are very much capable of, like, reproducing 95% of the features and just rebuilding the app from scratch. Now, is it going to be bug free? Is it going to, you know, is everything like implemented to like, you know, perfection in the same way that, you know, the actual app is? It’s like, that takes like a lot of time still, like, you know, for like a human to go and click and verify and, you know, make sure that, you know, it’s like, the designs are like consistent That, you know, there’s like no bugs here or there, that the settings panel, like, you know, when you click that button, it actually does the thing that you expect. I think verification definitely becomes a bottleneck. We have people on the team complain, there’s too much code to review. That’s what we’re trying to solve for. Andrew Ambrosino That there’s so much code to review now, both on your own machine and from another peer. Thibault Sottiaux We’re going to have to figure that out. Yeah, you’re already reviewing the code the first time because the agent is just presenting it to you. And then you have to review the code produced by your peers. (Time 0:36:41)
  • Attach Evidence To Every PR
    • Attach automated evidence to PRs (screenshots, reproduced click paths) to demonstrate end-to-end behavior.
    • Use these artifacts to reduce reliance on manual code review and prove fixes worked. Transcript: Andrew Ambrosino It’s so easy to code things now, right? Yeah, I mean, we have gotten the Codex app to be pretty good at… Through some skills that we have of running itself, clicking around, screenshotting itself for evidence and uploading it to the PR. There’s like a lot that’s pretty interesting there, especially when we make this like more async or when the models get really fast at this stuff. I don’t know exactly what it looks like yet, but there is a lot there around like, hey, here’s a bug fix. This is exactly like what it looked like when it was happening. And here’s exactly what it looks like now with the same exact click path. And so like, maybe that’s the turning point that code review becomes less important when it’s like you can verify that part instead. So you have to kind of like do less through the code as a proxy. (Time 0:40:19)
  • Team Adopted Codex Internally Fast
    • Internal adoption at OpenAI happened quickly: research teams added dev boxes and used Codex as it became usable.
    • The Codex team used the app themselves to train and iterate models like 5.3 Codex. Transcript: Thibault Sottiaux One of the examples was we had a lot of complaints on compaction. You know, compaction was like something that people felt like whenever you would hit compaction, you know, people would complain. It’s like, it’s losing too much context. And so we sort of like solve that end to end. And like, you know, we decided to do like, end to end RL training and, you know, introduce compaction, like within research, and then, you know, make the model like, you know, itself, Like, you know, very familiar with the concept of compaction and like producing like optimal, like sort of like delegating to itself, like across time. And, you know, once we had that, and we had solved it at the model level, like sort of like the harness problem became like so much easier, because it was just like, oh, just let the model Do it. And it’s going to be like very reliable. So through that, and like through that collaboration, it just felt like, like the momentum has been like very strong. And I were so like able to improve like models and like, you know, ship a model like roughly a monthly cadence. And then we took a bit of a different bet and a different approach with the Codex app, which turned out to be an awesome thing to just try and do. Not just force ourselves and trying to cram everything into the TUI. I mean, it was a great challenge, right? You know, you were like, I’m, you know, it’s like, let’s build an app. Like, just like, where do I get started? And then, you know, just like, you just got obsessed by it. It’s hard not to. Yeah. I mean, it’s like, how was it to just like, you know, build something that was quite contrarian, I suppose. Andrew Ambrosino Yeah. I mean, I remember you and I talking about whether or not like early on we were like we don’t know if we’ll ship this yeah like we’ll we’ll try it out we’ll see if we can get there with something That we love and see if we can get i remember saying like let’s get some pmf internally let’s let’s get everybody at open ai to want to use this thing without being forced to use it. Let’s see if we can do it, right? We did. And it was adopted very quickly. I mean, the minute it was barely usable, the research folks put dev boxes on it, right? Which was like this crazy hack at the time. Thibault Sottiaux Yes, yes. But now they use it for everything. Yeah. Yeah, it was including in training 5.3 Codex. And so I think I feel really good about having hit the point where everyone technical at the company, almost everyone at the technical at the company uses Codex. But the people who use it the most are actually building Codex and building the models. (Time 0:43:03)