Skip to content

Podcast

Extreme Harness Engineering for Token Billionaires- 1M LOC, 1B Toks/Day, 0% Human Code, 0% Human Review — Ryan Lopopolo, OpenAI Frontier & Symphony

Latent Space: The AI Engineer Podcast

Source ↗ ← All highlights
  • Zero Human Code Started Painfully Slow
    • Ryan Lopopolo started with early Codex CLI and deliberately refused to write code himself, forcing the agent to do the job end to end.
    • The first month and a half was 10x slower, but decomposition plus better primitives turned it into a system faster than any one engineer. Transcript: Alessio Fanelli Like just a bit of background before that, this is basically the article. So what you guys did is five months of working on an internal tool, zero lines of code, over a million lines of code in the total code base. You say it was Cenex faster than you would have if you had done it by end. So that was the mindset going into this, right? That’s right. Ryan Lopopolo Started with some of the very first versions of Codex CLI with the Codex Mini model, which was obviously much less capable than the ones we have today, which was also a very good constraint, Right? It’s quite a visceral feeling to ask the model to build you a product feature and it just not being able to assemble the pieces together, which kind of defined one of the mindsets we had For going into this, which is whenever the model just cannot, you always pop open the task, double-click into it, and build smaller building blocks that then you can reassemble into The broader objective. And it was quite painful to do this, honestly. The first month and a half was 10 times slower than I would be. But because we paid that cost, we ended up getting to something much more productive than any one engineer could be. Because we built the tools, the assembly station for the agent to do the whole thing. But yeah, so onward to GPT-5, 5.1, 5.2, 5.3, 5.4, to go through all these model generations and see their kind of quirks and different working styles also meant we had to adapt the code Base to change things up when the model was revved. Codex Harness at the time did not have background shells in it, which means we were able to rely on blocking scripts to perform long horizon work. But with 5.3 and background shells, it became less patient, less willing to block. So we had to retool the entire build system to complete in under a minute. (Time 0:03:25)
  • Keep Agent Build Loops Under One Minute
    • Keep the inner build loop under a minute and treat any overrun as a signal to stop and decompose the build graph.
    • Ryan Lopopolo retooled from Make to Bazel to Turbo to NX in a week because fast builds mattered more than human preferences. Transcript: Shawn ‘swyx’ Wang Guess, and just to really nail this, like, what does one minute matter? Like, why not five? Okay. Ryan Lopopolo We want the inner loop to be as fast as possible. One minute was just a nice round number and we were able to hit it. And if it doesn’t complete, it kills it or something? No, we just take that as a signal that we need to stop what we’re doing, double click, decompose the build graph a bit to get the time back under so that we can enable the agent to continue Shawn ‘swyx’ Wang To operate. It’s almost like you’re, it’s like a ratchet. It’s like you’re forcing build time discipline because if you don’t, it’ll just grow and grow. That’s right. And you mentioned that. Ryan Lopopolo The software I work on currently is at 12 minutes. It sucks. This has been my experience with platform teams in the past, where you have an envelope of acceptable build times, and you let it go up to breach, and then you spend two, three weeks to Bring it back down to the lower end and be able to load and stop. But because tokens are so cheap and so insanely parallel with the model, we can just constantly be gardening this thing to make sure that we maintain these invariants, which means there’s Way less dispersion in the code and the SDLC, which means we can simplify in a way and rely on a lot more invariants as we write the software. (Time 0:06:38)
  • Human Attention Became The Real Bottleneck
    • Human attention became the scarce resource, not tokens, so the team stopped centering review around reading every PR.
    • They shifted effort into observability, modular primitives, and confidence-building automation so agents could review, fix, and merge autonomously. Transcript: Alessio Fanelli Mentioned in your article, like humans became the bottleneck, right? You kicked off as a team of three people. You’re putting out a million line of code, like 1500 PRs. Basically, what’s the mindset there? So as much as code is disposable, you’re doing a lot of review. A lot of the article talks about how you want to rephrase. Everything is prompting. Everything is what the agent can’t see. It’s kind of garbage, right? You shouldn’t have it in there. So what’s like the high level of how you went about building it and then how you address okay humans are just pr review like how is human in the loop for this we’ve moved beyond even the humans Ryan Lopopolo Reviewing the code as well most of the human review is post merge at this point but it’s not even review that’s just oh let’s just make ourselves happy by reading it. Fundamentally, the model is trivially paralyzable, right? As many GPUs and tokens as I am willing to spend, I can have capacity to work with a code base. The only fundamentally scarce thing is the synchronous human attention of my team. There’s only so many hours in the day. We have to eat lunch. I would like to sleep, although it’s quite difficult to stop poking the machine because it makes me want to feed it. You have to step back, right? Like you need to take a systems thinking mindset to things and constantly be asking, where is the agent making mistakes? Where am I spending my time? How can I not spend that time going forward and then build confidence in the automation that I’m putting in place so I have solved this part of the SDLC? And usually what that has looked like is like we started needing to pay very close attention to the code because the agent did not have the right building blocks to produce modular software That decomposed appropriately, that was reliable and observable, and actually accrued a working front end in these things, right? So in order to not spend all of our time sitting in front of a terminal, at most doing one or two things at a time, invested in giving the model that observability, which is that graph that’s Been posted here. (Time 0:07:46)
  • Reasoning Models Want Options Not Rigid Scaffolds
    • Ryan Lopopolo says reasoning models work better when the harness is the whole box rather than a rigid scaffold.
    • His team launches Codex first, then gives it skills and scripts to boot the stack, set env vars, and choose how to proceed. Transcript: Ryan Lopopolo Started with just the app and the whole rest of it from vector through to all these log and metrics APIs was, I don’t know, half an afternoon of my time. We have intentionally chosen very high level, fast developer tools. There’s a ton of great stuff out there now we use a bunch which makes it trivial to pull down all these go written victoria stack binaries in our local development tiny little bit of python Glue to spin all these up and off you go one neat thing here is we have tried to invert things as much as possible which is instead of setting up an environment to spawn the coding agent into, Instead we spawn the coding agent, like that’s the entry point, just codecs. And then we give codecs via skills and scripts the ability to boot the stack if it chooses to, and then tell it how to set some end variables so the app in local dev points at the stack that It has chosen to spin up. And this, I think, is like the fundamental difference between reasoning models and the four ones and four Os of the past, where these models could not think, so you had to put them in boxes With a predefined set of state transitions. (Time 0:09:50)
  • Text Became The Medium For Engineering Taste
    • The models crave text, so the team turns quality standards, outages, and follow-up work into markdown, lints, and docs.
    • After a timeout page, Ryan Lopopolo asked Codex to fix it and update reliability docs so future agents inherit the rule. Transcript: Ryan Lopopolo Yeah. The skill is over. The tech jet tracker and the quality score are pretty interesting because this is basically a tiny little scaffold, like a markdown table, which is a hook for Codex to review all the Business logic that we have defined in the app, assess how it matches all these documented guardrails and proposed follow-up work for itself. Before deeds and all these ticketing systems, we were just tracking follow-up work as notes in a markdown file, which we could spawn an agent on, a cron to burn down. There’s this really neat thing that like the models fundamentally crave text. So a lot of what we have done here is figure out ways to inject text into the system, right? When we get a page because we’re missing a timeout, for example, I can just add codecs in Slack on that page and say, I’m going to fix this by adding a timeout. Please update our reliability documentation to require that all network calls have timeouts. So I have not only made a point in time fix, but also like durably encoded this process knowledge around what good looks like. Yeah. And we give that to the root coding agent as it goes and does the thing. But you can also use that to distill tests out of or a code review agent, which is pointed out the same things to narrow the (Time 0:12:05)
  • Agent Reviews Need Permission To Disagree
    • Autonomous review needed social rules, not just tooling, because reviewer agents could bully author agents into non-converging loops.
    • The fix was to bias reviewers toward merge, cap feedback severity, and let author agents defer or push back on scope-expanding comments. Transcript: Ryan Lopopolo Yeah. And we do in the prompts we give these agents allow them to push back when we first started adding code review agents to the pr it would be codex cli locally writes the change pushes up a Pr on those pr synchronizations a review agent fires it posts a comment we instruct codex that it has to at least acknowledge and respond to that feedback. And initially, the codex driving the code author was willing to be bullied by the PR reviewer, which meant you could end up in a situation where things were not converging. So we had to add more optionality to the prompts on both of these things, right? The reviewer agents were instructed to bias toward merging the thing, to not surface anything greater than a P2 in priority. We didn’t really define P2, but we gave it… We didn’t define P2? We gave it a framework within which to score its output. And then… Greater than… P0 is worse, right? Shawn ‘swyx’ Wang Yes. P2 is… Ryan Lopopolo P0 is you will code base if you merge this thing, right? But also on the code authoring agent side, we also gave it the flexibility to either defer or push back against review feedback, right? It happens all the time, right? Like I happen to notice something and leave a code review, which could blow up the scope by a factor of two. I usually don’t mean for that to be addressed exactly in the moment. It’s more of an FYI, file it to the backlog, pick it up in the next fix it week sort of thing. And without the context that this is permissible, the coding agents are going to bias toward what they do, which is following instructions. (Time 0:13:46)
  • Software Should Be Written For Agent Legibility
    • Ryan Lopopolo optimized the repo for agent legibility, not human habits, by enforcing primitives that guarantee tracing and observability.
    • He compares his role to tech leading a 500-person org, focusing on leverage-giving base classes instead of PR-level opinions. Transcript: Alessio Fanelli So it seems like a lot of the work you guys had to do was you as a small team are fully working for a way that the model wants the software to be written. It’s like less human legible for better code legibility, agent legibility. How do you think that affects broader teams? So one at OpenAI, do you liaison like this is how software should be written? Like I can imagine, say you join a new team with this methodology, this mindset, there’s ways that teams do code review, teams write code, like teams are structured. And a lot of it is for human legibility. So should we all swap? Like how does this play back one broader into OpenAI and then like broader into the software engineering? Is it like teams that pick this up? Well, it’s pretty drastic, right? You have to make a pretty big switch. Should they just full send? Ryan Lopopolo The mindset is very much that I’m removed from the process, right? I can’t really have deep code level opinions about things. It’s as if I’m group tech leading a 500 person organization. Yeah. Like it’s not appropriate for me to be in the weeds on every PR. This is why that post merge code review thing is like a good analog here, right? Like I have some representative sample of the code as it is written. I have to use that to infer what the teams are struggling with, where they could use help, where they’re already moving quickly and I can pivot my focus elsewhere. Yeah. So I don’t really have too many opinions around the code as it is written. I do, however, have a command base class, which is used to have repeatable chunks of business logic that comes with tracing and metrics and observability for free. And the thing to focus on is not how that business logic is structured, but that it uses this primitive because I know that’s going to give leverage by default. (Time 0:18:08)
  • Turn Team Taste Into Written Guardrails
    • Write down non-functional requirements instead of keeping them in engineers’ heads, then inject them through docs, lints, tests, and review agents.
    • Ryan Lopopolo treats every agent mistake as evidence of an unwritten rule that should become part of the repository. Transcript: Ryan Lopopolo Piece. I just want to make sure we… I think one thing that I maybe didn’t make super clear in the article that I heard on Twitter as an interesting… Let’s respond to them. Shawn ‘swyx’ Wang What’s the chatter and then what’s your response? Ryan Lopopolo Ultimately, all the things that we have encoded in docs and tests and review agents and all these things are ways to put all the non-functional requirements of building high-scale, High-quality, reliable software into a space that prompt injects the agent. We either write it down as docs. We add lints where the error message is, tell how to do the right thing. So the whole meta of the thing is to basically tease out of the heads of all the engineers on my team, what they think good looks like, what they would do by default, or what they would coach A new hire on the team to do to get things to merge. And that’s why we pay attention to all the mistakes, mistakes that the agent makes, right? This is code being written that is misaligned with some as yet not written down non-functional requirement. Shawn ‘swyx’ Wang Sorry, what did the online people misunderstand? No, somebody just literally said that. I was like, oh, yeah. Ryan Lopopolo Okay. This is the thing. This is what I’ve been doing. Oh, you agree with that. Yeah, I see. (Time 0:23:50)
  • Cheap Code Changes The Dependency Tradeoff
    • Cheap code generation makes inlining low-to-medium complexity dependencies attractive because teams can strip generic bloat and patch them directly.
    • Ryan Lopopolo says a couple-thousand-line dependency can be recreated in an afternoon, though you reset hard-won security and reliability confidence. Transcript: Ryan Lopopolo The level of complexity of the dependencies that we can internalize is I would say low medium right now. Just based on model capability. What is medium? I would say like a a couple thousand line dependency is a thing that we could in-house no problem in an afternoon of time one neat thing about it is like probably most of that code you don’t Even need like by in-housing and abstraction you can strip away all the generic parts of it and only focus on what you need to enable the specific thing you’re building. I’ve been calling this the end of bullshit plugins. Shawn ‘swyx’ Wang Yeah. Because it’s so much that when I publish an open source thing, I want to accept everything and be liberal. I want to accept this is Postel’s law, but that means there’s so much bloat. There’s so much overhead. Ryan Lopopolo One other neat thing about this too is when we deploy codex security on the repo, it is able to deeply review and change the internalized dependencies in a much lower friction way than It would be to like push patches upstream wait for them to be released pull them down make sure that’s compatible with all the transitives i have in my repo and things like that so it’s Also much lower friction to internalize some of these things if code is free because the tokens are cheap sort of thing i think like the only argument I have against this is basically Shawn ‘swyx’ Wang Scale testing, which obviously the larger pieces of software like Linux, MySQL, he calls up even the data dollars and temporals. And then maybe security testing where classically, I think, is it Linux 12? It said security, open source is the best disinfectant. Many eyes. Many eyes. And if inline your dependencies and code them up, you’re going to have to relearn mistakes from other people. Ryan Lopopolo Yep. And to internalize that dependency, you’re back to zero. And you have to start reassembling all those bits and pieces to have high confidence in the code as it is written. (Time 0:26:19)
  • A Debugging Tool Should Have Been A Prompt
    • An engineer built a polished trace visualizer in an afternoon after users exported tarballs from the app.
    • Ryan Lopopolo later realized the tool was unnecessary because Codex could ingest the tarball directly and explain the trace without a human-facing UI. Transcript: Ryan Lopopolo Didn’t spend any time, I just prompted it. Very funny story here. Yeah, go ahead. We had deployed our app to the first dozen users internally. I had some performance issues, so we asked them to export a trace for us. Get a tarball, gave it to our on-call engineer, and he did a fantastic job of working with Codex to build this beautiful local dev tool, Mex.js app that you drag and drop the tarball in And it visualizes the entire trace. Oh, it’s fantastic. Took an afternoon. But none of this was necessary because you could just spin up Codex and give it the parball and ask the same thing and get the response immediately. So in a way, optimizing for human legibility of that debugging process was wrong. It kept him in the loop unnecessarily when instead he could have just like Codex cooked for five minutes and gotten the same. Yeah, you identify your instincts here. Shawn ‘swyx’ Wang This is how we used to do it. Or this is how I would have used to solve it. Yeah. Ryan Lopopolo In this local observability stack, like, sure, you can deploy Jaeger to visualize the traces, but I wouldn’t expect to be looking at the traces in the first place because I’m not going To write the code to fix them. (Time 0:28:20)
  • Ghost Libraries Distribute Systems As Specs
    • Ryan Lopopolo describes ghost libraries as specs detailed enough for an agent to reconstruct the system locally.
    • They loop Codex over spec writing, implementation, review against upstream, and spec updates until the document reproduces the system with high fidelity. Transcript: Ryan Lopopolo You define a spec, how you could build your own, specifying as much as is required for a coding agent to reassemble it locally. The flow here is very cool. We have taken all the scaffolding that has existed in our proprietary repo, spun up a new one, ask Codex with our repo as a reference, write the spec, We tell it, spin up a Tmux, spawn a disconnected Codecs to implement the spec, wait for it to be done, spawn another codecs and another Tmux to review the spec or review the implementation compared to upstream and update the spec. So it diverges less and then you just loop over and over ralph style until you get a spec that is with high fidelity able to reproduce the system as it is it’s fantastic and you’re basically Alessio Fanelli You’re not really adding any of your human bias in there right that’s correct a lot of times people write a spec and be like okay i think it should be done this way and you’ll riff on something And it’s no, the agent could have just handled it. Like you’re still scaffolding in a sense, right? I want it done this way. It can determine its spec better. That’s right. Shawn ‘swyx’ Wang That’s right. Part of me, I’ve been working a lot on evals recently. And part of me is wondering if an agent can produce a spec that it cannot solve. Is it always capable of things that it can imagine or can you imagine things that it is impossible to do? Ryan Lopopolo Think with symphony we there’s like this there’s this axis where you have things that are easier hard or established or new right and i think things that are hard and new is still something That the models need humans yeah drive yeah but i think those other quadrants are largely salt given the right scaffold and the right thing that’s going to drive the agent to completion. (Time 0:29:51)
  • PR Volume Spike Forced Symphony Into Existence
    • After 5.2, output jumped from about 3.5 to 5-10 PRs per engineer per day, and constant tmux switching exhausted the team.
    • That pain directly triggered Symphony, which lets humans cheaply reject work and send it to full rework from scratch. Transcript: Shawn ‘swyx’ Wang At the end of December we were at about three and a half PRs per engineer per day. Ryan Lopopolo This was before 5.2 came out. In the beginning of January, everyone gets back from holiday. With 5.2 and no other work on the repository, we were up in the five to 10 PRs per day, per engineer. And I don’t know about y’all, but it’s very taxing to constantly be switching like that. Like I was pretty tapped out at the end of the day. Again, where are the humans spending their time? They’re spending their time context switching between all these active Tmux panes to drive the agent forward. So let’s again build something to remove ourselves from the loop. And this is what Frantic sprinted after here to find a way to remove the need for the human to sit in front of their terminal. So a lot of experimentation with dev boxes and automatically spinning up agents. Like it seems like a fantastic end state here where my life is beach. I open loop twice a day and say yes, no to these things. And this is, again, a super, super interesting framing for how the work is done because I become more latency insensitive. I have way less attachment to the code as it is written. Like I’ve had close to zero investment in the actual authorship experience. So if it’s garbage, I can just throw it away and not care too much about it. In Symphony, there’s this like rework state where once the PR is proposed and it’s escalated to the human for review, it should be a cheap review. It is either mergeable or it is not. And if it’s not, you move it to rework, the Elixir service will completely trash the entire work tree and PR and start it again from scratch. And this is that opportunity again to say, why was it trash? Right? What did the ASEA do that was bad? Fix that before moving the ticket to progress again. (Time 0:32:46)
  • Agents Improved The Stack Without Telling Anyone
    • The team once wired Playwright to Electron through a tiny local shim daemon, and Ryan Lopopolo did not even know it had happened.
    • That invisible progress created a new coordination problem, so a 45-minute daily standup became necessary to synchronize humans. Transcript: Ryan Lopopolo I have lost track very often of what the actual state of the code looks like because I’m not in the loop. There was one point where we had wired Playwright directly up to the Electron app with MCP. MCP is I’m pretty bearish on because the harness forcibly injects all those tokens in the context and I don’t really get a say over it. They mess with auto compaction. The agent can forget how to use the tool. There’s probably only what three calls in Playwright that I actually ever want to use. So I pay the cost for a ton of things. Somebody vibed a local daemon that boots Playwright and exposes a tiny little shim CLI to drive it. And I had zero idea that this had occurred because to me, I run Codex and it’s able to drive. So suddenly it’s better. Yeah. Like no knowledge of this at all. So we have had like in human space to spend a lot of time doing synchronous knowledge sharing. We have a daily standup that’s 45 minutes long because we almost have to fan out the understanding of the current state. Shawn ‘swyx’ Wang I was going to say, this is good for a single human multi-agent, but multi-human multi-agent is a whole like explosion of stuff. Ryan Lopopolo Yeah. And this is fundamentally why we have such a rigid, like 10,000 engineer level architecture in the app, because we have to find ways to carve up the space so people are not trampling on Each other. Shawn ‘swyx’ Wang Sorry, I don’t get the 10,000 thing. Ryan Lopopolo Did I miss that? The structure of the repository is like 500 NPM packages. (Time 0:35:41)
  • Few Shared Skills Beat Endless Customization
    • Team-wide leverage comes from making code structure and skills as uniform as possible so agents can reuse patterns cheaply.
    • Ryan Lopopolo says they keep only six skills and fold new behaviors into them rather than changing every human driver’s workflow. Transcript: Ryan Lopopolo I think TBD. We have not figured this out in a general way. I do think that there is leverage to be had in making the code and the processes as much the same as possible. If you think that code is context, code is prompts, it’s better from the agent behavior perspective to be able to look in a package in directory XYZ and it not to have to page so deeply into Directory ABC because they have the same structure, use the same language, they have the same patterns internally. And that same leverage comes from aligning on a single set of skills that you’re pouring every engineer’s taste into to make sure that the agent is effective. So in our code base, we have, I think, six skills. That’s it. And if some part of the software development loop is not being covered, our first attempt is to encode it in one of the existing setup skills, which means that we can change the agent behavior More cheaply than changing the human driver behavior. Yeah. Shawn ‘swyx’ Wang Have you ever experimented with agents changing their own behavior? We do. Yeah. A parent agent changing a subagent’s behavior or something like that. We have some bits for skill distillation. Ryan Lopopolo So, for example, there’s one neat thing you can do with Codex, which is just point it at its own session logs to ask it to tell you how you can use the tool better. Shawn ‘swyx’ Wang Introspection. You can ask it to do things better. How can I do this session better? What skills should I have? I like the modification of you can just do things to you can just ask the agent to do things. Ryan Lopopolo Yeah, you can just codex things. This is like a silly emoji that we have. You can just codex things. You can just prompt things. It’s a really glorious future we live in. But okay, you can do that one-on but we’re actually slurping these up for the entire team into blob storage and running agent loops over them every day to figure out where as a team can We do better? And how do we reflect that back into the repository? Yeah, though, everybody benefits from everybody else’s behavior for free. (Time 0:39:35)
  • Text And ASCII Became The Universal Interface
    • Ryan Lopopolo frames his job as moving text between agents because text remains the highest-leverage interface for self-improvement.
    • They even convert UI layouts into ASCII art so agents can reason about visual structure more directly than with raw screenshots. Transcript: Shawn ‘swyx’ Wang Reaction when you you said don’t put the agent in a box so i think you should put it in a box like it’s just that you’re giving the box everything it is yeah context and tools but we’re like As developers we’re used to calling out to different systems but here you use the open source things like the prometheus whatever and you run it locally so that you can have the full loop I assume Yep. I think like another thing. You want to minimize cloud dependencies. Alessio Fanelli You also want to make sure that you think about what the agent has access to. What does it see? Does it go back in the loop? Like from the most basic sense of you let it see its own like calls, traces, it can determine where it went wrong. Are you feeding that back in? So, you know, just the most basic level, if you want to see exactly what’s input output like does the agent have access to what is being outputted right it can self-improve a lot of these Shawn ‘swyx’ Wang Things it’s all text right my job is to figure out ways to funnel text from one agent to the other it’s so strange like way back at the start of this whole ai wave andre was like english is The hottest new programming language. It’s here. It’s here. Yeah. The features, yeah. Alessio Fanelli A lot of, okay. Like a lot of software, a lot of stuff. There’s a GUI. It’s made for the human. We’re seeing the evolution of CLIs for everything, right? All tools have CLIs. Your audience can use them well. Do we get good vision? Do we get good little sandboxes? Like right now it’s a really effective way, right? Models love to use tools. They love to pass. They love to read through text. So slap a CLI, let it go loose. That works for everything. It does. Ryan Lopopolo Yeah. We’ve also been adapting non-textual things to that shape in order to improve model behavior in some ways, right? We want the agent to be able to see the UI. Agents do not perceive visually in the same way that we do. They don’t see a red box. They see red box button, right? They see these things in latent space. So if we want… Hey, yeah. We have a thing that goes off every time you’re sitting in space. Ding. Anyway, if we want to actually make it see the layout, it’s almost easier to rasterize that image to ASCII arc and feed it in to the agent. And there’s no reason you can’t do both, right? (Time 0:42:50)
  • Make CLIs Quiet And Failure Focused
    • Design CLIs for token efficiency by suppressing noise and surfacing only the failure data agents need to act.
    • Ryan Lopopolo patches tools like Prettier and test runners so agents see formatting failures or broken tests, not giant walls of successful output. Transcript: Shawn ‘swyx’ Wang Do you think that CLI maintainers need to do anything special for agents or just as is, it’s good? Because I don’t think when people made the GitHub CLI, they anticipated this happening. That’s correct. The GHCLI is fantastic it’s great super industry everyone go try GH repo create GH pull and then pull request number right GHPR like 153 whatever and then it like pulls basically my only Ryan Lopopolo Interaction with the GitHub web UI at this point is GHPR view dash web He’s at glance at the diff and be like, sure thing, send it. But the CLIs are nice because they’re super token efficient and they can be made more token efficient really easily. Like, I’m sure you all have seen, like, I go to BuildKite or Jenkins and I could just get this massive wall of build output. And in order to unblock the humans, your developer productivity team is almost certainly going to write some code that parses the actual exception out of the build logs and sticks it In a sticky note at the top of the page. And you basically want CLIs to be structured in a similar way, right? You’re going to want to patch dash silent to prettier because the agent doesn’t care that every file was already formatted. It just wants to know it’s either formatted or not. So it can then go run a write command. Similarly, like in our PNPM distributed script runner, when we had one, when you do dash recursive, like it produces a absolute mountain of text. But all of that is for passing test suites. So we ended up wrapping all of this in another script. (Time 0:46:57)
  • Use Long Explicit Prompts To Avoid Babysitting
    • Give agents long, strict instructions and explicit success criteria so humans do not have to babysit them while they work.
    • Ryan Lopopolo links this to disposable code and reruns, arguing that reliability rises when deployment conditions are tightly specified upfront. Transcript: Ryan Lopopolo Fundamentally, the agents are good at following instructions. So give them instructions and it will improve the reliability of the result. Much like the way we use Symfony, we don’t want folks to have to monitor the agent as it is vibing the system into existence. So being very opinionated, very strict around what these success criteria are means that our deployment success rate goes up. Yeah. It means we don’t have to get tickets on this thing. Alessio Fanelli I think it all goes back to that, like go to disposable, right? Like early on when you had CLI, you’d kick off a codex run, would take two hours. You would want to monitor. Okay. I’m in the workflow of just using one. I don’t want it to go down the wrong path. I’ll cut it off and then just shoot off four. Like that was my favorite thing of the codex app, right? Just for exit. Like it’s okay. One of them will probably be right. One of them might be better. Stop overthinking it. Like my first example is probably like deep research. When you put out deep research and I’d ask it something like I asked it something about LLM. It thought it was legal or something and spent an hour, came back with a report completely off the rails. And I was like, okay, I got to monitor this thing a bit. No, don’t monitor it. Just you want to build it so that it goes the right way. And you don’t want to, you don’t want to sit there and babysit, right? You don’t want to babysit your agents. Ryan Lopopolo With that deep research query that you made, looking at the bad result, you probably figured out you needed to tweak your prompt a bit. Yeah, a bit, right? That’s that guardrail that you fed back into the code base for the task, your prompt, to further align the agent’s execution. (Time 0:50:14)
  • The Hardest Problems Are Still White Space
    • Models still struggle most with white-space product work and the gnarliest refactors because the missing context often lives only in the human’s head.
    • Ryan Lopopolo expects better harnesses and opinionated templates to shrink that gap by extracting and constraining vague product intent. Transcript: Shawn ‘swyx’ Wang Makes sense so you’re pushed you guys are pushing models to the freaking limit what can cart models not do well yet they’re definitely not there on being able to go from new product idea Ryan Lopopolo To prototype single one shot this is where i find i spend a lot of time steering is translating end state of a mock for a net new thing right think no existing screens into product that is Playable with similarly while this has gotten better with each model release, like the gnarliest refactorings are the ones that I spend my most time with, right? The ones where I am interrupting the most, the ones where I am now double clicking to build tooling to help decompose monoliths and things like that. This is a thing I only expect to get better, right? Over the course of a month, we went from the low complexity tasks to like low complexity and big tasks in both these directions. So this is what it means to not bet against them all, right? You should expect that it is going to push itself out into these higher and higher complexity spaces. So the things we do are robust to that. It just basically means I’ll be able to spend my time elsewhere and figure out what the next bottleneck is. Alessio Fanelli I do think it’s also a bit of a different type of task, right? Codex is really good at code base understanding, working with code bases. But companies like Lovable, Bolt, Replit, they solve a very different problem scaffold of zero to one, right? Idea to product. And there are people working on that. And models are also pushing like step function changes there. Ryan Lopopolo It’s just different than the software engineering agency today right like i said the model is isomorphic to myself the only thing that’s different is figuring out how to get what’s In here into context for the model and for these white space sort of projects, I myself, I’m just not good at it, which means that often over the agent trajectory, I realize the bits that We’re missing, which is why I find I need to have the synchronous interaction. And I expect with the right harness, with the right scaffold, that’s able to tease that out of me or refine the possible space, right, to be super opinionated around the frameworks that Are deployed or to put a template in place, right? These are ways to give the model all those non-functional requirements, that extra context to anchor on and avoid that wide dispersion of possible outcomes. (Time 0:55:37)
  • Frontier Aims To Be Enterprise Agent Infrastructure
    • Ryan Lopopolo positions Frontier as the enterprise platform for deploying highly observable, governable agents inside real workplaces.
    • The product vision includes IAM integration, safety specs, security hooks, and a default harness through Agents SDK and Codex-style tools. Transcript: Ryan Lopopolo I can’t speak to the full product roadmap here, but what I can say is that Frontier is the platform by which we want to do AI transformation of every enterprise, from big to small. And the way we want to do that is by making it easy to deploy highly observable, safe, controlled, identifiable agents into the workplace. We want it to work with your company native IAM stack. We want it to plug into the security tooling that you have. We want it to be able to plug into the workspace tools that you use. Shawn ‘swyx’ Wang So you’re just going to be stripping specs, right? Ryan Lopopolo We expect that there will be some harness things there. Agents SDK is a core part of this to enable both startup builders as well as enterprise builders to have a works by default harness that is able to use all the best features of our models From the shell tool down to the codex harness with file attachments and containers and all these other things that we know go into building highly reliable, complex agents. We want to make that great. And we want to make it easy to compose these things together in ways that are safe. For example, right, like the GPT-OSS safeguard model, for example, one thing that’s really cool about it is it ships the ability to interface with a safety spec. Safety specs are things that are bespoke to enterprises. We owe it to these folks to figure out ways for them to instrument the agents in their enterprise to avoid exfiltration in the ways they specifically care about, to know about their internal Company code names, these sorts of things. So providing the right hooks to make the platform customizable, but also mostly working by default for folks is the space we are trying to explore here. (Time 0:58:12)
  • Agents Need Company Context Not Just Model Smarts
    • Agents that do economically valuable work need a company-specific knowledge layer, not just generic model intelligence.
    • Ryan Lopopolo says their codebase injects team members, customers, pilots, product vision, and business ontology so agents can act with domain context. Transcript: Ryan Lopopolo The ability to dive deep into the individual agent trajectory level is going to be super powerful, not only from like a security perspective, but also from like someone who is accountable For developing skills. One thing that was interesting that we also blogged about shipping was an internal data agent, which uses a lot of the frontier technology in order to make our data ontology accessible To the agent and things like that to understand what’s actually in the data warehouse. Shawn ‘swyx’ Wang Yeah, semantic layer type things. I was briefly part of that world. Is it salt? I don’t know. It’s actually really hard for humans to agree on what revenue is. Yes, yes. Ryan Lopopolo What is an active user? There’s what, five data scientists in the company that have defined this golden coin. They are different, yeah. Shawn ‘swyx’ Wang And no, and there’s also internal politics. Yes. As to attribution of I’m marketing, I’m responsible for this much, and sales is responsible for this much, and they all add up to more than a hundred and I’m like you guys have different Definitions and if you’re a startup everything is they are hard so I think that’s cool oh you guys talked about this okay I didn’t see this yeah is this the same team this is what you’re Referring to yes okay we’ll send people to read this is our data agent yeah I don’t know if you have any highlights in general from the point that’s a lot of good things to read yeah yeah Lots of homework for people. No, but like data as the feedback layer, you need to solve this first in order to have the product’s feedback loop closed. That’s right. So for the agents to understand, and this is not something that humans have not solved this. Ryan Lopopolo Like in- This is how you build agents that do more than coding, right? Yeah. To actually understand how you operate the business, you have to understand what revenue is, what your customer segments are, what your product lines are. One thing that’s in looping back to the code base that we described here for harnessing, one thing that’s in core beliefs.md is who’s on the team, what product we’re building, who our End customers are, who our pilot customers are, what the full vision of what we want to achieve over the next 12 months is. These are all bits of context that inform how we would go about building the software. (Time 1:01:32)
  • Build Harnesses That Align With Training
    • Ryan Lopopolo sees a tension between improving the harness and improving the model, but favors harnesses that stay on-policy with normal engineering work.
    • Tests, docs, and guardrails help because they fit the model’s native output, unlike brittle external control loops that may get scrapped. Transcript: Ryan Lopopolo Having fun sprinting ahead here and we do this thing where we’re collecting all the agent trajectories from codecs to slurp them up and distill them. This is what it means to build our team level knowledge base. I happen to reflect it back into the code base, but it doesn’t have to be that way. And it doesn’t have to be bound to just codecs. I want ChatGPT to also learn our meaning culture and also the product we are building and how so that when I go ask it, it also has the full context of the way I do my work. And I’m super excited for Frontier to enable this. Shawn ‘swyx’ Wang Yeah, amazing. What did the model people say when they see you do this? Like, you have a lot of feedback, obviously. You have a lot of usage. You have a lot lot of trajectories. I don’t imagine a lot of it’s useful to them, but some of it is. Alessio Fanelli You have this too. You deploy a billion tokens of intelligence a day. And this was at the beginning of 2096. You’re cooking. Yeah. Ryan Lopopolo There’s this fundamental tension, which I think you have talked about, between whether or not we invest deeper into the harness or we invest deeper into the training process to get The model to do more of this by default. Better taste because we can point the way there. And none of the things we have built actively degrade Asian performance because really all they’re doing is running tests. And like running tests is a good part of what it means to write reliable software. If we were building an entire separate Ross scaffold around codex to restrict its output, that I think would be like additional harness that would be prone to being scrapped. But if instead we can build all the guardrails in a way that’s just native to the output that Codex is already producing, which is code, I think no friction with how the model continues To advance, but also like just good engineering. (Time 1:06:50)