Skip to content

Podcast

Autoresearch, Agent Loops and the Future of Work

The AI Daily Brief: Artificial Intelligence News and Analysis

Source ↗ ← All highlights
  • AI Agents Running Overnight ML Research
    • Andre Karpathy’s AutoResearch hands the full ML experiment loop to an AI agent that edits train.py and runs fixed five-minute training runs.
    • The human writes program.md (a research strategy doc) and the agent keeps only commits that improve validation BPB, iterating dozens to hundreds of times overnight. Transcript: Nathaniel Whittemore On Saturday, Andre, who was on the founding team at OpenAI, and who was previously the director of AI at Tesla, and who you might remember from coining such terms as vibe coding last February, And who has now suggested we are in a different era of agentic engineering as of this February, again tweeted on Saturday, I packaged up the auto research project into a new self-contained Minimal repo if people would like to play over the weekend. It’s basically NanoChat LLM training core stripped down to a single GPU, one file version of around 630 lines of code, then the human iterates on the prompt.md, and AI agent iterates On the training code.py. The goal is to engineer your agents to make the fastest research project indefinitely and without any of your own involvement. In the image, which he shared alongside it, every dot is a complete LLM training run that lasts exactly five minutes. The agent works in an autonomous loop on a Git feature branch and accumulates Git commits to the training script as it finds better settings of lower variation loss by the end of the neural Network architecture, the optimizer, all the hyperparameters, etc. You can imagine comparing the research progress of different prompts, different agents, etc. Part code, part sci-fi, and a pinch of psychosis. As a caption to the image, he wrote, One day, frontier AI research used to be done by meat computers in between eating, sleeping, having other fun, and synchronizing once in a while using Soundwave Interconnect in the ritual of a group meeting. That era is long gone. Research is now entirely the domain of autonomous swarms of AI agents running across compute cluster megastructures in the skies. The agents claim that we are now in the 10,205th generation of the codebase. In any case, no one could tell if that’s right or wrong as the code, quote-unquote, is now a self-modifying binary that has grown beyond human comprehension. This repo is the story of how it all began. So let’s talk about what auto research actually is, at least in the version that was released by Andre. Auto research is a system for training a small language model. Basically the kind of model that powers all of these AI tools, but much smaller. The type of model that could one day run on, for example, an edge device like a phone. The goal is to make a model as good as possible at understanding and generating text. Normally, or classically, a human researcher would sit there tweaking the training setup, doing things like adjusting the model’s architecture, changing how fast it learns, experimenting With different optimization strategy. They’d run an experiment, check the results, decide what to try next, and repeat. That’s basically the core loop of machine learning research, and it’s bottlenecked by how fast a human can iterate. AutoResearch instead hands that entire loop to an AI agent. And it does so in an intentionally simplified and tiny way. In this repo, there are just three files that matter. The first is prepare.py, which is fixed infrastructure that doesn’t change. It downloads the training data, trains a tokenizer, and handles evaluation. The second is train.py. This contains the entire GPT model definition, the optimizer, and the training loop. This is the single file the AI agent is allowed to edit. Everything in it is fair game. The model architecture, the hyper parameters, the batch size, the attention parameters, the learning rate schedule, literally everything. The third file is program.md, and this is the most conceptually important one especially in the context of this idea of these loops being larger primitives. It’s a markdown file, plain text, written in English, that contains the instructions for the AI agent. It describes how the agent should behave as a researcher, what kind of experiments to try, what to be cautious about, when to be bold versus conservative. This is the file that the human in this equation edits. So the way that this is going to work is you point an AI agent like Claude or Codex or whatever at the repo and tell it to read program.md and start experimenting. The agent reads the instructions, looks at the current state of train.py, decides on a modification to try, makes the edit, and kicks off a training run. Every training run has a fixed 5-minute budget. When the run finishes, the system evaluates the model on a validation set and produces a single number. In this case, that’s validation bpb or val bpb, which stands for validation bits per byte. In this case, lower is better. The agent then makes a decision. If the new val bpb is lower than the previous best, the change is kept, it gets committed to a git feature branch, it becomes the new baseline, and the agent builds on top of it for the next Experiment. If the valbpb is the same or higher, the change is discarded. The agent reverts to the previous best version and tries something different. Then the loop repeats indefinitely. (Time 0:02:44)
  • Agent Loop As A Universal Primitive
    • The loop pattern abstracts to four steps: human writes strategy, agent runs experiments, a clear metric decides what stays, and repeat many times.
    • Craig Hewitt and others say the architecture and mindset matter more than the specific code or model. Transcript: Nathaniel Whittemore Example of the agent loop that’s about to eat everything. 1. Human writes a strategy doc. 2. Agent executes experiments autonomously. 3. Clear metric decides what stays and what gets tossed. 4. Repeat 100x overnight. The person who figures out how to apply this pattern to business problems, not just ML research, is going to build something massive. The code is almost irrelevant. The architecture and mindset is everything. Daniel Meisler called this automation of the scientific method. And It’s Me Chase also noticed that this would be valuable for things outside of ML research as well. He writes, while this was made for self-improving LLMs, the framework could be applied to anything. One, AI agent reads context and previous results. Two, proposes targeted code edits. Three, runs a fast reproducible experiment. Four, gets an objective scalar score. Five, Git commits only the winners or reverts. Six, repeats forever on a feature branch. (Time 0:08:49)
  • Company Brain File Prevents Agent Amnesia
    • Vadim from Vugola built company-wide loops where every agent reads and appends to a shared learnings.md to avoid pure amnesia.
    • This turned isolated agent runs into a network that accumulates knowledge and runs tens of thousands of experiments. Transcript: Nathaniel Whittemore I built a version of this for my whole company. The core problem with most agent setups, they output something and stop. The agent writes an email, sends an email, generates code, done. The next time it runs, it starts from zero. No memory of what worked, no memory of what failed, pure amnesia. That’s not automation, that’s a script you babysit. The fix is one principle, close the loop. Every agent in my setup on OpenClaw reads a shared brain file before doing any work, then writes back to it after. I call it learnings.md. It’s baked into every agent’s system prompt. Before starting work, read learnings.md. After completing work, append what you learned to learnings.md. That’s the foundation. One file, all agents read it, all agents write to it. Now they’re not isolated processes. They’re a network that accumulates knowledge. So basically, Vadim is describing a loop for the entire agentic process of his company. In an article on X, he writes, Most marketing teams run around 30 experiments a year. The next generation will run 36,500 plus easily. Things like new landing pages, new ad creative, maybe a subject line test. Except what if you applied an experiment loop? (Time 0:15:45)
  • When Agent Loops Excel
    • Agentic loops work best where a clear, objective score exists and iterations are fast and cheap.
    • Whittemore lists requirements: scorable metric, fast/cheap iterations, bounded action space, low cost of failure, and traceable outputs. Transcript: Nathaniel Whittemore I think you’re going to see this loop applied to a huge range of activities, but where it’s going to initially be most successful are areas where there are five things that are true. First, there is a score, something that is scorable. In other words, that the loop can tell better from worse without asking a human. The more subjective worse or better is, the harder this is going to be, although even that’s not impossible. You just have to build some sort of objective scoring into the system. The second requirement is that iterations are fast and cheap. Basically, that bad attempts waste minutes, not months. The environment needs to be bounded, with the agent having a defined work and action space. The cost of a bad iteration needs to be low, i.e. You’re not going to try this live with legal filings, and the agent needs to be able to leave traces. (Time 0:17:59)
  • High-Value Targets For Agent Automation
    • Top-quadrant automatable tasks include code generation, game NPC behavior, ad bid optimization, algorithmic trading, and LLM training.
    • These tasks combine seconds-long iteration with fully automated evaluation, making them ideal for loops. Transcript: Nathaniel Whittemore Up in the top quadrant where iteration speed is seconds and evaluation can be fully automated is things like code generation. Some of the other ones that Claude came up with were game AI and NPC behavior, ad bid optimization, algorithmic trading. And then, of course, we’ve got LLM training research, according to Andre Karpathy. (Time 0:19:07)
  • Agent Loops Become Cross-Functional Primitive
    • Agent loops will become a cross-functional primitive like email or spreadsheets, used by product managers, sales reps, analysts, recruiters, QA, and lawyers.
    • Examples include PRD-driven development loops, overnight lead scoring, portfolio backtests, and contract risk flagging. Transcript: Nathaniel Whittemore This is something that people are going to within their existing roles, in the same way that meetings or slide decks or email or spreadsheets are primitives that people use and cut across Every function. What we’re going to have in the future is things like this. A product manager writing a PRD, kicking off a Ralph loop before dinner and reviewing the PR in the morning. A sales rep writing targeting criteria and tone guidelines, pointing a loop at 200 leads overnight and reviewing the top 30. A financial analyst defining constraints, looping through portfolio allocation backtests and reviewing the optimized output. A recruiter writing a scoring rubric, looping through 500 resumes and reviewing flagged edge cases. A QA engineer writing acceptance criteria and then looping through test generation and execution. A lawyer writing a risk flag checklist and looping through a stack of vendor contracts. Now, interestingly, there is already very clearly a lot of work to productize this. (Time 0:20:10)
  • Shift Focus To Arena Design And Evaluators
    • Focus on higher-level skills: design the arena (program.md), build scoring/evaluation, and decompose problems for agent loops.
    • These roles, not low-level execution, become the comparative advantage as agents handle iteration. Transcript: Nathaniel Whittemore This is something that people are going to within their existing roles, in the same way that meetings or slide decks or email or spreadsheets are primitives that people use and cut across Every function. What we’re going to have in the future is things like this. A product manager writing a PRD, kicking off a Ralph loop before dinner and reviewing the PR in the morning. A sales rep writing targeting criteria and tone guidelines, pointing a loop at 200 leads overnight and reviewing the top 30. A financial analyst defining constraints, looping through portfolio allocation backtests and reviewing the optimized output. A recruiter writing a scoring rubric, looping through 500 resumes and reviewing flagged edge cases. (Time 0:20:10)
  • Agents Need A Semantic Shared Memory
    • The next step is massive asynchronous multi-agent collaboration with shared memory so agents can share failed experiments and prune search trees.
    • Andre and others argue Git isn’t sufficient; agents need semantic memory of decisions, reasoning, and negative results. Transcript: Nathaniel Whittemore Andre himself wrote about this on Sunday. The next step for auto-research, he says, is that it has to be asynchronously massive collaborative for agents. The goal is not to emulate a single PhD student, it’s to emulate a research community of them. Current code synchronously grows a single thread of commits in a particular research direction, but the original repo is more of a seed, from which could sprout commits contributed By agents on all kinds of different research directions or for different compute platforms. GitHub is almost but not really suited for this. Has a softly built-in assumption of one master branch which temporarily forks off into PRs just to merge back a bit later. I’m not actually sure what this collaborative version should look like, but it’s a big idea that is more general than just the auto-research repo specifically. Agents can in principle easily juggle and collaborate on thousands of commits across arbitrary branch structures. Existing abstractions will accumulate stress as intelligence, attention, and tenacity cease to be bottlenecks. Other people picked up this theme. Blake Heron writes, The missing layer is memory across the swarm. Right now, each agent runs in an isolated thread with no awareness of what other agents tried, what worked, what conflicted. Git tracks code changes but not decisions, reasoning, or failed experiments. You need a semantic memory layer underneath the branches so Agent 47 knows Agent 12 already tried that direction and it didn’t converge. (Time 0:21:41)
  • Point A Loop At Work Tasks You Can Score
    • Try identifying repetitive tasks in your work where you know what ‘better’ looks like and build a scoring function an agent can use.
    • Point an agentic loop at that task overnight to preview how your job might shift higher-level. Transcript: Nathaniel Whittemore One interesting experiment to run this week is to, as you’re working, find the things that you repeatedly do or are part of doing where you know right now what better looks like. Ask if you could encapsulate that judgment clearly enough for an agent to use it as a score. If you can, you might be able to point a loop at that part of your job to work on your behalf overnight, and that likely gives you a preview of the next version of your job. (Time 0:24:03)