Skip to content

Podcast

Schrödinger’s Apocalypse

The AI Daily Brief: Artificial Intelligence News and Analysis

Source ↗ ← All highlights
  • Agentic Engineering Rewrote Programming
    • AI agents shifted programming from typing code to orchestrating agentic workflows in December, creating large leverage for top-tier agent engineering.
    • Nathaniel Whittemore notes higher model coherence and long-task tenacity letting users manage parallel code instances via English goals rather than line-by-line coding. Transcript: Nathaniel Whittemore The last couple of months have seen a steady growing acknowledgement of just how significant the disruption of AI is. This shift came first to those who are actually in the industry. Just last week, OpenAI founder Andre Karpathy wrote, It’s hard to communicate how much programming has changed due to AI in the last two months. Not gradually and over time in the progress-as way, but specifically this last December. There are a number of asterisks, but in my opinion, coding agents basically didn’t work before December and basically worked since. The models have significantly higher quality, long-term coherence and tenacity, and they can power through large and long tasks, well past enough that it is extremely disruptive To the default programming workflow. As a result, programming is becoming unrecognizable. You’re not typing computer code into an editor like the way things were since computers were invented. That era is over. You’re spinning up AI agents, giving them tasks in English, and managing and reviewing their work in parallel. The biggest prize is in figuring out how you can keep ascending the layers of abstraction to set up long-running orchestrator clause with all of the right tools, memory, and instructions That productively manage multiple parallel code instances for you. The leverage achievable via top-tier agentic engineering feels very high right now. In my opinion, this is nowhere near business-as time in software. (Time 0:00:52)
  • Howard Marks Frames Autonomous AI As Labor Replacement
    • Howard Marks’ memo frames AI as an unprecedented rapid capability leap that can act autonomously and replace knowledge-worker tasks at scale.
    • The memo distinguishes chat, tool-using, and autonomous agent levels, highlighting labor replacement at the task level. Transcript: Nathaniel Whittemore The memo is called AI Hurdles Ahead. In it, Marks writes, my main reason for writing this addendum is to address significant changes that have taken place in AI over the three months since I published Is It a Bubble? First, he said, there’s the pace at which developments in AI are occurring. That speed is unlike anything we’ve seen before now, and this has implications that have never existed. AI is growing at speeds that greatly outpace the technological innovations of the past. Nothing has ever taken hold of the pace AI has. It’s able to change the world at a speed that approaches instantaneous, outpacing the ability of most observers to anticipate or even comprehend. The second important thing that’s happened has been an incredible leap ahead in AI’s capabilities. Level 1 is chat AI. Level 2 is tool using AI. Level 3 is autonomous agent. At this level, the user doesn’t tell AI what to do. The user gives it a goal as well as the parameters of the desired output. The agent does the work, checks it, and submits a finished product. This is labor replacement at the task level, not assistance, replacement. Marx continues, the most significant thing that distinguishes AI is something we’ve never dealt with in connection with prior technological developments, AI’s ability to act autonomously. The bottom line, Marx concludes, is that AI is very real, capable of doing a lot of work that heretofore has been done by knowledge workers, and growing extremely rapidly in terms of Applications. (Time 0:02:49)
  • Manaus Diversion Showed AI Helps But Humans Decide
    • Whittemore recounts an emergency diversion to Manaus and how LLMs helped translate, find rentals, and plan routes during the ordeal.
    • Despite AI assistance, he emphasizes human discretion from airline and hotel staff as decisive in resolving real exceptions. Transcript: Nathaniel Whittemore And this idea of human wants, both in terms of their ability to expand, but also just in terms of their manifestation in reality and theory, was the subject of my ponderance, written In the midst of a 20-hour forced layover in the Amazonian rainforest, that I called, We’re all missing the most important market force that will shape AI, or, My plane made an emergency Landing in the Amazon and all I got was this lesson about the future of the world. The piece reads, It’s a weird week, man. A bomb cyclone blizzard with the force of a Category 2, nearly 3 hurricane shut down New York and the rest of the East Coast. This was problematic for lots of reasons, not least of which was that it completely torpedoed our family’s return from Uruguay to the Hudson Valley. Meanwhile, back home, the latest AI Doomer sci-fi, I say that with a lot less derision than it probably sounds, struck a nerve deep enough to rip the throats out of IBM, Visa, and many Others just because of what AI might do. As I sit here in Manaus, Brazil, I find myself contemplating how my was this lesson about the future of the world. The piece reads, family’s experience over the last 24 hours or so demonstrates just how wrong I think we are about how AI ends up playing out in the economy. I’ll give you a moment to finish ralphing at the utter LinkedIn-ness of that statement, and then let me explain why we’re all missing the most important market force that will shape AI. When we got the notification that we were making an emergency landing in Manaus, the trip had already been an utter calamity. A few days earlier in the middle of the night, we got the text notification from Delta that, almost assuredly, our upcoming trip from Montevideo to JFK was going to get 86’d by the impending Snowstorm. The options for rescheduling weren’t great. It was basically stick around Uruguay till Friday, when we were supposed to have gotten home Monday morning, or scurry on Monday to do a new multi-leg trip through Sao Paulo and Atlanta. We figured that even if things were still gnarly in New York on the back end, solving that from Georgia was easier than solving that from Sao Paulo. Dutifully, we drove the two hours from Jose Ignacio to the Montevideo airport, returned our tiny VW rental, and let the kids scarf some Mickey D’s before the 27 hours of upcoming travel. In retrospect, it was the last peaceful moments of optimism we’d have for some time. We got to the check-in line and instantly it was clear that something was wrong. I can’t check you in, said the attendant. Wait, what? Why? We can’t check in anyone whose final destination is New York. But the storm is over. We’re not even getting there until tomorrow when it will be even more over. And we’ve got stops in Sao Paulo and Atlanta. Let us get stuck there. I can’t. It’s our policy. And after a call to her supervisor, it remained their policy. We didn’t have a lot of great options. Turn around and hang out for another five days. Or call Delta, have them delete the final Atlanta to JFK leg so we could at least get to the US. Atlanta it was, and after some frantic searching, we booked what seemed like the last rental car in America to do the 15-hour drive home from Jackson Hartfield. Fast forward about 10 hours. We’ve made it through the first leg of the flight, a couple hours in the actually kinda excellent GRU airport in Brazil, and all of us, including four-year Gus and seven-year Alden, Are passed out dreaming of a next day full of Kia Souls and a dozen Wawa and Red Bull stops. That is until, at 4am, the captain gets over the loudspeaker and says that, sorry, a generator has stopped working and we have to make an emergency diversion into Manaus. That’s the capital of the Amazon for those keeping track at home. We hadn’t even made it out of Brazil. So much for, at least if we get stuck, it will be in the US. Airports are stressful at the best of times. 300 people dropped out of the sky into a place many of them had never heard of and at the mercy of the gods of airplane mechanics, hotel availability, and Brazilian customs authorities, And you’ve got something else entirely. But this is supposed to be the setup to a story about AI, right? We’re now sitting here at the lovely hotel Villa Amazonia in the old part of Manaus, waiting for a room to be ready so we can catch a few winks before trudging back to catch another plane. Hopefully a new one, to be honest, will, somehow, someway, get us back to the US of A. It is absolutely undeniable how much AI has made this experience better. I’ve used LLMs to translate back and forth in a language I barely know how to say thank you in, researched the safety profile of different areas. It’s not a war zone, but it’s not low risk either. Gee, thanks, ChatGPT, real reassuring. And I’ve also used ALMs to hunt for rental cars, planned driving routes, and of course, reassure myself that Airbus 330s really can fly with just one generator in those tense 45 minutes Between when we got the announcement and when we touched down. And yet, as awesome as AI has been, every part of the story has really been about human interaction and human discretion, either that went for us or against us. The attendant at MVD and her supervisor, who didn’t buck a clearly stupid policy that might have made sense 24 hours earlier but certainly didn’t anymore. The customer service reps of the Delta Diamond Medallion status line, who ranged from wildly unhelpful on the one end of the spectrum to hustling to find us a flight to Philly on the other. The hotel staffers, who overlooked that we hadn’t technically booked our kids on the reservation, and who hustled to get us in a room before the 3pm check-in time. (Time 0:15:38)
  • Perfect Compliance Creates Brittle Systems
    • Perfect rule-following by AI creates brittleness because human systems rely on discretionary exceptions as a shock absorber.
    • Whittemore argues markets and customers value the possibility of exception, citing loyalty tiers and the Delta Diamond line as products of human discretion. Transcript: Nathaniel Whittemore You can probably see where I’m going with this. A world where AI agents perfectly followed the policy all the time would be, in many, many real-world contexts, much worse than the one where humans follow it only imperfectly. Call it the paradox of perfect compliance. But couldn’t AI have grace and flexibility programmed in as well? Sure, and as we design agent-led systems, it will probably be important to remember that in people’s real lived experience, exceptions are as important as rules. But kindness as governance, an unspoken and yet nearly universal aspect of well-functioning human systems, is hard to program. Small acts of bureaucratic rebellion tend not to be the byproduct of clear, rational calculations. Instead, they are felt decisions. They are a split-second judgment call that comes on the heels of the utterly relatable exhale of an exhausted parent at their wits’ end just trying to keep it together for their even More exhausted kids. There’s something in the pleas of the person being helped that suggests that, as bad as this situation is, there’s something else they’re going through that’s even harder. Which brings us to this weekend’s market freakout cause du jour. The latest AI Doomer fanfic slash thought exercise is a fictional dispatch from 2028 describing an AI-driven economic crisis. This one isn’t about the fallout of an AI bubble popping because of a performance plateau. Instead, it’s a meditation on what happens if AI actually gets as good as we think it will. Basically, so bullish it’s bearish. The piece is from well-respected market research firm Satrini and is well-constructed and worth reading. And boy, did people read it. Nine million on the ExPost alone. Bloomberg, The Wall Street Journal, and many more wrote articles about the piece as the latest leg of the SaaSpocalypse cleaved billions off of tech and finance stocks. In other words, markets actually moved on a literal work of fiction. There is a ton of great debate to be had around the piece, which is of course happening right now, and which makes the outcome of them having shared it likely better in the medium and long Run than if they hadn’t shared it, even if DoorDash stockholders don’t really agree right now. I’m not really interested in a point-by rebuttal. What I do want to point out is that like most analysis on both the bear and bull side, it rests on an assumption so deeply embedded that almost no one questions it, that because markets Reward efficiency, efficiency is inevitable. This efficiency gospel isn’t exactly wrong, but it mistakes means for ends. And here is my main point. Markets don’t exist to be efficient. Markets exist to serve human preferences. Outside of the efficiency gospel, the value of efficiency is primarily in how it improves a company’s ability to serve human wants and needs, not an ends in and of itself. Confusing the two is like saying the point of a restaurant is great ingredients and a clean kitchen. Too much of the AI discourse on both bear and bull sides makes exactly this mistake. We’ve thought a lot about how much more efficient AI will make things, but too little about what we and other humans of the future are going to want. AI might make every part of a company’s operations more efficient, but will that company’s customers actually want to interact with the new, more efficient version on the other side? What are the chances that they actually reject it in favor of a more human version? Will they actually be willing to pay a premium for a less rawly efficient experience because they like that version of the experience better? Markets make this confusing. Investors are the high priests of the efficiency gospel, and the day-to excitement of market moves tends to lead more media attention to be focused on the stock story than the value Created to the end consumer. Indeed, for the market priests and priestesses, the value to the end consumer is actually secondary to the value to the shareholder, but that only lasts for so long. A company can live for a long time because the markets like it, but not forever. Ultimately, the buck stops with the customer. And when it comes to the customer, human institutions are not outcome-generating machines. At least not exclusively. In many cases, they’re also or even primarily agency-validating systems. There’s plenty of evidence to suggest that people are willing to pay for the possibility of being an exception. The chance that someone will look at your situation and deviate from the script. The knowledge that the person across the counter could break the rule for you, even if they don’t. Friction isn’t always waste. Think about all the ways capitalism has invented for us to transform the possibility of exception into the exception is the norm. Loyalty programs, status tiers, premium service. (Time 0:20:53)
  • Doom Loop Versus Demand Expansion Debate
    • The Citrini ‘2028 Global Intelligence Crisis’ sparked market panic by imagining extreme efficiency-driven demand collapse, but critics argue demand expands when costs fall.
    • Opponents (Noah Smith, Kobayesi, Citadel) stress historical elasticity of wants and that cheaper production often creates new consumption and industries. Transcript: Nathaniel Whittemore I’m not really interested in a point-by rebuttal. What I do want to point out is that like most analysis on both the bear and bull side, it rests on an assumption so deeply embedded that almost no one questions it, that because markets Reward efficiency, efficiency is inevitable. This efficiency gospel isn’t exactly wrong, but it mistakes means for ends. And here is my main point. Markets don’t exist to be efficient. Markets exist to serve human preferences. Outside of the efficiency gospel, the value of efficiency is primarily in how it improves a company’s ability to serve human wants and needs, not an ends in and of itself. Confusing the two is like saying the point of a restaurant is great ingredients and a clean kitchen. Too much of the AI discourse on both bear and bull sides makes exactly this mistake. We’ve thought a lot about how much more efficient AI will make things, but too little about what we and other humans of the future are going to want. AI might make every part of a company’s operations more efficient, but will that company’s customers actually want to interact with the new, more efficient version on the other side? What are the chances that they actually reject it in favor of a more human version? Will they actually be willing to pay a premium for a less rawly efficient experience because they like that version of the experience better? Markets make this confusing. Investors are the high priests of the efficiency gospel, and the day-to excitement of market moves tends to lead more media attention to be focused on the stock story than the value Created to the end consumer. Indeed, for the market priests and priestesses, the value to the end consumer is actually secondary to the value to the shareholder, but that only lasts for so long. A company can live for a long time because the markets like it, but not forever. Ultimately, the buck stops with the customer. And when it comes to the customer, human institutions are not outcome-generating machines. At least not exclusively. In many cases, they’re also or even primarily agency-validating systems. There’s plenty of evidence to suggest that people are willing to pay for the possibility of being an exception. The chance that someone will look at your situation and deviate from the script. The knowledge that the person across the counter could break the rule for you, even if they don’t. Friction isn’t always waste. Think about all the ways capitalism has invented for us to transform the possibility of exception into the exception is the norm. Loyalty programs, status tiers, premium service. The entire premium loyalty economy is a multi-billion dollar bet that people will pay for guaranteed access to generally favorable human discretion. A couple of years ago, I decided I was being stupid not to concentrate airline loyalty on a single airline and so pick Delta. We spent a fair bit of scratch on a Delta SkyMiles card, so I got to Diamond last year. Turns out there’s a special 24-hour line just for Diamond members. And man, have I put that thing to the test the last few days. Even as Reddit rages at 2, 3, 4, even 5-hour wait times with Delta in the wake of the blizzard, I’ve been able to get a real live human being on the phone in under a minute a half dozen times. The point is, Delta isn’t trying to automate the Diamond line. The Diamond line is the product. Automate that, and you’ve eliminated the thing people are paying for. I’m not trying to be Pollyannish about the magnitude of AI disruption. Anyone who listens to the podcast knows how enormous a change I think we’re living through, and how profoundly challenging this next middle part could be, even though I’m optimistic For the long term. But a big strand of the most urgent concerns are predicated not just on the scale of disruption, but the speed. This type of doomerism rejects comparisons to the past because the paradigm shift was more gradual while this one is happening everywhere all at once. The core question these arguments tend not to grapple with is, just because AI could do something, will it always be called to do so? If you live in the efficiency gospel, the answer is of course yes. If a non-human intelligence can perform the same task more efficiently, it will inevitably be tapped to do that task, at the expense of the human who used to do it. But efficiency is not destiny. Indeed, efficiency is only one type of market force. Humans have agency. Humans have purchasing power. (Time 0:22:47)
  • Design Around Customer Preferences Not Pure Efficiency
    • Don’t assume efficiency is destiny; prioritize customer preferences when designing AI-driven changes.
    • Whittemore advises companies to remember markets serve human wants, not efficiency for its own sake, and preserve valued discretionary experiences. Transcript: Nathaniel Whittemore What I do want to point out is that like most analysis on both the bear and bull side, it rests on an assumption so deeply embedded that almost no one questions it, that because markets Reward efficiency, efficiency is inevitable. This efficiency gospel isn’t exactly wrong, but it mistakes means for ends. And here is my main point. Markets don’t exist to be efficient. Markets exist to serve human preferences. Outside of the efficiency gospel, the value of efficiency is primarily in how it improves a company’s ability to serve human wants and needs, not an ends in and of itself. Confusing the two is like saying the point of a restaurant is great ingredients and a clean kitchen. Too much of the AI discourse on both bear and bull sides makes exactly this mistake. We’ve thought a lot about how much more efficient AI will make things, but too little about what we and other humans of the future are going to want. AI might make every part of a company’s operations more efficient, but will that company’s customers actually want to interact with the new, more efficient version on the other side? What are the chances that they actually reject it in favor of a more human version? Will they actually be willing to pay a premium for a less rawly efficient experience because they like that version of the experience better? Markets make this confusing. Investors are the high priests of the efficiency gospel, and the day-to excitement of market moves tends to lead more media attention to be focused on the stock story than the value Created to the end consumer. Indeed, for the market priests and priestesses, the value to the end consumer is actually secondary to the value to the shareholder, but that only lasts for so long. A company can live for a long time because the markets like it, but not forever. Ultimately, the buck stops with the customer. And when it comes to the customer, human institutions are not outcome-generating machines. At least not exclusively. In many cases, they’re also or even primarily agency-validating systems. There’s plenty of evidence to suggest that people are willing to pay for the possibility of being an exception. The chance that someone will look at your situation and deviate from the script. The knowledge that the person across the counter could break the rule for you, even if they don’t. Friction isn’t always waste. Think about all the ways capitalism has invented for us to transform the possibility of exception into the exception is the norm. Loyalty programs, status tiers, premium service. The entire premium loyalty economy is a multi-billion dollar bet that people will pay for guaranteed access to generally favorable human discretion. A couple of years ago, I decided I was being stupid not to concentrate airline loyalty on a single airline and so pick Delta. We spent a fair bit of scratch on a Delta SkyMiles card, so I got to Diamond last year. Turns out there’s a special 24-hour line just for Diamond members. And man, have I put that thing to the test the last few days. Even as Reddit rages at 2, 3, 4, even 5-hour wait times with Delta in the wake of the blizzard, I’ve been able to get a real live human being on the phone in under a minute a half dozen times. The point is, Delta isn’t trying to automate the Diamond line. The Diamond line is the product. Automate that, and you’ve eliminated the thing people are paying for. I’m not trying to be Pollyannish about the magnitude of AI disruption. Anyone who listens to the podcast knows how enormous a change I think we’re living through, and how profoundly challenging this next middle part could be, even though I’m optimistic For the long term. But a big strand of the most urgent concerns are predicated not just on the scale of disruption, but the speed. This type of doomerism rejects comparisons to the past because the paradigm shift was more gradual while this one is happening everywhere all at once. The core question these arguments tend not to grapple with is, just because AI could do something, will it always be called to do so? If you live in the efficiency gospel, the answer is of course yes. If a non-human intelligence can perform the same task more efficiently, it will inevitably be tapped to do that task, at the expense of the human who used to do it. But efficiency is not destiny. Indeed, efficiency is only one type of market force. Humans have agency. Humans have purchasing power. Even in the Citrini report, the white-collar labor folks aren’t out of consumer power yet. If human desire runs counter to efficiency, as it often does, there’s every reason to think that the old maxim that the customer is always right will provide a serious counterweight To the unstoppable market advance of the machines. (Time 0:22:50)
  • Schrödinger’s Apocalypse Captures AI Uncertainty
    • Enormous uncertainty about AI’s macro effects makes many debates more literary than analytic, so remain ready to change views quickly.
    • Whittemore cites Derek Thompson’s ‘Schrodinger’s apocalypse’ framing: simultaneous plausible futures of dramatic change and near-normalcy. Transcript: Nathaniel Whittemore But a big strand of the most urgent concerns are predicated not just on the scale of disruption, but the speed. This type of doomerism rejects comparisons to the past because the paradigm shift was more gradual while this one is happening everywhere all at once. The core question these arguments tend not to grapple with is, just because AI could do something, will it always be called to do so? If you live in the efficiency gospel, the answer is of course yes. If a non-human intelligence can perform the same task more efficiently, it will inevitably be tapped to do that task, at the expense of the human who used to do it. But efficiency is not destiny. Indeed, efficiency is only one type of market force. Humans have agency. Humans have purchasing power. Even in the Citrini report, the white-collar labor folks aren’t out of consumer power yet. If human desire runs counter to efficiency, as it often does, there’s every reason to think that the old maxim that the customer is always right will provide a serious counterweight To the unstoppable market advance of the machines. Safetyists have long advocated some type of pause to allow us more time to adapt. I think we might be underestimating the extent to which human consumer preferences will do that all on their own. It’s entirely possible I’m wrong, and the forces of the efficiency gospel are too strong to resist. But I’m on hour 30, or 40, or 50, or who knows by the time you’re reading this, stranded in who-knows Brazil with two small kits. AI as information guide has been amazing, but exactly zero times have I wished I could have a more efficient AI to interact with. What I’ve wanted was a human being who looked at our situation and decided to break the rules just a little to help us get home. Efficiency is not destiny. And ultimately, and now I’m done reading myself and back to just talking as myself, the thing to note here is just that as compelling as all of these arguments sound, as many holes as there Are in one, as many better points in another, the reality is that we are all just grasping and guessing at a future that we cannot know. Abundance author Derek Thompson writes, The level of uncertainty is so high and the quality and supply of real-world, real-time information about AI’s macroeconomic effects so Paltry that very serious conversations about AI are often more literary than than genuinely analytical. I feel lucky to have been able to have conversations about the frontier of AI with executives and builders at Frontier Labs, economists at AI conferences, investors in AI, and other AI folks at off-the dinners, where important truths can theoretically be shared without risk. I can’t emphasize enough that nobody knows anything is about as close to the reality here as three words are going to get you. Nobody knows what’s going to happen this year, or next year, or the year after that. There is no secret cigar-filled room of people who have unique access to some authentic postcard from the future. When you drill down underneath the bluster, the boomerism, the fear, the anxiety, what’s there at the bottom is genuine uncertainty, a vacuum into which storytelling is flooding. The frontier labs don’t really know what they’re building exactly. The economists don’t really know how to model the thing they’re claiming they’re building. I wish more people talked about and thought about this subject through that sort of lens. We’re trying to model the economy-wide effects of a technology whose properties the frontier labs can’t even really describe yet. (Time 0:25:54)