Skip to content

Podcast

OpenAI’s Big Reset + A.I. In the Doctor’s Office + Talkie, a Pre-1930s LLM

Hard Fork

Source ↗ ← All highlights
  • OpenAI Loosened Microsoft Deal To Unlock More Cloud Demand
    • OpenAI’s Microsoft reset mainly frees it to sell through other clouds, easing a compute bottleneck that had constrained growth.
    • Kevin Roose says customers on AWS or Google Cloud can now use OpenAI models without moving to Azure, while the old AGI revenue clause disappears through 2030. Transcript: Kevin Roose So Microsoft and OpenAI have, of course, been partners for many years. Microsoft remains the biggest investor in OpenAI. Their stake is valued at about $135 billion. But their relationship has also been strained over the years by various factors. And this week, they seem to be sort of consciously uncoupling or at least rewriting their partnership agreement and allowing OpenAI to be a little bit more promiscuous in who they do Deals with. Casey Newton Yeah, I mean, OpenAI just had this real challenge, which was that until this week, they were really only allowed to serve their models on Microsoft’s infrastructure. And one thing we talk about on the show a lot is just that a lot of the big cloud service providers, their infrastructure is just maxed out. And Microsoft is one of those. And so for OpenAI’s revenue to grow, they needed to find other ways that they could deliver their services. And so to my mind, that was maybe the most important thing about this deal. Kevin Roose Yeah. So under this new rewritten version of the Microsoft and OpenAI deal, Microsoft will no longer have to share revenue with OpenAI. The new deal also removes the part of the original agreement that had to do with AGI. The old agreement said that basically once OpenAI reached AGI, Microsoft would stop getting certain revenue share payments. But under the new agreement, OpenAI will keep sharing revenue with Microsoft until 2030, no matter what benchmarks they hit. So the AGI clause is gone. Casey Newton And I, for one, will be sad to see it go because I think it was sort of the funniest clause in the entire AI world, right? It was basically like, well, if we ever get to a point where OpenAI says the magic word, then the entire world changes, and now they’re not allowed to say the magic word anymore. Right. Kevin Roose So AGI has been poorly defined for many years, and everyone’s got their own definition. But it did have this one interesting, like, contractual stipulation. And now even that is off the table. So now we just have sort of AGI as evaluated by Vibes. So Casey, what did you make of this loosened partnership between OpenAI and Microsoft? Casey Newton Well, I think it seems like probably a good deal for both of them, right? Like there was a moment when it seemed for both companies, like being very, very closely aligned was the best thing for both, arguably for a time it was. But with all of the various revenue, compute, and customer needs that both of these companies are now trying to serve, I think it’s benefiting both of them to play the field and sign other Partnerships. So my read on this was like, this is basically good for both of them. But what about you? Kevin Roose Yeah, I think it’s good for both of them. I think it’s a little better for OpenAI. They got most of what they wanted. I think the bigger deal for them is the ability to work with other cloud providers. So now they can work with Amazon or Google Cloud Platform. And big corporate customers who use those cloud platforms can now use OpenAI models. They don’t have to go to Azure to do that. I think that allows them to strike these other bigger deals and to reach other corporate customers who may have been limited before by the fact that it’s really hard and annoying to change Cloud providers. Yeah. Casey Newton And speaking of big deals, Kevin, (Time 0:03:24)
  • AI Bubble Talk Shifted From No Demand To Scarcity
    • The AI boom’s skeptic narrative flipped from weak demand to too much demand for available compute.
    • Casey Newton notes even the biggest firms lack infrastructure to serve usage, while Kevin Roose says critics recently doubted anyone would fill those expensive data centers. Transcript: Casey Newton By the way, I think that this is a really important point and a reason that we are like talking about it to a big general audience, which is that the story that you just described, Kevin, Is one of a world where no one has the resources they need to serve the demand for AI that they have. Where we’re still sort of seeing a lot of skepticism. There’s so much bubble talk. I just want to like posit that as a really important point in understanding what sort of bubble this is, because even the biggest companies do not have the resources that they need to Serve that demand. Kevin Roose Yeah, and I think that’s a good point. And it’s really a profound shift in the way that skeptics have been talking about this AI boom. I remember just even a couple of months ago, the leading sort of strain of criticism was that these AI companies would never be able to generate the demand to pay for all of the expensive Data centers and infrastructure projects they wanted to do. And now that’s shifted to, well, there’s so much demand. What if they can’t build enough to support the demand they have? (Time 0:08:18)
  • Stargate Retrenchment Looks Like IPO Prep
    • Stargate’s slowdown looks less like retreat and more like OpenAI moving capital-heavy projects off its own balance sheet.
    • Kevin Roose says leasing third-party capacity helps OpenAI look cleaner for a future IPO after halting or shrinking several planned data-center expansions. Transcript: Kevin Roose This was another story that hit this week. The Financial Times reported that Stargate, OpenAI’s joint $500 billion infrastructure project, is also undergoing a bit of a shift. The FT reported that in recent weeks, OpenAI has halted planned data centers in the UK and Norway, declined to expand its flagship site in Abilene, Texas, and seen several senior figures Tied to Stargate leave for rival meta. The FT further notes that OpenAI has shifted to leasing capacity from third parties instead of building out all of their own facilities. Casey, what did you make of this? Casey Newton I think this was a case where, like, reality has just finally intruded on the Stargate project. Like, when all of these deals were getting announced initially, this is how they sounded. Well, we’re going to spend one trillion dollars that we don’t have to build 40 quadrillion data centers. And at the time, people said, that kind of seems like a lot. Can you guys actually live up to that? And they said, yeah, just watch us. Well, guess what? They could in another changing course. Kevin Roose Yeah, I don’t think this signals that they are sort of retreating from their compute ambitions. I think it’s more about like they are realizing that if they want to go public, which they do, they need to sort of get their house in order. And one way to get your house in order is to move some of this data center and infrastructure building off of your balance sheet and on to third parties. (Time 0:09:37)
  • AI Firms Are Torn Between Vision And Financial Reality
    • AI companies are split between believers in near-infinite demand and finance teams trying to justify spending to cautious investors.
    • Kevin Roose says OpenAI’s internal tension reflects an industrywide struggle between world-changing optimism and revenue models that actually pencil out. Transcript: Kevin Roose I mean, I think there are competing forces within all of the big AI companies right now. One side is sort of the indefinite optimists, the people who think that demand for AI is just going to be essentially infinite and that as much compute and as much money as they need to Spend acquiring compute, it will all be paid back many times over because the world is about to change into something most of us barely recognize. And so kind of just trust us on that is sort of one camp. And then there are the sort of, you know, the number crunchers who are trying to fit all of this into a kind of financial projection that will make sense to investors who are not as convinced That the world is about to change forever and who want to see things like, what is your plan for actually making the revenue that you’re going to need to pay for all this stuff? So I think this is happening in a way at OpenAI that is now, because of Berber’s story, is out there. But I think this kind of tension exists at all of the big AI companies. And so I think right now what we’re seeing is kind of that power struggle breaking out into the open. Casey Newton Yes. And for what it’s worth, OpenAI did call this story prime clickbait, which I think just refers to clickbait that’s really, really good. Is that what that means? Yes. It’s sort of like Wagyu clickbait. Kevin Roose Yes, exactly. This clickbait was dry-aged for a month before it was served, and it’s delicious. Yeah, and I think one thing I want to flag on this is that these growth projections that OpenAI reportedly did not hit, those were in 2025. I think it is fair to wonder if something has changed in just the last few months because of the enormous rapid growth of tools like Codex and CloudCode. We have seen just reports of astronomical growth in those tools. So it may be that OpenAI was having some growth issues late last year, but that because of this agentic coding boom, things have started to turn around. (Time 0:11:38)
  • Chatbot Subscriptions Are Splitting Into Two Markets
    • Chatbot businesses are bifurcating into cheap mass-market tiers and premium plans for professionals who extract far more value.
    • Kevin Roose says casual users may accept $8 or ad-supported access, while heavy users will pay much more for stronger models and higher limits. Transcript: Casey Newton This week, the information had this really interesting story where apparently OpenAI projected at the start of the year that it’s $8 a month subscription, which is called ChatGPT Go, which sort of gives you a little bit of the good stuff, but not as much as if you’re paying $20 or more for ChatGPT. They predicted that its Go subscriptions would grow 36 times this year to 112 million people. While meanwhile, its $20 a month plus subscriptions would fall 80% to about 9 million. So that’s like a really interesting business pivot that I would love to know more about. Of course, it sounds a lot like the new Netflix plan that they rolled out a while back, right? Where it’s sort of like, well, you know, it’s going to be a lot cheaper, but we’ll show you ads. I was curious, like what you make of that strategy? Because, you know, part of me feels like, well, they’d much rather have, you know, the $20 subs and the $8 subs, but maybe there’s just a lot more of those $8 subs out there. Kevin Roose Yeah, I think what’s happening here is that the market is essentially splitting in two, right? There’s the sort of casual hobby users who are using AI chatbots like ChatGPT, like Claude for sort of souped up Google queries to, you know, help them write emails and maybe only using It a couple times a day. And if you’re doing that, you probably don’t want to pay 20 bucks a month. You’re probably more comfortable paying eight bucks a month, or maybe you don’t want to pay anything at all, and you’d just rather use the free ad-supported tier of all of this stuff. And then there’s the professional users for whom this is worth way more than 20 bucks a month and who are willing to pay many multiples of that to get access to the latest models, to have Higher rate limits. And so I think all of the companies now are sort of, you know, doing this kind of experimentation with how much can we charge the professional users without losing them to a rival company? And how cheap can we make the kind of lower end subscriptions or the free tiers so that people who are more casual users won’t be tempted to go use Google instead. (Time 0:13:52)
  • Elon Musk’s Trial Threatens Headaches More Than Collapse
    • Elon Musk’s case against OpenAI may be more disruptive than existential because donor control claims against nonprofits are unusually weak.
    • Casey Newton says only two of Musk’s original 26 claims survived, though a win could force more than $150 billion back under nonprofit control. Transcript: Kevin Roose Casey, can you remind us what this case is about? Yes. Casey Newton So Elon Musk was famously one of the co-founders of OpenAI. He gave the company some of its initial funding, but left in a power struggle between himself, Sam Altman, Greg Brockman, and some others. And a few years after all of that went down, and notably after Elon started his own AI company, he sued OpenAI and said, I have been defrauded. This was only ever supposed to be a nonprofit, and you’ve gone and turned it into one of the world’s most valuable companies through its for-profit arm. So he is suing to stop all of that. If he wins, any winnings will be given to OpenAI’s nonprofit arm. Notably, Kevin, he made 26 claims when he originally filed this lawsuit in 2024, but only two have survived to trial unjust enrichment and breach of charitable trust. Kevin Roose So the trial is just getting underway. They’ve done jury selection and they’ve had a couple witnesses testify. Elon Musk himself took the witness stand on Tuesday and said, quote, this lawsuit is very simple. It is not okay to steal a charity. He also said that if OpenAI is allowed to get away with this, quote, it will give license to looting every charity in America. Basically, he is saying this thing that started as a nonprofit that was supposed to continue as a nonprofit became through some corporate restructurings, a for-profit company that Has raised many billions of dollars, and that if this is legal to do, every charity would do this. Why wouldn’t you want to take your donors’ money and turn yourself into a well-funded startup? Yes. Casey Newton Now, one inconvenient truth that Elon Musk faces here, which is that OpenAI’s for-profit business is still controlled by a nonprofit. There’s this foundation that houses the public benefit corporation. And while I do empathize with those who say, hey, it really seems like the nonprofit hasn’t done all that much. And you know, most of their money is being used for for-profit activities. This was litigated and the nonprofit, you know, still does have like voting control over the for-profit. Yeah. Kevin Roose So Elon Musk is saying this is a case of looting a charity. OpenAI’s lawyers have accused Elon basically of just being bitter that the company has succeeded without him. Its lead counsel, William Savitt, said during the trial, quote, we are here because Musk didn’t get his way at OpenAI. My clients had the nerve to go on and succeed without him. Mr. Musk did not like that. They have also been pointing out that Elon had also wanted to make OpenAI, have a for-profit subsidiary back when he was with the company, and that he’s just mad that he didn’t get to control It. Casey Newton Yeah, and to underline that, like in 2017, 2018, there are emails from Elon Musk where he talks about turning this into a for-profit. So, you know, whatever concerns he had about looting the charity, you know, today, like he did not have them back at the time. Kevin Roose Right, he also wanted to fold OpenAI into Tesla. That was revealed in some of these emails. Tesla, of course, being a for-profit company. So it seems like this is not exactly a consistent and principled stand. But Casey, what are the stakes here? Like if Elon Musk does manage to convince a jury that this was a case of OpenAI looting a non-profit for its own commercial gain. Like, what could the remedies be? Could this be fatal for OpenAI? Or is this just sort of an attempt to slow them down and distract them with a big trial? Casey Newton I think that it is much more the latter. Like, based on my reading of the case and what I’ve seen sort of legal experts say about it, the whole case is very unusual that it even made it to trial. Like for the most part, if you donate money to a nonprofit, you actually don’t have a say in what happens to it after that. So it’s very unusual that the judge even granted him standing to sue here. And as I noted, she threw out most of his claims. That said, let’s say that, you know, there’s some single digit percentage of him winning something here. What he wants to do is to take more than $150 billion that is currently under the control of the for-profit business and give that back to the nonprofit, which would create a lot of headaches And roadblocks for OpenAI as it tries to build out Stargate and do everything else it wants to do. (Time 0:17:05)
  • Top AI Labs May Rise Together Instead Of One Winning
    • AI competition is not necessarily zero-sum because several top labs may rise together as adoption expands.
    • Kevin Roose argues that if a company stays in the model top tier, broader AI uptake could lift multiple boats rather than crown one winner. Transcript: Kevin Roose Yeah, I think the lawsuit and this ongoing litigation between Elon Musk and OpenAI has been very distracting for OpenAI. But like as a journalist and as a person who wants to know more about the inner workings of how these companies run, I think it’s been actually very valuable for a lot of these emails and Early communications between open AI leaders to be released as part of this litigation. I have found it very useful in understanding some of the early dynamics at open AI. And it also just illustrates the degree to which these projects are all just sort of fueled by grudges, right? There’s sort of one level of interpretation, which is like all of these people are just like obsessed with building the machine god and that this is all sort of related to their visions Of the future. And then there’s like another more base level, which is just like these people are all just rivals and they have these petty longstanding grudges and they just don’t like each other Very much. And so you can interpret a lot of what happens in AI through the lens of personal animus. Casey Newton Yes, I’ve said this before and it is rude, a shocking percentage of the AI industry is just people who decided they didn’t want to work with Sam Altman and who now have their own companies. Right. Kevin Roose So, Casey, some people have been looking at all of this drama and intrigue surrounding OpenAI, from the trial to the Microsoft deal to these missed growth projections and saying some Version of like, OpenAI is in trouble. They are not going to make it to an IPO. They are going to sputter out and maybe end up in some real hot water. And maybe Elon Musk wins this trial. And it’s sort of the end of open AI as we know it. What do you make of those gloomy predictions? Casey Newton Yeah, I mean, look, there are some fundamentals for open AI that remain worrisome, right? They’re planning to burn tens and tens of billions of dollars in cash before they achieve profitability. They still have this very ambitious infrastructure build out that is quite expensive. And so like, I’m not going to sit here and say that like all of the numbers seem to pencil out for this company. On the whole, like, if I try to, you know, put myself into the shoes of their CFO, and I look through all of the stories that we just talked about, I think these seem like smart things to me, You know, it kind of seems like they’re starting to dot their I’s and cross their T’s and get this company in a shape where retail investors will be excited to invest in the stock, which, By the way, I think they will be. So, yeah, it’s one of these companies where, like, it is a generationally weird enterprise. But when I look at this particular set of stories, I think they’re basically doing the right thing. What do you think? Kevin Roose Yeah, I mean, I think there’s this interesting fallacy in the AI industry where it’s like, there will be only one winner, right? Everything is zero sum. If OpenAI is having a bad month, it’s because, you know, Anthropic is having a good month or Google Demind is having a good month and vice versa. Like their sort of growth comes at the expense of all the others. And I think that’s, that feeling is shared by among others, the executives of these companies. But I just don’t think it’s true. Like, I think that there are going to be a handful of companies that are just going to kind of rise and fall together, right? That if your models are in the sort of top tier, you are going to be fine as long as they stay in the top tier and the sort of rising tide of AI adoption will sort of lift all boats. (Time 0:21:25)
  • AI Became Routine In Medicine At Unusual Speed
    • Adam Rodman says AI in medicine may be the fastest-adopted medical technology ever, moving from novelty to weekly routine in under two years.
    • The biggest mainstream uses are AI scribes that draft notes from conversations and decision-support tools like OpenEvidence. Transcript: Kevin Roose Of the show. You are a primary care physician. So when we last talked to you in late 2024, I think this was a moment where the medical community was starting to say, wait a minute, these AI models are getting pretty good at things like Diagnostics. But I think a lot of the field was still kind of in wait and see mode. Now, almost two years later, we have a lot of new tools and a lot of new studies about the use of AI in medicine. So just catch us up on what has been going on with AI in medicine for the last year and a half. Adam Rodman Yeah, it’s been crazy. AI and medicine has gone from, well, depending on how you measure it, it’s probably the fastest adopted medical technology of all time. We went from this being super novel, almost no one used AI tools, to this being a routine part of most doctors’ weekly practice. Casey Newton And give us a sense of, like, the AI stack for a doctor. What are the tools that they are using right now and how? And particularly, like, what are the mainstream doctors, like the people that, you know, aren’t yet on the bleeding edge? The normies? Yeah, if you will. Adam Rodman Yeah, so the biggest sort of normal doctor technology, which most of your listeners or a good portion of your listeners have encountered, are what are called AI scribes. That’s a sort of voice-to algorithm that listens to you talk to your patients and then writes a first draft of your note. And these have gone from like kind of a novel experimental technology to commodity in probably less than two years. They’re everywhere. Doctors really like them and then patients really like them because they spend more time talking. And then the second sort of, I’d say, normal doctor use case is for decision support. So there’s this one company called Open Evidence that has created a free tool that has gone from, again, zero to crazy numbers of adoption. I will tell you, younger doctors like my residents use it all the time. I don’t know the actual numbers, but it’s probably close to half of U.S. Doctors are using this right now. (Time 0:29:20)
  • OpenEvidence Went From Niche Tool To Constant Companion
    • OpenEvidence has spread so fast that Adam Rodman says younger doctors use it constantly for second opinions, next steps, and fast evidence lookup.
    • Kevin Roose cites the company’s claim that doctors consulted it one million times in a single 24-hour period. Transcript: Kevin Roose Wow. So yeah, the statistics that I’ve seen are that more than 40 percent of doctors now are using this, which is pretty crazy uptake for something that was just started a couple of years ago Back in 2022. In March, Open Evidence reported that in a single 24-hour period, doctors consulted the AI system a million times. I’ve been fascinated by Open Evidence. I’ve never used it myself, but I have friends who are doctors or nurses, and they have said what you’ve said, that basically just everyone, especially on the younger end of medicine, Is just using this thing constantly. So give us a sense of, like, how this open evidence tool works. Like, what situations is it used for? And what are its strengths and weaknesses? Adam Rodman Oh, that’s a great question. So how open evidence works, like, all of these tools is a trade secret. But it uses some sort of, like, retrieval augmented generation and an evidence retrieval tool. And they have all these deals with the big medical journal. So New England Journal of Medicine, JAMA. And when you ask a clinical query, it searches the evidence and then tries to identify high quality sources. And then it always grounds what’s coming back in the literature. So you have gray hairs like me who kind of use open evidence the way that I would use a Google search or one of the old tools. So I use it as a souped up way to search the literature rapidly and often go to the primary sources or I use it as a faster way to get a reference. So a drug that I haven’t dosed in a long time, open evidence pulls the drug monographs from the FDA. I can very quickly pull that up. Younger doctors, I have noticed, and I don’t know this empirically, but younger doctors are more likely to ask questions like, what could be going on? Can you give me a second opinion? What is the next thing that I should do? So ways that I don’t traditionally use decision support or reference tools, but sort of a new way. And of course, younger doctors also use it in the reference ways that I do. (Time 0:31:21)
  • Doctors Like AI More When They Bring It Themselves
    • Doctors seem unusually positive about AI partly because many tools are self-adopted rather than imposed from management.
    • Adam Rodman says physicians are in a BYO AI phase, with some using plain ChatGPT, Gemini, or Claude for decision support. Transcript: Casey Newton How doctors are feeling about all of this. We saw a survey from the American Medical Association that found that more than 80% of physicians now report using AI professionally. Is that physicians racing out and grabbing these tools and bringing them to the office because they’re so helpful? Or is this the classic case of a CEO saying, hey, you got to use AI or you’re out of here? Adam Rodman So doctors are BYO AI. A lot of that AI use is AI scribes and decision support software. And I’ll tell you, some people are just using straight up like ChatGPT or Gemini or Claude for the decision support software. So I think one of the reasons doctors thus far have been more positive about it than perhaps the overall population is they’re largely tools that doctors are bringing themselves that They think make their lives better, and at least not yet many things that are being imposed upon us. (Time 0:35:00)
  • Use Health Chatbots For Prep Not For Treatment Decisions
    • Use chatbots for general health education, visit prep, and symptom exploration, but do not let them make treatment or medication decisions.
    • Adam Rodman’s green-yellow-red framework allows diabetic meal planning and question prep, yet warns against using LLMs to judge chemotherapy or doctor-prescribed management. Transcript: Casey Newton Give us a flavor of what you’re telling them because, you know, I am definitely somebody who has looked up my symptoms before I’ve gone to the doctor, and I would say I found it enormously Helpful. But I can also imagine, you know, more skeptical doctors being annoyed, you know, at a patient telling them, you know, what ChatGPT says to do. Adam Rodman So, yeah, so here’s my, I’ll give you my spiel. This is, I give them a, what is it? A green light, yellow light, red light. So the green light uses are general health questions. So I have recently diagnosed with diabetes. I really love seafood. Can you help come up with a diabetic diet for me? The green light uses are also preparing for clinic visits. So I’m about to go see Dr. Rodman. I want to make sure that I ask the right questions. Here is the last note or the last thing he wrote. Obviously, strip out anything identifying. Don’t put your personal health information. And like help come up with a good questions to ask him. And then other green light activities might be like wearable data. I don’t know how good they are at wearable data, but I will tell you if a patient is going to give me like five years of their Apple Watch data, they’re probably going to get better from ChatGPT Than from me pretending to look at five years of Apple Watch data because it’s a 20-minute visit. The yellow light, Casey, I think is a lot of the things that you’re saying. So I tell my patients it’s okay to explore new symptoms. It’s even okay to seek out second opinions when talking to a chatbot that can really help prepare you, as long as you understand that it’s not a replacement for a doctor, and that is the First step to talking to a human being. So LLMs are really powerful, and I mean, there is some evidence, of course, that when any human uses them, you don’t always get laboratory-level performance. They can give you dangerous advice. But diagnosis and exploring symptoms, as long as you use it in a way to prep to see your doctor, can be very helpful. The red light, what I tell them never to do, is ask medical management decisions. Don’t say, my doctor said to do this. Is this right? I have cancer. God forbid you have cancer. Is this the right chemotherapy option? A lot of those decisions are so nuanced, taking so much information. Those are things that the models don’t do well. And they’re so sycophantic, they can convince you that they’re saying the right thing even when they’re wrong. (Time 0:36:52)
  • Chatting With Your Medical Record Is Not Yet Smart
    • Dumping your full medical record into an LLM does not automatically produce better health advice because records are messy, contradictory, and error-filled.
    • Adam Rodman says records mix tables, copied-forward notes, and outright mistakes, so privacy risks rise faster than practical value. Transcript: Adam Rodman Not yet, but I think it could be at some point. I mean, so ChatGPT for health pulls in your data from the medical record and lets you chat with your medical records. Now, reason number one for concern is privacy. That’s obviously going to have your entire medical history going to an AI company. It’s also going to not be redacted by you in a way to remove identifiable things. Reason number two, I think if we’re talking about health record data, it’s really messy. They include tabular data. They include copy forwarded data that’s been copied and pasted. And they also, if you’ve ever read your health records, they include things that are wrong. There’s a lot of errors or misdocumented things in your health data. And it turns out that just copying a bunch of information, like, LLMs aren’t magical. You can’t just copy your entire medical record in and think that you’re going to get good performance. And I would never bet against the technology. I think that we will get to the point that we have ways to build representations of humans and understand their health. But right now, there’s like no advantage to just dumping everything in an LLM, which is what ChatGP2 for Health theoretically would allow you to do in a way that would allow you to better Understand your health. (Time 0:40:37)
  • AI Refill Trials Do Not Justify Autonomous Prescribing
    • Autonomous prescription renewals are a narrow proof of concept, not evidence that AI should write new prescriptions on its own.
    • Adam Rodman says Utah’s trial only refills existing low-risk drugs, while real follow-ups can catch dangerous side effects like Stevens-Johnson syndrome. Transcript: Casey Newton Well, so globally, no, we should not be having LLMs write prescriptions for people. Adam Rodman The trial in Utah in particular is a refill. So a doctor has already written a prescription within the last 12 months. And I guess the idea is that it saves the primary care doctor time from having to review and refill. I’ll tell you, if you talk to most doctors, yes, it is annoying to get refill requests. No, that is not the thing that drives us crazy. This is not like a use case that we’re screaming for. I think it’s being done as a proof of concept of can this work in the real world. This trial in and of itself is not dangerous. Prescription refills, and I think there’s no opiates. There’s no dangerous drugs in it. And a doctor has to have written the original one. But even if it does work in this, that does not mean we should be having autonomous AI systems write new prescriptions. That is not safe, and it’s not a good idea yet. Kevin Roose See, I think this is a case where like this is sort of rent-seeking behavior on the part of doctors or doctor organizations. Like when I have gotten refills for prescriptions, I meet with a doctor for, you know, six to eight minutes. They say, how’s it working? I say, great. They say, are you having any side effects? I say, no. They say, okay, I’ll write you a refill. And the whole process just seems totally designed to like get me to pay up for another doctor visit and not give me any actual good medical advice. So if I can play devil’s advocate, like, do you think that the sort of resistance to programs like this are motivated by just wanting to keep people coming to the doctor and paying for Those visits? Adam Rodman So first, aren’t most of your prescription refills just done as in you call the pharmacy and they send an automated thing to your doctor and they click the yes button and you never talk To them? Kevin Roose No, for some, they make you actually do an office visit and maybe they want you to take your blood pressure again or whatever. Adam Rodman So I’ll do the devil’s advocate back. Let’s say I prescribe a fairly common antidepressant and they want it to be refilled. What I don’t know is that this patient may be, the silly question you get in the clinic, may be new lesions forming in your mouth. And it’s an early ulcer. And if we don’t pick it up within 24 to 48 hours, you may develop like Steven Johnson syndrome. So potentially life-threatening complication. And the reason there are certain types of drugs, including antihypertensives, is that they can be high risk and we need follow-up. Now, is that everything? No. And definitely there should be more things over the counter. I don’t think that most doctors are sitting around saying, I wish I had more medication follow-up visits. And the reason some of these things exist is that there can be very dangerous symptoms. (Time 0:42:05)
  • AI’s Biggest Health Gains May Come From Basic Care
    • AI may improve life expectancy more through mundane care access and better screening than through flashy miracle drugs.
    • Adam Rodman points to tools for earlier cancer detection and broader diabetes or stroke prevention rather than betting first on AI-discovered CRISPR breakthroughs. Transcript: Casey Newton Let me ask you about another one. This one actually seemed like just an unqualified good. The Mayo Clinic announced this week RedMod, this AI system that identified subtle changes in routine CT scans up to three years before a pancreatic cancer diagnosis. And this was like many, many, many percentage points better in detecting pancreatic cancer than human beings. So to me, this is like the sort of thing I keep waiting for AI to do. And it seems like it’s actually doing it. And of course, that’s very exciting with something like pancreatic cancer, which is notoriously difficult to detect and has like very low survival rates. Yeah. Adam Rodman And this is so like completely out of the discourse of like autonomous AI agents. There’s really exciting stuff happening. So the Mayo Clinic, there have been some great studies on breast detection. A lot of these algorithms have gotten so good that they’re able to identify breast cancer better than, I shouldn’t say better than people, but in a workflow that has a good detection Rate. And then in picking up like potentially cancerous polyps when you get a colonoscopy. So there’s a lot of exciting and really positive things that are coming. And I’m, I mean, at the end of the day, we’ll need to see how RedMod works in the real world and a trial. But I’m really optimistic about that sort of technology. Kevin Roose Do you think that if AI meaningfully extends life expectancy for people, it will be because of new AI discovered drugs or because of changes to routine healthcare that are made more Efficient or more accurate by AI? Adam Rodman Number two, I think that when you talk about AI drug discovery, the part of the pipeline that’s so difficult is not necessarily the coming up with the new compounds. It’s running the clinical trials and getting it through the regulatory process, which can probably be sped up, but not as much as the discovery. You know, if we get this right, there’s so many people in the US who don’t have access to a doctor, who don’t have access to very basic medications, who can’t control their diabetes because Of lack of access. And I’m really hopeful that if we do this wisely, we can, you know, get people more access to care, which, I’m doing my knock on wood, hopefully will improve health outcomes. (Time 0:44:51)
  • Medical AI Could Erode Skills Before It Fully Replaces Them
    • The biggest near-term risk in medical AI may be de-skilling doctors during training, not robots replacing them outright.
    • Adam Rodman cites a Poland study where colon-polyp detection fell six percentage points after three months of relying on assistance technology. Transcript: Kevin Roose There’s a lot of worry right now about sort of AI in schools and education, some of the cognitive atrophy that people are worried about. Oh, if we start using AI to do all of our work, we’re not going to have the basic skills. Is that something you’re worried about for like recent medical school graduates where maybe they would have had to hold all this stuff in their brains a few years ago and now they can Just ask a chatbot and maybe that’s going to erode some of their skills as a physician? Yes. Adam Rodman So that is the biggest worry that I actually have about sort of the short to medium term de-skilling of the workforce. We have some evidence. There was a sort of scary study last year from Poland on a trial where they gave doctors, not a language model, but a polyp detecting technology. And they looked at their ability to detect polyps, so potentially cancerous lesions in the colon before using it, and then after using it for three months. And when not using it, their ability to detect polyps dropped by six percentage points. So these are skilled doctors using technology, and they lose six absolute percentage points of their ability to detect potentially cancer in three months. And then imagine that you’re learning to do it for the first time. Will you ever gain those skills? So like at Harvard Medical School, and medical schools I think everywhere, this is our big worry, which is how will this affect us to train the new generation of doctors? And it’s like every other field. Like you talk about debugging code. In order to become a new doctor, you go through all this training because you need to make mistakes and you need to have someone above you who knows what’s going on so those mistakes won’t Hurt patients. And that’s just how education works, and this threatens that. Casey Newton I mean, it’s interesting, though, because it’s like, you know, it’s probably true that because I had access to a graphing calculator, like if you took it away from me, I’d be worse at Like plotting parabolas on a graph. But the solution to that is that I just keep using the calculator, you know? Adam Rodman So like, I’m not sure how big of a problem this really is. I’ll also say there’s something, there’s something deeply ingrained in human society that middle-aged people complain about young people. So I think whenever we talk about de-skilling, we have to keep that in mind. Kevin Roose I want them to be consulting the hive mind before they weigh in on my specific condition. It doesn’t threaten me as a patient to know that they are using open evidence or something similar. But I’m guessing for a lot of people, that would seem strange. And maybe there are some physicians who don’t advertise how much they’re using AI because their patients might think less of them. Do you think that’s happening? Adam Rodman Oh, yeah. I think there are, in certain situations, in certain places, I bet there’s social pressure to say that you’re not using AI, that there’s some ego on the line. I don’t see that. But again, I’m an AI researcher, so I don’t think anyone would say that to me. Casey Newton To me, it just seems weird to hold as your standard for what makes a good doctor that they have memorized a maximum amount of material, right? Like, that’s basically what we’re talking about. It’s sort of like, you know, the taxi drivers in London that have to, like, learn every single street and, like, hold them all in their heads. It’s very impressive, but I’m fine with them using the GPS. Yeah, and I think it’s less about, so it is about memorizing. Adam Rodman It’s more at this point, right, with where AI is now, it’s more having sort of that knowledge and we’ll call it wisdom to know when the system might be suggesting something wrong, which Is something that right now, and this may change, we get by seeing a lot of cases and reflecting on them. So right now, you’re going to get the best performance if you have an experienced human trained in the old-fashioned way with an AI system. But I think your guys’ point is at some point that might not matter. The AI systems might just outperform all of us. (Time 0:48:00)
  • Better Medical Models Likely Need More Health Data
    • Strong medical privacy protections likely slow model improvement because high-quality labeled health data is exactly what these systems need.
    • Adam Rodman says better medical LLMs require training or evaluation on health data, though he stresses patient privacy and ownership remain important. Transcript: Kevin Roose The AI models be better if we were less protective of privacy for medical data? I mean, that’s such a loaded question. Adam Rodman So the first thing that I’m going to say before I answer that is patient privacy is very important. And we should respect people’s privacy and their ownership over their data. But yeah, so in short, like the reason they’re not better at certain things is that you need to get LLMs better, you need to label and then train them on the sort of labeled health data. And there’s all in the US, there are appropriately many restrictions on how health data can be used. I suspect that these companies like OpenAI, by having ChatGPT for health, they will gain some more of their own data, which they say they’re not going to train on. I trust that they’re not going to train on it, but they’ll be able to use that data to at least evaluate their models and try to make them better. Kevin Roose I think they should train on it. I mean, obviously, that’d be a huge illegal violation of privacy, But it would also make the AI doctors better. Much better, yeah. And I think a lot of people would be sort of willing to make that tradeoff. So I at least think there should be a little checkbox when you go to the doctor that says, like, I’m okay having my personal health data used to train AI models. Casey Newton I for one would check it. Yeah, in exchange for like 30% off your giant medical bill. You get a coupon. Exactly. You would get a coupon. (Time 0:51:41)
  • Talkie Is A Forecasting Test Built From The Past
    • David Duvenaud built Talkie to test whether a model restricted to pre-1930 knowledge can earn a long forecasting track record.
    • The goal is to ask past-bounded models about headlines, wars, and other events, then compare predictions with what actually happened decades later. Transcript: Kevin Roose David Duveneau, welcome to Hard Fork. Thank you very much, Kevin. So this project, Taki, is fascinating. It is a vintage LLM. Explain why you and Nick and Alec made this thing. So this all started a year ago. David Duvenaud Was me and Nick were interested in forecasting? Like specifically, can we teach machines how to forecast like five or 10 years ahead of time? Like what is the big picture going to be? Just because we have our own sort of like pet ideas about what the future is going to be. We don’t think people should take our word for it. And we also don’t think that people should trust machine forecasts unless they have a track record going back like decades and decades. So the idea here is that if we could build a model who really only knew about the world up to a certain date, we can ask it to forecast like five or 10 years ahead of time, like ask it, what’s The New York Times headline going to be five years from now? Or is there going to be another great war or something? And we can iterate and see like, what kinds of things are predictable? What does it take? Like, how far out can things be foreseen? And then hopefully, eventually, we’ll have machines that have like a hundred year track record of forecasting. And then we can ask them, you know, in 2026, what do you think is going to happen like, you know, two or four, eight years from now? (Time 0:57:45)
  • Historical Training Data Leaks Through Messy Metadata
    • Talkie’s post-1930 knowledge mostly comes from dirty archival metadata rather than secret internet access.
    • David Duvenaud says later prefaces, historian notes, updated editions, and wrong dates can leak facts like Hitler or FDR into supposedly older corpora. Transcript: Kevin Roose Is that proof that there’s been some kind of contamination of the training data with more recent data? David Duvenaud Oh, there’s definitely contamination. And this is sort of like one of the ongoing, like, things that we’re going to have to just keep revisiting again and again and refining. So we have a classifier that tries to look for things that are anachronistic. And especially if you want to use this for forecasting or to evaluate forecasting, it’s really important that we really nail this issue. So we have all sorts of ideas for like canaries and things that we think the model should just never assign any likelihood to. Like think of, I don’t know, Nagasaki and Hiroshima. Like before World War II, like those two towns would just never show up in the same sentence ever almost, except for like some weird coincidence. So you can just tell whether there’s been leakage about important events if the model just thinks that there’s any chance that you’ll see those particular names together. So this, anyway, like we’ve done, we’ve made a bunch of efforts to avoid leakage. We know there’s leakage right now, so you shouldn’t use it to evaluate your forecasting scaffold yet. Kevin Roose But how is it getting that data if it’s only being fed, scanned, OCRed books from archival sources? (Time 1:02:15)
  • Historic AI Needs Warnings Not Sanitized History
    • Preserving a period model’s realism means leaving its biases intact, then warning users rather than laundering history.
    • David Duvenaud says Talkie filters nothing from training data, but a modern model flags potentially upsetting racist outputs in the public demo. Transcript: Casey Newton Speaking of problematic content, some people found that Taki gives racist responses to questions that are basically like, you know, would you let a Black professor teach your child? I can see how that might be historically but I’m curious if you anticipated it and how you feel about it. Yeah. David Duvenaud So, you know, it was also very clear to us that it had these kind of responses. I mean, I guess I’m a professor myself and my sort of first instinct is like, let’s let people see this if they want to and just don’t surprise anyone and don’t be flippant about it because, You know, it really can be upsetting to some people. And especially if we just like treat it sort of insusantly. So the way we threaded the needle was we did zero like filtering of the data set for like problematic content to remember, right? We want to just like show what the actual sort of state of knowledge or state of thought was in the past. It would defeat the purpose of the project if we put our thumb on the scale. But for the public demo where you can talk to Taki, we just had a modern model with modern sensibilities. Just read every response, and at the end, once it’s generated, if it is deemed problematic, just slap a warning and say, oh, this might have something upsetting, like click if you want To see it. Casey Newton Right. Kevin Roose The description I loved, this came from Gavin Leach today, was that Taki is creating beautiful prose by a terrible person, which is consistent with some of my tests, which is like, this Thing actually does write quite well and actually to my ear, like much more literary than some of the more recent models trained on more recent data. But yeah, it is clearly the product of its time, or at least the time of its data. David Duvenaud Yeah, and the prose is really cool, because it’s very refreshing style. And actually, if you feed it to one of the AI detectors, it usually says like 100% human, which is kind of funny. But then I guess, as you mentioned, like a terrible person. I mean, right now it’s kind of ends up being this sort of like average person. And depending on like, it’ll just randomly answer all sorts of different voices. But that’s one of the next things we’re planning to work on is helping you talk to more specific people or in specific sort of states of knowledge or times and places, because that, I think, Allows you to answer more coherent questions than just talk to, like, the hive mind of 1930 or whatever. (Time 1:05:17)