Skip to content

Podcast

A.I. Goes to War + Is ‘A.I. Brain Fry’ Real? + How Grammarly Stole Casey’s Identity

Hard Fork

Source ↗ ← All highlights
  • AI Shrinks Intelligence Haystacks
    • Militaries are using AI to process massive sensor and communications data and turn weeks-long planning into near real-time operations.
    • Kevin Roose and Casey Newton describe dashboards that narrow huge haystacks of drone, camera, and intercept data into actionable leads. Transcript: Casey Newton All right, Kevin, let’s get into the biggest news of the week, which is the war in Iran. Specifically, we want to talk about what we know about how AI is being used in this fight. Kevin Roose Yeah, and I think the reason to talk about this is not just because it’s happening, it’s the biggest story in the world, but also because I think this is really a turning point in the use Of AI in the military. We’ve been hearing for years and reading science fiction books and listening to people talk about the use of AI in military applications. But now I think we are starting to see exactly how these tools are being used on the battlefield and what kind of effects they might be having. We are. Casey Newton And I’ll say up top that anytime you’re talking about the use of technology war, there is always the risk that you are just passing along propaganda, right? Because both the military and the contractors have a vested interest in telling you, hey, we have some real gee whiz new stuff, and it’s totally changing the game, right? Everybody has an incentive to tell you that. And yet, as you and I have dug into it, we do believe that there are some notable ways that AI are being used, and I think it is worth mentioning them. If for no other reason than I think it’s been the experience in the United States over the past couple of decades that tools that are deployed abroad during times of war sometimes come Back home after the war and wind up being used against American citizens. (Time 0:03:08)
  • Human In The Loop Can Become Rubber Stamp
    • Military statements claim humans remain in the loop, but analysts warn AI that selects and times targets can effectively push humans to just press the fire button.
    • Casey and Kevin warn of pressure to defer decision-making as models improve. Transcript: Casey Newton Now, one question that is coming up a lot is, to what extent, if any, is the military starting to offload decisions to AI, right? Is it the case that there is some military commander that is typing into a chatbot, hey, should I send the missile here or there? And the military’s public statements are that they are not doing this, right? They are sort of taking care to say, no, like humans are in the loop here. We are relying on human judgment. But there are other experts that are saying, at some point, if you’re going to be consulting with a chatbot and the chatbot is getting smarter and smarter, before too long, it’s probably Not going to feel very different from the AI actually just making the decision for where to shoot a missile. Kevin Roose Yeah, I think that’s a really good point. I think there is a difference between a fully autonomous weapon that can sort of do everything from selecting the target to like firing the weapon all on its own with no humans in the loop. But I think what you’re talking about is sort of a system that can do everything except fire the weapon. It can sort of select the target. It can tell you the right timing. It can like identify all the objects in the surveillance footage. And it can kind of give the military officials the confidence they need to go ahead and push the button. And there’s some worry that (Time 0:09:22)
  • Claude Embedded In Military Targeting
    • Anthropic’s Claude is reported to be integrated into Palantir’s Maven Smart System and has suggested hundreds of targets with coordinates and priorities.
    • Reporting says Maven plus Claude turned weeks-long battle planning into real-time operations used for target selection. Transcript: Kevin Roose Yeah, and I think Claude and Anthropic have come up a lot in recent weeks for obvious reasons. They had this big fight with the Pentagon. But it’s also the case that right now in this war in Iran, Claude is the only AI model that has actually been deployed inside classified military systems. Casey Newton So to the extent that AI is having an effect in Iran, it is probably Claude. Yes. And the Washington Post had a story about AI and the war in which they said that Claude was so essential to operations that if for some reason Anthropik said, hey, we want you to stop using Claude, the military would push back and say, we’re actually going to force you to continue to use this product. So just again, the continued strangeness of the situation, the Pentagon has now formally declared Claude and Anthropic to be a supply chain risk. This week, Anthropic sued over that. Yeah. Kevin Roose And there’s also been a lot of reporting coming out over the past week or two about the actual ways that Claude is being used and deployed in the military. There’s been some reporting on this system built by Palantir called Maven Smart System, which from what I can tell is kind of a real-time dashboard for intelligence that basically Allows you to pull in a bunch of drone footage and sensor data and track a bunch of supplies and troop movements and things like that. And by the way, this is the system that caused a huge controversy at Google in the late 2010s. Casey Newton And, you know, Google’s like quit over this. They did not want the company involved with Project Maven. And eventually Google dropped the contract. When they did, Palantir stepped in and eventually brought on Claude. Right. Kevin Roose And so Claude has been integrated into Maven Smart Systems since 2024. And the reporting that I’ve seen over the past week, including in this article in The Washington Post, said that this combination of the Maven Smart System built by Palantir and Claude Has already suggested hundreds of targets, issued precise location coordinates, and prioritized those targets according to importance. And according to this same article, it says that the use of Maven and Claude has turned weeks-long battle planning into real-time operations. (Time 0:11:55)
  • Data Centers Become Frontline Targets
    • Iran targeted cloud data centers in the UAE and Bahrain after U.S. strikes, disrupting banking and local services hosted on AWS.
    • Casey Newton highlights data centers as low-defense asymmetrical targets that can create broad civilian impact. Transcript: Casey Newton Well, so as you know, there’s been this huge buildout of AI infrastructure throughout the Middle East over the past several years. We’ve seen these multibillion-dollar projects being signed and built in Saudi Arabia and United Arab Emirates and Qatar. And these deals involve basically all of the big American tech giants, Amazon, Microsoft, and Google. And I would say there are sort of like two major pieces of infrastructure that are relevant here. One is data centers, right, which are being used to run AI systems and also just provide basic cloud hosting and storage services to all sorts of companies. And then you have fiber optic cables, which connect those data centers to the rest of the world. So let’s maybe talk about the data centers first. So the Guardian reported that on the morning of March 1st, which was the day after the initial U.S. Attacks in Iran, Iran responded by striking a couple of Amazon data centers in the UAE, and they also damaged a third one in Bahrain. And in the immediate aftermath of that, people in those countries were opening up their phones, and they couldn’t check their bank balances. They couldn’t order a taxi. It seems like a lot of services in those countries were being hosted on AWS, and they just didn’t have access to those services anymore. Afterwards, Iran put out a statement that said that they had gone after the data centers to identify the role that they played in supporting the enemy’s military and intelligence activities. Kevin Roose That’s so interesting. So they were basically targeting data centers rather than, say, troops, because they thought it could actually be more disruptive if it turned out that the U.S. Or Israel or any of the other allied nations were running their services on data centers located in the Middle East. Casey Newton Yeah, well, I mean, and also, like, data centers are a great target. Like, they’re just sitting there. They don’t have any defenses, right? So you can just send a few missiles over there and do an asymmetric amount of damage. And so now, Kevin, people are starting to question the logic of doing all these multi-billion dollar deals in the Middle East. They’re saying, hey, should this really be a linchpin of global AI infrastructure if it’s just kind of a rough neighborhood and all of the investments that you’re going to build there Are just going to be kind of perpetually at risk. (Time 0:15:58)
  • Undersea Cables Are Tactical Chokepoints
    • Fiber optic lines through the Strait of Hormuz are critical chokepoints; disruption would be hard to fix mid-conflict and could sever regional connectivity.
    • Kevin Roose flags undersea cables as high-risk infrastructure in wartime. Transcript: Casey Newton As of press time, as we record this, these lines have not been attacked or disrupted, but everyone is keeping a really close eye on it because were they to be disrupted, there is just simply No obvious way to fix them in the middle of a live war. Kevin Roose Casey, how does this all make you feel that AI is playing such an important and central role in an ongoing war in Iran? I mean, this to me just feels like the frog is being boiled, right? Casey Newton Like, when I think of all of the potential violent uses of AI, data analysis is not among those that gets me most nervous. Although, of course, I do have concerns about, you know, domestic surveillance. But I also know how rapidly these systems are advancing. I know the pressures that are quite apparent in our military to use AI for ever more things. I worry that there aren’t going to be appropriate safeguards on those things. (Time 0:19:22)
  • AI Brain Fry Is Distinct Cognitive Strain
    • BCG survey found 14% of AI users reported ‘AI brain fry’—a cognitive strain from excessive oversight or interaction with AI beyond capacity.
    • Respondents described feeling like they had “12 browser tabs open in my head” and being drained by tool management. Transcript: Casey Newton Study. You surveyed 1,488 workers in January of this year from all different disciplines, lots of different companies. What kind of questions did you ask these workers? Julie Bedard Yeah, we asked them all kinds of questions around how they use AI, how they feel at work, you know, traditional burnout metrics. We asked some, you know, sort of proxies for cognitive ability. And we did throw in a question about AI brain fry. We said specifically, like, what do you think about this thing that could be AI brain fry? Like, are you feeling that? Casey Newton And tell us how you define AI brain fry and what the survey results told you about it. Julie Bedard I mean, we defined it as really like a type of cognitive strain. So we said it was mental fatigue. It was related to excessive use of interaction with or oversight of AI. And it was about being beyond one’s cognitive ability. So it’s sort of like I’m using the tool, but it feels beyond my ability to process it. So 14% of people who use AI said that they felt this. And I was especially surprised by the extent to which they told us about it. We asked, you know, free-ended, like, just tell us, what is this thing? What does it show up? How does it feel to you? And people wrote a lot, right? (Time 0:29:00)
  • Marketing Faces The Most Brain Fry
    • Marketing roles report the highest AI brain fry because their workflows are heavily disrupted and involve many iterative, synthetic-content tasks.
    • Julie notes marketing tasks like image creation and synthetic panels drive unclear done criteria and heavy oversight. Transcript: Kevin Roose Your study, you found that people in certain industries tended to experience AI brain fry more frequently. I was struck by marketing seems to be the place where people are feeling it the most. And people in areas like management and law and compliance reported significantly less brain fry. Do you have a theory on why that is? Yeah. Julie Bedard So the short answer is, unfortunately, our survey, at least scientifically, was not designed to answer that question. But I have my theories based on other work that I’ve done. And, you know, three years ago, I worked with some of the models to try to predict skill disruption. I was trying to figure out like which jobs will change the most. And one of the jobs that changed the most from a skill perspective was marketing manager. A marketing manager was 90% disrupted from a skill perspective. So that’s sort of the first fundamental piece about marketing is like they’ve tended to adopt and it’s a really different way of working because of the power of the tools. The next thing, if I really just think about like what is Brain Fry, like it’s about the iteration, it’s about the oversight. A lot of marketing lends itself to that. Like in the field, we see stories of folks who are doing image creation. They’re doing synthetic consumer panels, right? They’re spinning up a bunch of campaigns at the same time. And it really lends itself to that definition of like, when do they know they’re done? When do they know the image is ready? Like, have they defined those success thresholds for themselves? I’m guessing they haven’t yet, right? Like, they haven’t figured out how do you do all the things to the right level of quality based on the outcome that you’re trying to drive for. Casey Newton It makes sense to me that, like, the more your job is changing, the more kind of vertigo you’re going to be experiencing as these new tools are introduced into your workplace. (Time 0:33:30)
  • Talk To Your Manager About AI Workflows
    • Engage managers and teams to reduce AI brain fry by clarifying when and how to use tools and defining outcomes over output.
    • Julie Bedard’s data shows manager engagement and team integration lowered reported brain fry. Transcript: Kevin Roose Yeah. Julie Bedard So if you’re an individual worker, I think first just acknowledging that this is a risk is the first thing. The second thing is really focusing on what you’re trying to achieve. It’s like back to that outcome piece. I mean, I know this is really basic, but if we were very clear about we’re measuring outcomes, not output, and we’re trying to get to the right answer, and what are those steps to help me Get there? And so, you know, from our data, we would say the things you could do is one, engage your manager. So managers who engaged in questions, we saw brain fry go down. And I think it’s about creating that sort of open dialogue about how should I use AI? When is it valuable? The other thing is to engage your team on this. So interestingly, when teams were using AI together, and they had better integrated it into their workflow, so like how I hand off work to Kevin, and Kevin does to Casey, we also saw brain Fry go down. And, you know, I don’t have the data to say exactly why, but my hypothesis would be is we’re not bottlenecking work in one person. (Time 0:40:35)
  • Define Success Thresholds For AI Tasks
    • Focus on outcomes not output when introducing AI and explicitly define success thresholds for AI-assisted tasks.
    • Bedard recommends teams decide what ‘done’ looks like to avoid endless iterations and overload. Transcript: Julie Bedard So if you’re an individual worker, I think first just acknowledging that this is a risk is the first thing. The second thing is really focusing on what you’re trying to achieve. It’s like back to that outcome piece. I mean, I know this is really basic, but if we were very clear about we’re measuring outcomes, not output, and we’re trying to get to the right answer, and what are those steps to help me Get there? And so, you know, from our data, we would say the things you could do is one, engage your manager. So managers who engaged in questions, we saw brain fry go down. (Time 0:40:36)
  • Casey Unwittingly Became A Grammarly ‘Expert’
    • Casey Newton discovered Grammarly’s Expert Review used famous writers’ names without their consent and produced generic, low-quality advice.
    • He tested the feature with Platformer pieces and saw suggestions attributed to Timnit Gebru, John Carreyrou, and Kara Swisher. Transcript: Kevin Roose Crazy experience by being selected against your will and without your permission as one of Grammarly, the AI kind of writing assistant. They have an expert network of people whose voices they have borrowed for the purposes of, I guess, making people’s writing better. So A, congratulations. Thank you. I assume the royalty checks are just overflowing your mailbox. But what actually happened here? You had a fascinating newsletter about this this week. Casey Newton Well, thank you. So this story I first learned about from The Verge. Their reporter, Stevie Bonifield, wrote about this, and it turned out that last summer, Grammarly had added this feature called Expert Review. I had not actually used Grammarly until this. Have you ever used it? No. So I decided, you know what, why don’t I sign up for the free trial and see what Grammarly can do for me? And if you go to the support page for this feature, it says that Expert Review, quote, is designed to take your writing to the next level with insights from leading professionals, authors, And subject matter experts. That sounds pretty cool, right? Well, scroll a little further down, Kevin, and you see the following disclaimer. References to experts in Expert Review are for informational purposes only and do not indicate any affiliation with Grammarly or endorsement by those individuals or entities. And so I read that and I thought, when you say that these insights come from leading professionals, what does the word from mean to you? Because it sounds like what you’re telling me is they don’t come from those experts at all. Kevin Roose Yeah. It’s like when you see like a tub of margarine and it’s like, you know, it’s like butter style product in very small type. Yeah. They have sort of an expert network with an asterisk. None of the experts were actually consulted and we didn’t actually hear from them in any way. Casey Newton Absolutely. So Stevie over at The Verge put a bunch of writing through expert review to see what sort of expert names would pop up. I was one of them. Congratulations. Thank you. You know, as you might imagine, Grammarly also picked a bunch of like actual famous people. So Stephen King, Neil deGrasse Tyson, Carl Sagan. And I decided to put this thing through my own paces and loaded up some recent columns that we published in Platformer and pasted them in to see what sort of experts it would suggest. And while I was never able to get my own name, Kevin, I did see a succession of people that sort of felt like if you made a list of people who would hate this idea the most, that is who Grammarly Had picked. So Timnit Gebru, a very vocal critic of AI systems, the way they are built and deployed, she showed up as a quote unquote expert. So did Julia Angwin, who is an investigative reporter. She writes for New York Times Opinion, and it used her writing, even though she has written a lot about how tech systems are used for privacy and surveillance in ways that are contrary To how we probably want them to be used. Julia, by the way, filed a class action complaint against Grammarly’s parent company on Wednesday, seeking to stop them from, quote, trading on her name and those of hundreds of other Journalists, authors, and editors, and to stop them from, quote, attributing words to them that they never uttered and advice that they never gave. Kevin Roose Wait, can I ask a question about the mechanics of this? Okay, so you’re writing in Grammarly, which I gather is sort of like a bolt-on to like a word processor. Yes. And it sort of detects the topic you’re writing about and then pops up a little like clippy thing that’s like, would you like Julia Angwin to edit this for you? Would you like Casey Newton to give this one a pass? Casey Newton Exactly. I’ll actually show you an example here. If you want to look at my laptop, you can see that here is the text that I wrote. And then in this little left-hand column, in this case, it just says Kara Swisher. Kara Swisher, my good friend, past hard forecast, legendary Silicon Valley journalist and podcaster, and someone who has absolutely no involvement with Grammarly. But her name just sort of pops up there with no disclaimer at all, right? And then when you sort of click in, it will offer this sort of Kara-inspired advice. And this is the point, Kevin, where I would like to talk about the kind of advice that this thing actually gives. Please. So you might expect, given that they were, you know, allegedly trying to borrow the expertise of real humans, that that expertise would seem like incredibly specific to that person, Right? Instead, what you’re getting is just a bunch of very generic advice about something that you might do. So I noted, for example, that, you know, we public, my colleague Ella Marchionos wrote a story in Platformer last week where she went to a protest at OpenAI, and there was a suggestion That Grammarly had said was inspired by John Carreyrou, the legendary investigative journalist who brought down Theranos. And the advice basically boiled down to try opening with a colorful scene and use a lot of rich details and characters, right? Like sort of the most absolute generic advice that you would ever imagine getting. And nothing like I would imagine the actual experience of sitting down with John Kerry Rue and saying, like, hey, how did you write Bad Blood? Kevin Roose Yeah. How did it say that Kara Swisher would edit a story? Casey Newton So I will just read you the piece of advice that it gave me. This was also a piece of advice about this protest story. The fake AI Kara said, could you briefly compare how daily AI users versus AI skeptics articulate risk, creating a through line readers can follow? A synthesizing sentence here may tighten the narrative arc. Kevin Roose I’m laughing because that is the exact opposite of how I imagined Kara Swisher would edit someone. Yeah. Casey Newton It would just be like a string of like four letter words and like, you know, this sucks. Do it over again. Yeah. It would say, stop wasting my time. You know, like that, that would be the advice that I, the thing that I just read, I just want to acknowledge, like it is word salad. Do you know what I mean? Totally. Like you can tell what, I don’t know what underlying model they’re using here. I’m guessing it is not a frontier one, right? It’s reading very like GPT2 to me, you know? So this advice is so bad, but let’s bring this into what I actually find upsetting about this, Kevin. Yeah, let’s make this about you. No, well, here’s the thing. I’m actually not going to make it about me because I have sort of just long since accepted that all of these companies are, have stolen all my intellectual property and are having their Way with it. Where I really feel bad is for the subscribers to Grammarly. These people are paying $144 a year to be able to use this glorified spell checker, okay? And they load this thing up, and then Grammarly them this service. And so if you are (Time 0:51:40)
  • Require Opt-Out And Compensation For Copied Identities
    • If companies misuse creators’ names, demand opt-out controls and transparency; creators should be paid or given control for any repurposed identity.
    • Grammarly later allowed unconsulted experts to opt out after public exposure. Transcript: Casey Newton We’ve thought about it. And if you’re one of our experts who we didn’t consult and we’re not paying, you can now opt out of this feature. How nice of them. So you can now send an email and say, I don’t want to be a part of this system anymore. And so, you know, I wrote the story and got a lot of comments on social media, like, you know, geez, that really seems like the least they can do. But Kevin, as we record this, I actually have some breaking news. What’s that? So I got an email from the spokeswoman over at Superhuman today. Superhuman is what Grammarly now calls itself. They did a rebrand last year, and they’re now sort of a bundle of mediocre products. Review as we reimagine the feature to make it more useful for users while giving experts real control over how they want to be represented or not represented at all. Dot, dot, dot, dot, dot, dot. Thanks for holding us accountable. We’re committed to getting it right next time and we’ll be transparent about how we improve. Wow. Results. Newton gets results. Newton getting some results. (Time 1:00:06)
  • SaaS Writing Tools Face Pressure From Frontier Models
    • Many niche SaaS tools risk obsolescence because free frontier models (Claude, Gemini, ChatGPT) can replicate basic features cheaply.
    • Casey and Kevin argue Grammarly-style prosumer products face an ‘asspocalypse’ as better free models eat subscriptions. Transcript: Casey Newton You had like whatever spell checker was in Google Docs and like that was, you know, probably gonna be the best tool available. Fast forward to today though, you got ChatGPT, you got Gemini, you got Claude. There are free versions of these services. If you want a quick grammar check, you can get it. My guess is that’s the experience that you just had. Kevin Roose Yeah, if I want a grammar check, I’m just copying and pasting into one of the AI models. I’m not using like a purpose-built thing for that or it’s now built into, you know, Google Docs. Yeah. Casey Newton And to, you know, emphasize a point, when you’re using Claude, as you did in your book, you’re using the latest and greatest version of Claude. If you are using some sort of startup that is like using the API of Anthropic, they’re not actually incentivized to give you the frontier model most of the time, right? Because that’s going to be very expensive. So they’re going to give you a model that’s a couple generations old because they can get a lower price and their margin is going to be better on it. (Time 1:02:14)