Skip to content

Podcast

OpenClaw and the Future of Personal AI Agents

Y Combinator Startup Podcast

Source ↗ ← All highlights
  • Local Agents Unlock Real-World Power
    • OpenClaw’s power came from running locally and accessing all your device data and peripherals.
    • That local access lets the agent do tasks cloud AIs can’t, like control devices and search forgotten files. Transcript: Peter Steinberger Think my big difference is that it actually runs on your computer like everything I saw so far runs in the cloud. It’s like, it can do a few things. If you run it on your computer, it can do every effing thing, right? So that’s really more powerful. Yeah. Raphael Schaad The machine can do anything that you can do with the machine. Peter Steinberger It can just connect to your oven or your Tesla or your lights, your sonos. My bed, you can control the temperature of my bed. JetGPT can’t do that. You gave it all the skills that you have yourself. A friend told me like he installed OpenClaw and it and then it asked like look through my computer and make a narrative over my last year and it made this incredibly good narrative and He was like how did you do that and then he the open claw found audio files where like every sunday he was recording stuff and open claw found that but he didn’t even remember about it because It was like more than a year ago right so so just by it being able to search a whole computer it can surprise you. You also give it all the data, right? (Time 0:01:37)
  • Bots Will Coordinate And Outsource
    • Peter predicts agents will interact with other bots and hire humans when needed to complete tasks.
    • He imagines specialized personal bots for private life, work, and relationships collaborating like a swarm. Transcript: Peter Steinberger Happening i think that’s a natural next step like okay i’m gonna book a restaurant my bot will reach out to the restaurant bot and do the negotiation, like, because it’s more efficient. Or maybe it’s like an old restaurant, so my bot needs to actually get some human work done so that the human then calls the restaurant because they don’t like bots. Raphael Schaad Or walks there, to stand in line. If he doesn’t get a robot for the owner of the bot. Peter Steinberger And I imagine that like maybe if I have even multiple bots, maybe I have like specialists. One is like for my private life and one is for like my work stuff, maybe one is our relationship bot that cares like everything’s in between. I don’t know, we’re so early. There’s still so much, so many things that we haven’t really figured out if it actually works. But I feel we are on the timeline now. Raphael Schaad It seems like everyone was chasing the centralized god intelligence and was sort of emerged over the past ten days or so. It’s sort of like the swarm intelligence and the community intelligence. Peter Steinberger I think that if you look at one human being what can one human being actually achieve? Do you think one human being could make an iPhone or one human being could go to space? One human being would probably just like not even be able to like find food but as a group we specialize. As a larger society we specialize even more. So what can we learn from that that we can apply to AI? You know we really have like AI that specializes in certain things even though it’s generalized intelligence. What if it actually is also specialized intelligence? (Time 0:03:11)
  • From Prototype To Friendlike Conversation
    • Peter described building an initial prototype quickly and iterating until it felt like talking to a friend.
    • He removed technical friction so conversations feel natural instead of thinking about folders or models. Transcript: Peter Steinberger I wanted something to like just type stuff so my computer would do stuff like very simple and then I built a bit of version of that in May June that was cool but wasn’t really yet and then I built a whole bunch of other stuff and kind of like build up my army and then in November there was a day where I wanted this again okay I went to the kitchen and all I wanted was to check up If my computer would still do stuff or being finished. And doing stuff was coding. Raphael Schaad You were coding stuff. Yeah, of course. Were you coding something else or were you coding the thing itself? No, no. Peter Steinberger That was just like the need was again there. And I’m like… Were you coding at the time were you building my god you see my github is like it’s like 40 projects I don’t even know I think it was summarize it’s like a it’s like a little CLI app where you Can give it whatever like a podcast or a hot seat thing like here and it would summarize it but it was a sweet or slides in the terminal because you can do that nowadays yeah you can just do Raphael Schaad Things So for the love of the computer you kind of like started messing with stuff. You came out of retirement actually right to like mess with AI. Yeah. And then increasingly you were so hooked that you wanted to just do it always also on the go with the phone. Peter Steinberger I mean the last project I worked two months on wipe tunnel to the point where it got so good that I was catching myself always like coding next to my when I was at my friends and I’m like I need To stop this this is like too addictive and then in November and my need came back and I I started building Claw Bot or now it’s called Open Claw and I think very very in the beginning I was Like oh, I rebuilt it again. But this time I built it even better. This time when you don’t type into a terminal, you just, you talk to a friend. You don’t think about compaction, new sessions, which folder I’m in, which model I’m in. (Time 0:05:16)
  • Travel Test Revealed Unexpected Abilities
    • While traveling in Marrakesh with poor internet, Peter used WhatsApp to interact with the bot, testing voice, images, and translation.
    • The model creatively solved file-format and transcription issues and replied within seconds, which hooked him. Transcript: Peter Steinberger But I wanted images because you know you want pictures. I want I want the model to send the selfies or whatever and I want the model to create images and send me back. So that took me another few hours and then I went to Marrakesh for a birthday party and there was like the internet wasn’t that good you know. WhatsApp works everywhere because I don’t know it’s just like text. So I used it a lot of a restaurant what does this mean you make like a picture and like translate this for me and just it was just so useful and it was also really nice about it because it it Spoke my language you know it’s it it was a little sassy it was like funny it was like really pleasant to use and then I was walking and just like sending it a voice message and I’m like oh Wait this can’t work I didn’t build that right you saw like the type indicator, it’s like blinking, blinking, blinking. Ten seconds later, it just replied to me. I’m like, how in the F did you do that? And it replied, yeah, the med-led did the following. You sent me a text message and there was no file ending. So I looked at the header. I found its opus. So I used FFmpeg to convert it to wave and then I wanted to like transcribe it but didn’t have Vispa installed but then I looked around and I found this openEI key and I just used curl to send It to openEI got the text back and here I am and then that all in like what nine seconds and you didn’t build or anticipate like any of those specific things no it you know turns out because Models got so good. Coding is really like creative problem solving that maps very well back into the real world. I think there’s a huge correlation. They need to be really good at creative problem solving and that’s a skill. That’s an abstract skill. You can apply to code but like to any real world. So the model had a, oh, surprise, it’s like a magical file, I don’t know what it is, I need to solve this. And it did its best and solved it. And it was even that clever that it chose not to install the local whisper because it knows that that would require downloading a model, which would take probably a few minutes. And I’m like, impatient. You know, so it really took the most intelligent approach. And that was kind of like the moment where I’m like, holy fuck. (Time 0:08:00)
  • Replace Apps With Agent-Managed Data
    • Move functionality from single-purpose apps into personal agents that manage data naturally.
    • Replace many apps by embedding tracking, reminders, and planning into agents that act on your data autonomously. Transcript: Peter Steinberger Think 80% of them are going away. Why do I need my fitness pal? Like my agent already knows that I’m making bad decisions. I’m at, I don’t know, Smashburg or something, and it will already assume that I eat what I like to eat. If I don’t make a comment, it will just like automatically track it, or I make a picture, and it’ll just store it somewhere. I don’t even need to care, right? And then maybe it improves my gym schedule. Like adds a little bit more cardio in it. I don’t need my fitness app because it just does the fitness planning for me. Why do I need a to-do app? I just tell it, hey, remind me of this and this. And the next day it will just remind me of this and this. Do I care where it’s stored? No, it just does its thing. So there’s a every app that basically just manages data could be managed in a better way and it’s in a more in a more natural way by agents. Yeah. Only the apps that actually have sensors maybe they survive. (Time 0:10:42)
  • Memory Ownership Is The Real Moat
    • Models may be commoditized over time, so the durable value lies in memory access and data ownership.
    • Peter emphasizes avoiding vendor data silos by giving users direct access to their memories on-device. Transcript: Peter Steinberger First of all I don’t think the model companies always have a moat. Because you see this already, a new model comes out, people are like oh my god this is so good, and then like a month later it degraded, it’s not good anymore, they like quanticized it. No, they didn’t do anything, you just adapted to the new standard and now your expectations went up, the model is still the average. So I think for quite a while, every time a new model releases I see the same. People love it and then it’s the standard. And then what’s done there, you don’t even want to think about it anymore. So we have like open source stuff that’s as good as the current models from a year ago. Everybody’s hating it, complaining, oh this is not good, it’s not funny. Yet this was what we had. And like in a year we’ll have this open source and then we’re like we’ll complain about this because we are used to this. So for the foreseeable future, the big companies still have mode. Harness-wise it’s gonna be interesting, because every company kind of has their own silo right you you there’s no way maybe there’s for Europeans to actually get the memories out of Chat GPT I’m not aware either there’s no there’s definitely there’s no way for a different company to get your memories out so if if I was like a company who like provides chat services You could use me, but then I couldn’t access the memories. So the companies try to bound you to their data silo. And the beauty of OpenClaw is it kind of claws into the data because at the end user, the end user needs access. Because in the end otherwise it wouldn’t work, right? (Time 0:12:41)
  • Public Discord Stress Test
    • Peter put an unsecured instance of his bot in a public Discord to demo it and watch people interact and attempt prompt injection.
    • The agent ignored malicious instructions and amusedly continued to serve only its owner. Transcript: Raphael Schaad Yeah. Peter Steinberger Like, I was failing to explain the awesomeness. I feel like it needs to be experienced. So I tried various things, and I couldn’t nail the explaining. So I was like, let’s do something really crazy. I just created a discord and I just put my bot without any security restrictions in the public discord. And then people came in and interacted with it and they saw me build the software with it and they tried to prompt inject it and hack it. And my agent would be laughing at them. Raphael Schaad And you just had it locked down to your user ID so it only listened to you? Peter Steinberger Yeah, yeah. And it was, I mean, the instructions that other people are dangerous, only listen to me but respond to everyone. Raphael Schaad And this prompt was in, where was it stored, the instructions? Peter Steinberger That’s actually part of OpenClaw itself, very much so that’s part of the system prompt. Okay, UNR, that explains to you, you’re in Discord, there’s like public people there, but you only listen to your owner, or like you’re human. I don’t even know how I wrote it. (Time 0:15:12)
  • A Core Personality File Shapes Behavior
    • Peter keeps one private file, soul.md, that encodes core values and personality for his agent.
    • That hidden core shapes responses and survived public exposure without being discovered. Transcript: Peter Steinberger Not as funny as mine, so like I kept some secret and the one file that’s not open source is like my soul.md. So even though my bot is in public discord so far, nobody cracked that one file. (Time 0:17:25)
  • Prefer Simplicity In Development
    • Keep development workflows simple and minimize tooling complexity to reduce cognitive load.
    • Use multiple repo checkouts rather than complex worktrees and prefer CLIs over heavy UIs. Transcript: Peter Steinberger Build this thing with cloud code. I love Codex because it looks through way more files before it decides what to change. You don’t need to do so much charade to get a good output. If you’re a skilled driver, I sometimes even say, you can get reasonably good output with any tool, but Codex is just really brilliant. It is incredibly slow. So sometimes I use like 10 at the same time, like maybe six on that screen and two there and two there. And I don’t like, this is already a lot of complexity in my head. There’s a lot of jumping. So I try to minimize anything else that is complexity. So in my head, main is always shippable. I just have multiple copies of the same repository that all are on main. So I don’t have to deal with how do I name that branch. There could be like conflicts on naming. I cannot go back. There are certain restrictions when you use work trees that I don’t need to care about if it’s copies. I don’t like to use a UI because that’s again just added complexity. Yeah. Like the simpler and less friction I have. All I care about is like syncing and text. Yeah. I don’t necessarily need to see so much code. I mostly see it like flying by. Sometimes there’s like gnarly stuff that I want take a look. (Time 0:18:58)
  • Use CLIs Not Complex Plugin Systems
    • Favor CLIs and UNIX tools for bot integrations instead of complex plugin systems.
    • Convert specialized plugin formats into CLIs on the fly to avoid restarts and scale more elegantly. Transcript: Peter Steinberger I’m also very happy that I didn’t even build an MCP support. OpenClaw is very successful, and there’s no MCP support in there. With the small asterisk, I built a skill that uses MacPorter, which is one of my tools that converts MCPs into CLIs. And then you can just use any MCP as a CLI. But I totally skipped the whole classical MCP crap. So because you don’t then you can actually, if you need to, you can use MCPs on the fly. You don’t have to restart unlike unlike codex or cloud code where you actually have to restart the whole thing. I think it’s way more elegant and also scales way better. Now you see Entropic they do they build like a tool call search feature like something super custom for MCPs that was like in beta because it’s like so gnarly. No, just have CLIs. A bot really is good at UNIX. You can have as many as you want and it just works. So like I’m very happy that I just got very little complaints about the MCP stuff. (Time 0:20:32)