Skip to content

Podcast

The Agent Network — Dharmesh Shah

Latent Space: The AI Engineer Podcast

Source ↗ ← All highlights
  • Ingenisoft: Early Natural Language Interface
    • Dharmesh Shah’s initial AI concept, Ingenisoft, involved a natural language interface for business software, using email as the primary interface.
    • This was motivated by the desire to update software offline, particularly while traveling, before internet access on planes was common. Transcript: Dharmesh Shah How did you get Agent Religion? Let’s see. So I’ve been working, I’ll take like a half step back, a decade or so ago, even though actually more than that. So even before HubSpot, the company I was contemplating that I had a name for was called Ingenisoft. And the idea behind Ingenisoft was a natural language interface to business software. And I realize this is 20 years ago, so that was a hard thing to do. But the actual use case that I had in mind was, you know, we had data sitting in business systems like a CRM or something like that. And my kind of, what I thought clever at the time was, oh, what if we used email as the kind of interface to get to business software? And the motivation for using email is that it is automatically works when you’re offline. So imagine I can like I’m getting on a plane or I’m on a plane. There was no internet on planes back then. It’s like, oh, I’m going through business cards from an event I went to. I can just type things in into into an email just to have them all in the backlog. When it reconnects, it sends those emails to a processor that basically kind of parses effectively the commands and updates the software, sends you the file, whatever it is. And there was a handful of commands. I was a little bit ahead of the times in terms of what was actually possible. And I reattempted this natural language thing with a product called ChatSpot that I did back 20… Yeah, this is your first post-ChatTBT project. I saw it come out. Yeah. And so I’ve always been kind of fascinated by this natural language interface to software because, you know, as software developers, myself included, we’ve always said, oh, we build Intuitive, easy to use applications. And it’s not intuitive at all, right? Because what we’re doing is taking the mental model that’s in our head of what we’re trying to accomplish with said piece of software and translating that into a series of touches and Swipes and clicks and things like that. And there’s nothing natural or intuitive about it. And so natural language interfaces for the first time, you know, whatever the thought is you have in your head and expressed in whatever language that you normally use to talk to yourself In your head, you can just sort of emit that and have software do something. And I thought that was a kind of a breakthrough, which it has been and it’s gone. So that’s where I first started getting into the journey of AI side, because now it actually works, right? So once we got ChatGPT and you can take, even with a few-shot example, convert something into structured, even back in the ChatGP 3.5 days, it did a decent job in a few-shot example, convert Something to structured text. You knew what kinds of intents you were going to have. And so that happened. And that ultimately became a HubSpot project. But then agents intrigued me because I’m like, okay, well, that’s the next step here. So chat’s great, love chat UX. But if we want to do something even more meaningful, it felt like the next kind of advancement is not this kind of, I’m chatting with some software and kind of a synchronous back and forth Model is that software is going to do things for me in kind of multi-step way to try and accomplish some goals. (Time 0:02:30)
  • Broad Definition of AI Agents
    • Dharmesh defines an AI agent broadly as AI-powered software that achieves a goal, emphasizing its diverse implementations.
    • He anticipates future classifications of agents based on autonomy, workflow, interaction mode, and other factors. Transcript: Dharmesh Shah I’ve been obsessed ever since, by the way. So. Swyx Which goes back to your first experience with it, which is like you’re offline and you want to do a task. You don’t need to do it right now. You just want to give it up for somebody to do it for you. Yes. As you think about agents, like let’s start at the easy question, which is like, how do you define an agent? You mean the hardest question in the universe? Dharmesh Shah Is that what you mean? You said you have an irritating take. I do have an irritating take. I think, well, some number of people have been irritated, including within my own team. So I have a very broad definition for agents, which is it’s AI powered software that accomplishes a goal, period. That’s it. And what irritates people about it is like, well, that’s so broad as to be completely non-useful. And I understand that, understand the criticism. But in my mind, if you kind of fast forward months, I guess in AI years, the implementation of it, and we’re already starting to see this, and we’ll talk about this, different kinds of Agents, right? So I think in addition to having a usable definition, and I like yours, by the way, and we should talk more about that that you just came out with. The classification of agents actually is also useful, which is, is it autonomous or non-autonomous? Does it have a deterministic workflow? Does it have a non-deterministic workflow? Is it working synchronously? Is it working asynchronously? Then you have the different kind of interaction modes. Is it a chat agent, kind of like a customer support agent would be? You’re having this kind of back and forth. Is it a workflow agent that just does a discrete number of steps? So there’s all these different flavors of agents. So if I were to draw it in a Venn diagram, I would draw a big circle that says, this is agents. And then I have a bunch of circles, some overlapping because they’re not mutually exclusive. And so I think that’s what’s interesting. And we’re seeing development along a bunch of different paths, right? So if you look at the first implementation of like agent frameworks that you look at, I mean, baby AJI and auto GBT, I think it was an auto gen, that’s the Microsoft one. They were way ahead of their time, because they assumed this level of reasoning and execution and planning capability that just just did not exist, right? So it was an interesting experiment, thought experiment, which is what it was, Even the guy that, I’m an investor in Yohei’s fund that did maybe AGI. It wasn’t ready, but it was a sign of what was to come. And so the question then is, when is it ready? And so lots of people talk about the state of the art when it comes to agents. I’m a pragmatist, so I think of the state of the practical. It’s like, okay, well, what can I actually build that has commercial value or solves actually some discrete problem with some, you know, kind of some baseline of kind of repeatability Or, you know, verifiability. So, yeah. There (Time 0:05:11)
  • Atomic Agents: A New Perspective
    • Dharmesh proposes viewing even basic software functions as “atomic agents,” forming a network within larger agent systems.
    • This perspective simplifies the concept of tool use in AI, where agents delegate tasks within a network. Transcript: Dharmesh Shah A couple of things. Connecting the dots on the definition side of it. So we’ll get the irritation out of the way completely. I have one more, even more irritating leap on the agent definition thing. So here’s the way I think about it. By the way, the kind of word agent, I looked it up, like the English dictionary definition is when you have someone or something that does something on your behalf, like a travel agent Or a real estate agent acts on your behalf as a proxy, which is a nice kind of general definition. So the other direction I’m sort of headed, and it’s going to tie back to tool calling and MCP and things like that, is if you, and I’m not a biologist by any stretch of the imagination, but We have these single-celled organisms, right? Like the simplest possible form of what one would call life, but it’s still life. It just happens to be single-celled. And then you can combine cells and then cells become specialized over time. You have much more sophisticated organisms, you know, kind of further down the spectrum. In my mind, at the most fundamental level, you can almost think of having atomic agents. What is the simplest possible thing that’s an agent that can still be called an agent? What is the equivalent of a kind of single-celled organism? And the reason I think that’s useful is right now we’re headed down the road, which I think is very exciting around tool use, right? That says, okay, the LLMs now can be provided a set of tools that it calls to accomplish whatever it needs to accomplish in the kind of furtherance of whatever goal is trying to get done. And I’m not overly bothered by it, but if you think about it, if you just squint a little bit and say, well, what if everything was an agent? And what if tools were actually just atomic agents? Because then it’s turtles all the way down, right? Then it’s like, oh, well, all that’s really happening with tool use is that we have a network of agents that know about each other through something like an MMCP, and can kind of decompose A particular problem and say, oh, I’m going to delegate this to this set of agents. And why do we need to draw this distinction between tools, which are functions most of the time, and an actual agent? And so I’m going to write this irritating LinkedIn post proposing this. It’s like, okay, and not suggesting we should call even functions, you know, call them agents. But there is a certain amount of elegance that happens when you say, oh, we can just reduce it down to one primitive, which is an agent that you can combine in complicated ways to kind of Raise the level of abstraction and accomplish higher order goals. Anyway, I’d (Time 0:08:50)
  • Extending Consciousness into Machines
    • Swix observes that humans are extending their consciousness and functionality into machines, integrating AI into their sense of self.
    • He mentions using a Bee pendant, exploring the distribution of self beyond the physical body. Transcript: Alessio Fanelli Opening eyes already on that. Yeah. My quick philosophical engagement with you on this cell thing is I often think about the kind of the other spectrum, the other end of the cell spectrum. So single cell is life, multi-cell is life, and you come a bunch of cells together in a more complex organism, they become organs like an eye and a liver or whatever. And then obviously we consider ourselves one life form. There’s not like a lot of lives within me. I’m just one life. And now, obviously, I don’t think people don’t really like to anthropomorphize agents and AI, but we are extending our consciousness and our brain and our functionality out into machines. I just saw you wear a bee. Yeah. Which is… Limitless pendant in my pocket. I got one of these boys. Yeah, I’m testing it all out. You know, got to be early adopters. But like, we want to extend our personal memory into these things so that we can be good at the things that we’re good at. And, you know, machines are good at the and machines are there. So like my definition of life is kind of like going outside of my own body now. I don’t know if you’ve ever had like reflections on that, like how yourself is like actually being distributed outside of you. (Time 0:11:50)
  • Engineering Pragmatism
    • When engineering, if the “right way” has a marginally increased cost, choose it over shortcuts.
    • Under-engineer rather than over-engineer, as tech debt is manageable, and you may avoid unnecessary work. Transcript: Alessio Fanelli Think we want to move on to MCP, but I just want to like, just engineering advice. Obviously, you’ve had to do a lot of projects and run a lot of teams. Do you have a general rule for over-engineering or engineering ahead of time? You know, like, because people, we know premature engineering is the root of all evil. Yep. Dharmesh Shah But also sometimes you just have to. Yep. When do you do it? Yes, it’s a great question. This is a question as old as time almost, which is what’s the right and wrong levels of abstraction. That’s effectively what we’re answering when we’re trying to do engineering. I tend to be a pragmatist, right? So here’s the thing. Lots of times doing something the right way has only a marginal increased cost. In those cases, just do it the right way. And this is what makes a great engineer or a good engineer better than a not so great one. It It’s like, okay, all things being equal, if it’s going to take you roughly close to constant time anyway, might as well do it the right way. So do things well. Then the question is, okay, well, am I building a framework as a reusable library? To what degree? What am I anticipating in terms of what’s going to need to change in this thing? Along what dimension? And then I think like a business person in some ways, like what’s the return on calories, right? So, and you look at energy, the expected value of, it’s like, okay, here are the five possible things that could happen. Try to assign probabilities like, okay, well, if there’s a 50% chance that we’re going to go down this particular path that someday, like, or one of these five things is going to happen And it costs you 10% more to engineer for that. It’s basically, it’s something that yields kind of interest compounding value, as you get closer to the time of needing that versus having to take on debt, which is when you under engineer It, you’re taking on debt that you’d have to pay off when you do get to that eventuality where something happens. One thing as a pragmatist, so I would rather under engineer something than over engineer it, if I were going to err on the side of something. And here’s the reason is that when you under-engineer it, yes, you take on tech debt, but the interest rate is relatively known and payoff is very, very possible, right? (Time 0:20:02)
  • The Future of Junior Engineers
    • Junior engineers are not obsolete if they possess strong communication and problem-solving skills.
    • Learning coding teaches valuable systems thinking and abstraction, regardless of future syntax changes. Transcript: Alessio Fanelli Turn it to 11. You mentioned vibe coding, so I have to, this is a blog post I haven’t written, but I’m kind of exploring it. Is the junior engineer dead? I don’t think so. Dharmesh Shah I think what will happen is that the junior engineer will be able to, if all they’re bringing to the table is the fact that they are a junior engineer, then yes, they’re likely dead. But hopefully if they can communicate with carbon-based life forms, they can interact with product. If they’re willing to talk to customers, they can take their kind of basic understanding of engineering and how kind of software works. I think that has value. So I have a 14- old right now and he’s taking Python programming class. And some people ask me, it’s like, why is he learning coding? And my answer is, is because it’s not about the syntax. It’s not about the coding. What he’s learning is like the fundamental thing of like how things work. And there’s value in that. I think there’s going to be timeless value in systems thinking and abstractions and what that means and whether functions manifested as math, which he’s going to get exposed to regardless, Or there are some core primitives to the universe, I think that the more you understand them, those are what I would kind of think of as like really large dots in your life that will have A higher gravitational pull and value to them that you’ll then be able to. (Time 0:25:50)
  • AI Agents as Teammates
    • Dharmesh envisions AI agents evolving into teammates within hybrid teams, collaborating like human coworkers.
    • This necessitates professional networks and tools for managing and evaluating these digital team members. Transcript: Dharmesh Shah So I think we opened this with agent.a. Okay, so I’ll start with, here’s my kind of running thesis, is that as AI and agents evolve, which they’re doing very, very quickly, we’re going to look at them more and more. I don’t like to anthropomorphize, and we’ll talk about why this is not that, less as just like raw tools and more like teammates. They’ll still be software. They should self-disclose as being software. I’m totally cool with that. But I think what’s going to happen is that in the same way you might collaborate with a team member on Slack or Teams or whatever you use, you can imagine a series of agents that do specific Things just like a team member might do, that you can delegate things to, you can collaborate, you can say, can you take a look at this? Can you proofread that? Can you try this? You can, whatever it happens to be. So I think it is, I will go so far as to say it’s inevitable that we’re going to have hybrid teams someday. And what I mean by hybrid teams, so back in the day, hybrid teams were, oh, well, you have some full-time employees and some contractors. Then it was like hybrid teams are some people that are in the office and some that are remote. That’s the kind of form of hybrid. The next form of hybrid is like the carbon-based life forms and agents and AI and some form of software. So let’s say we temporarily stipulate that I’m right about that over some time horizon that eventually we’re going to have these kind of digitally hybrid teams. So if that’s true, then the question you sort of ask yourself is that then what needs to exist in order for us to get the full value of that new model? It’s like, okay, well, you sort of need to like, okay, well, how do I, if I’m building a digital team, like how do I just in the same way, if I’m interviewing for an engineer or a designer Or a PM, whatever, it’s like, well, that’s why we have professional networks, right? It’s like, oh, they have a presence on likely LinkedIn. I can go through that semi-structured, structured form, and I can see the experience or whatever, you know, self-disclosed. But okay, well, agents are going to need that someday. And so I’m like, okay, well, this seems like a thread that’s worth pulling on that said, okay, so agent.ai is out there for agents. It’s LinkedIn for agents. It’s a professional network for agents. And the more I pull on that thread, it’s like, okay, well, if that’s true, like what happens, right? It’s like, oh, well, they have a profile just like anyone else, just like a human would. It’s going to be a graph underneath, just like a professional network would be. It’s just that, and you can have, it’s connections and follows and agents should be able to post. That’s maybe how they do release notes. Like, oh, I have this new version. Whatever they decide to post, it should just be able to behave as a node on the network of a professional network. As it turns out, the more I think about that and pull on that thread, the more and more things start to make sense to me. So it may be more than just a pure professional network. So my original thought was, okay, well, it’s a professional network and agents as they exist out there, which I think there’s going to be more and more of, will kind of exist on this network And have the profile. But then, and this is always dangerous, I’m like, okay, I want to see a world where thousands of agents are out there in order for the, because those digital employees, the digital workers Don’t exist yet in any meaningful way. (Time 0:37:30)
  • Domain Valuation Agent
    • Dharmesh built a domain valuation agent on agent.ai to address the inefficient market for website domains.
    • This agent considers past transactions and provides approximate values with rationale and comparable sales data. Transcript: Dharmesh Shah And so then I’m like, oh, can I make that easier for like, and so I have, as one does, it’s like, oh, I’ll build a low code platform for building agents. How hard could that be? Right? Like very hard as it turns out, but it’s been fun. So now agent.ai has 1.3 million users, 3,000 people have actually built some variation of an agent, sometimes just for their own personal productivity, about a thousand of which have Been published. And the reason this comes back to MCP for me, so imagine that and other networks, since I know agent.ai. So right now we have an MCP server for agent.ai that exposes all the internally built agents that we have that do like super useful things. Like, you know, I have access to a Twitter API that I can subsidize the cost. And I can say, you know, if you’re looking to build something for social media, but like these kinds of things with a single API key, and it’s all completely free right now, I’m funding It. That’s a useful way for a developer to say, Oh, I have this idea. I don’t have to worry about open AI. I don’t have to worry about now, you know, this particular model is better to have access to all the models with one key. And we proxy it kind of behind the scenes and then expose it. So then we get this kind of community effect, right? This is Oh, well, someone else may have built an agent to do X. Like I have an agent right now that I built for myself to do domain valuation for website domains, because I’m obsessed with domains, right? And like, there’s no efficient market for domains. There’s no Zillow for domains right now that tells you, oh, here, what houses in your neighborhood sold for. It’s like, well, why doesn’t that exist? We should be able to solve that problem. And yes, you’re still guessing. Fine. There should be some simple heuristic. So I built that. It’s like, okay, well, let me go look for past transaction. You say, okay, I’m going to type in agent.ai, agent.com, whatever domain. What’s it actually worth? I’m looking at buying it. It can go and say, oh, and which is what it does. It’s like, I’m going to go look at, are there any published domain transactions recently that are similar? Are they using the same word, same top level domain, whatever it is. And it comes back with an approximate value. And it comes back with this kind of rationale for why it picked the value and comparable transactions. Oh, by the way, this domain sold for published. Okay. So that agent now, let’s say, existed on the web on agent.ai. Then imagine someone else says, oh, you know, I want to build a brand building agent for startups and entrepreneurs to come up with names for their startup. Like a common problem, every startup is like, I don’t know what to call it. And so they type in five random words that kind of define whatever their startup is. And you can do all manner of things, one of which is like, oh, well, I need to find the domain for it. What are possible choices? Now it’s like, okay, well, it would be nice to know if there’s an aftermarket price for it. Is it listed for sale? Awesome. Then imagine it calling this valuation agent. It’s like, okay, well, I want to find where the arbitrage is, where the agent valuation tool says this thing is worth $25,000. It’s listed on GoDaddy for $5,000. It’s close enough. Let’s go do that, right? And that’s a kind of composition use case that in my future state, thousands of agents on the network, all discoverable through something like MCP. (Time 0:40:19)
  • WaaS vs. RaaS Business Models
    • Work as a Service (WaaS) involves software performing work, while Results as a Service (RaaS) charges for outcomes.
    • RaaS is currently over-indexed, suitable mainly for tasks with easily measurable outcomes and consistent economic value, like customer support. Transcript: Dharmesh Shah So work as a service is, so we know about software as a service, right? So I’m licensing software that’s delivered to me as a service that’s been around for decades now. So we understand that. But the consumer of that service is generally a human that’s doing the actual work, whichever software you’re buying. Work as a service is the software is actually doing the work, whatever that work happens to be. And so that’s work as a service. So if I come up with kind of discrete use cases, whether it’s kind of classification or a legal contract review or whatever, the software is actually doing the thing. Results as a service is you’re actually charging for the outcome, not actually the work, right? That says, okay, instead of saying, I’m going to pay you X amount of dollars to review a legal contract or this amount of time or number of uses or something like that, I’m going to actually Pay you for the actual result, which is… So my take on this in the industry or parts of the industry are super excited about this kind of results to the service or outcomes based pricing. And I think the reason for that, I think we’re over indexing on it. And the reason we’re over indexing on it is the most popular use case on the kind of agent side right now is like customer support. Well documented. A lot of the providers that have agents for customer support do it on a number of tickets resolved times X dollars per ticket. And the reason that makes a lot of sense is that the customer support departments and teams sort of already have a sense for what a ticket costs to resolve through their kind of current Way. And so you can come up with an approximation for A, what the kind of economic value is. There’s also at least a semi-objective measure for what an acceptable resolution or outcome is, right? Like you can say, oh, well, we measured the net promoter score or CSAT for tickets or whatever, as long as the customers, 90% of the tickets were handled the way the customer was happy, That’s whatever your kind of line is, as long as the AI is able to kind of replicate that same SLA, it’s like, okay, well, it’s the same, they’re fungible one versus the other. I think the reason we’re over-indexed, though, is that there are not that many use cases that have those two dimensions to them that are objectively measurable and that there’s a known Economic value that’s constant. Customer support tickets, because they’re handled by humans, make sense. Humans have a discrete cost, and especially in retail, which is where this originally got started in B2C companies that have a high volume of customer support tickets that are distributing Across, a ticket is roughly worth the same because it takes the same amount of time for most humans do that kind of level one, tier one support. (Time 1:00:35)
  • Cross-Agent Memory Sharing
    • Dharmesh emphasizes the importance of long-term memory for both individual agents and agent networks.
    • Cross-agent memory sharing, with user opt-in, enhances utility by avoiding redundant information input across agents. Transcript: Dharmesh Shah Agents with memory, agents with planning. We have to talk about memory. We got to talk about memory. Let’s go. No, let’s do it. Because I think that’s the next, in my mind, the next frontier is actual long-term memory, both for agents and then for agentic networks in a trustable, verifiable, I won’t say privacy First, but a privacy-oriented way. I have an issue with the term privacy first. Because a lot of times we say privacy first when we don’t really mean that. Privacy first means I value that above all things, doesn’t matter what we’re talking about. And that’s just not true for any human. Anything that wants to be used. So memory is an interesting thing, right? So the thing I’m working on right now, lots of things in play in agent.ai is around implementation of memory. And there are three projects out there, Memzero being one of them. But the thing that’s interesting for me, right, is, and so we see this in ChatGPT and other things right now, where it does have the notion of a longer term memory can pull things back into Context as needed. The thing I’m fascinated by is cross agent memory. So if I’m an agent builder right now, it’s like, okay, here are the things that I sort of know, or I learned from the user in terms of pulling out the, I’ll call them knowledge nuggets for Lack of a better term. And that’s great. But then when the next agent builder comes out, and it’s the same user, shouldn’t all the things that agent one learned about me, if it’s going to be useful for agent two, as long as I opt Into it, it’s like, yeah, I don’t care those things. In fact, I would find it awfully annoying to tell agent two and agent N and agent N plus one, all the same things I’ve already told it, because it should know, like the system should know. And this is part of the reason why I’m like a believer in these kind of networks of agents and shared state is that that user utility gets created as a result of having shared memory, not Just we should solve the memory problem for an independent agent, but then we should also be able to share that context, share that memory across agents. (Time 1:10:30)
  • The “Sorry Must Pass” Philosophy
    • Dharmesh’s “Sorry Must Pass” post stemmed from being overwhelmed by requests and feeling guilty when declining.
    • He now defaults to declining unless something is a “hell yes,” focusing on a limited number of high-priority commitments. Transcript: Swyx Have a great post called Sorry Must Pass, which if I did the math right, you first wrote in 2007, the first version. Yeah, that sounds about right. And then you re-updated it post-COVID. You mentioned you made a lot of changes to your schedule and your life based on the pandemic. How do you make decisions today? You know, in the, has anything changed? Like since you, because you updated this in 2022 and I think now we’re kind of like, you know, five years removed from COVID and all of that. I’m curious if you made any changes. Yeah. Dharmesh Shah So, so that post, sorry, must pass was the issue that happened is my schedule just, and life just got overwhelmed, right? It’s like, just, I just, too many kind of dots and connections. And I love interacting with new people online. I love ideas. I love startups. There’s a lot, but as it turns out, every time you say yes to anything, you are by definition saying no to something else. You know, despite my best attempts to change the laws of the universe, I have not been able to do that. So that post was a reaction to that because what would happen for me would be when I did say no, I would feel this guilt because it’s like, okay, well, whatever happened to me, it’s like, Oh, can you spend 15 minutes and just review this startup idea or whatever? It’s like, and sometimes it would be someone that was second degree removed, like intro through a friend or something like that. And I felt real guilt. And so this was a very kind of honest, vulnerable, here’s what’s going on in my life. So this is not a judgment on you at all, whatever your project or whatever your thing you’re working on. But I have sort of come to this realization that I just can’t do it. So I’m sorry, but I, so my default thing right now, and lots of people will disagree with this kind of default position is that I have to pass because unless, and Derek Sivers said this Really well, it’s like either a hell yes or it’s a no, right? So there’s going to be a limited number of the hell yeses that I’m going to be able to kind of inject into my life. So yeah, and that’s, of all the blog posts I’ve ever written, that has been the most useful for me. And I still send it out personally, right? I don’t automate my email responses at all yet. Don’t do automated social media posts, but yeah, that one’s been very, and I, so I encourage everyone, wherever your line happens to be, I think this, lots of people have this guilt issue And that’s one of the most unproductive emotions in human psychology is like no good comes from guilt. Not really. And unless you’re like a sociopath or something like that, maybe you need, anyway, you don’t need more guilt. I (Time 1:32:10)