Skip to content

Podcast

Organizational Context for AI Coding Agents With Dennis Pilarinos

Software Engineering Daily

Source ↗ ← All highlights
  • Context Is Decision Grade Knowledge
    • Context equals the decision-grade information both people and agents need to make correct choices.
    • Dennis defines it as tribal knowledge across code, docs, PRs, chat and telemetry that explains why things were done. Transcript: Kevin Ball It’s interesting. Yeah. Even setting context about context feels very meta, but yeah, it is one of the things that I think is a term that is very overloaded. I think certainly within the last few years, we’ve seen this evolution from prompt engineering to context engineering. So if I think about it purely from a technology perspective, it’s all the information that the machines need in order to make the right decision so that they have what I would consider Decision grade context. From a person, it’s all effectively the tribal knowledge that exists as a consequence of working in an organization in order to get your job done. So if you’re new to an organization, your context for how that team works is probably relatively insignificant, right? You don’t know a lot on day one. And getting up to speed is really that context loading. Why do we do certain things certain ways? Why have things been done certain ways? Who do I talk to about things? All of that kind of stuff. You know, it’s this interesting problem because it’s the intersection both of like people and their context and machines and their context. And I think there’s actually a lot of analogies. (Time 0:03:13)
  • Context Lives Across Many Systems
    • Context lives everywhere: Slack, PRs, code, docs, bug trackers and production telemetry.
    • Unblocked aggregates those disparate sources so queries reflect the full operational picture, not just source code. Transcript: Kevin Ball I think context lives both for AI agents and for people and all the systems that people use to build applications, right? So it’s in Slack. It’s in Teams. It’s in your code and the history of that source code. It’s in the pull requests that exist there. It’s in your bug trackers, it’s in your documentation systems, it’s in your runtime production systems, right? Like in Sentry and Datadog. It’s basically all the places that your application uses to operate and all the places that you used in order to figure out what to build and how to build it. That’s where that context layer, I think, lives. (Time 0:04:58)
  • From Q&A To Slack And IDE Answers
    • Unblocked started as a Q&A platform where engineers ask questions and get high-quality answers quickly.
    • It expanded to IDE, Mac app and Slack integrations that reply in seconds when confident. Transcript: Kevin Ball For sure. I think one of the hardest problems is to really understand where the context lives, so finding it. And then that’s effectively a form of a search problem. And then trying to understand, well, what is actually the truth? Because what you’ll find is that you actually get drift across these different systems, right? The source code will say one thing, a Slack conversation says another, a Jira ticket says another. And so being able to reconcile or do effectively conflict resolution in a way that doesn’t lead a person askew is actually a really hard problem. And so, I mean, again, we keep talking about it within the context of AI agents, but imagine talking to someone who just constantly lied to you around the questions that you had, right? Like you’re not going to trust them. They quickly become out of the loop of people you chat with to get information. And so that person, what are the characteristics of that person? Well, they’ve typically been around the code base or the application for a while. They’ve participated in a lot of those conversations. They understand why things have been done. They understand what the truth is for all of those things. So in the same way that you’d want to replicate that person, the software that Unblock tries to build basically does the same thing. We’d like to be thought of as like a member of your team who’s been aware of every conversation that’s ever been had for every part of your code base. So you can ask questions and get answers from it, basically. (Time 0:06:08)
  • Conflict Resolution Is Central
    • Reconciling conflicting sources is a core challenge: docs, code and chat often drift.
    • You must perform conflict resolution and build a trustworthy ‘team member’ that remembers conversations and decisions. Transcript: Kevin Ball To really understand where the context lives, so finding it. And then that’s effectively a form of a search problem. And then trying to understand, well, what is actually the truth? Because what you’ll find is that you actually get drift across these different systems, right? The source code will say one thing, a Slack conversation says another, a Jira ticket says another. And so being able to reconcile or do effectively conflict resolution in a way that doesn’t lead a person askew is actually a really hard problem. (Time 0:06:11)
  • Provide Agents Decision Grade Context
    • Give agents the same end-to-end context as humans to improve code generation and reasoning.
    • Expose the context engine via APIs, CLI, or skills so orchestrators and IDEs can call it. Transcript: Kevin Ball Augmented a lot of the agent so that they have very similar context. So as part of their reasoning experience, they have the full end to end context to make the decision for when they’re actually going and writing code. (Time 0:09:24)
  • Make Context Permission Aware At Runtime
    • Enforce permissions at runtime so the context engine only uses data the requester can access.
    • Bind identities across GitHub, Slack, Notion to respect access rules during queries. Transcript: Kevin Ball Someone asks a question about Project Tantalus, some arbitrary project. What’s the release date for Project Tantalus? And that lives in a Google Doc I currently have access to. Our demo is very straightforward. You ask the question, you get the response. We revoke access in that Google Doc. You ask the question and Unblock can’t answer the question. It says, based on the information about you and what I can find, I don’t know what the release date is for Project Tantalus, even though it exists in the underlying knowledge base that Exists there. So at runtime, you have to be able to understand who that person is, so a strong sense of identity, what access control they have across those disparate data systems. (Time 0:14:19)
  • Enrich Noisy Data With Derived Artifacts
    • Data enrichment is essential because teams rarely keep docs up to date.
    • Unblocked diffs every PR and embeds those changes to build a richer knowledge graph from existing artifacts. Transcript: Kevin Ball Says all my documents are super well organized and up to date, right? It just doesn’t exist. And so I think that is the predisposition that you need to have when you build a product like this, either as a company like ours, or if you’re building something internally. So you get a lot of noise and expecting a team to organize and keep their documents up to date, I think is a bit of a fool’s errand. No one’s going to go to the heavy lifting, both to start and ongoing to maintain that. So I’ll give you an example of one of the things that we do just at that data layer. For people who ask historical questions, we actually look at every single pull request that’s ever been opened within that source code, within the repositories that we’ve been granted Access to. And what I found in my experience is that most developers aren’t great at writing pull request descriptions, right? They’re usually relatively brief and they don’t give you a ton of context. So what we actually do is a diff on the pull request and embed that and basically build a background knowledge graph effectively for that organization. (Time 0:15:42)
  • Use Hybrid RAG With Agentic Tools
    • Combine lexical and semantic search in a hybrid RAG plus agentic approach to handle temporal and fuzzy queries.
    • Expose tools/APIs so orchestrators and agents can call the context engine programmatically. Transcript: Kevin Ball I hate using all these terms because they feel like word sounds or whatever, but it’s an agentic, but underneath hybrid RAG system that does both lexical and semantic search. So increasingly what you’re seeing is that RAG is run out of its limitations and that a lot of these agentic tools can actually effectively meta go and get the answers that they need themselves. So it’s interesting to see, I was talking to someone about it this morning, where you think about Paul Graham’s famous software is eating the world, right? Everything basically has an API. And if the tool can call into that API, and it does it in a way that respects the permissions model, what have you, you can start to really light up a whole bunch of these scenarios. So we expose a series effectively of tools across this so that those orchestrators, those agents can actually go and call into that. (Time 0:17:38)
  • Expose Fast Versus Thorough Modes
    • Let users choose fast vs thorough responses to balance speed and accuracy.
    • Dennis observed most users prefer waiting for higher-quality answers rather than quick but wrong ones. Transcript: Kevin Ball In terms of debugging it, the underlying technology continues to get better where you can start talking to it like a person and it will explain itself. If you think about even two years ago, people were deeply concerned with hallucinations. And I think that people are just less sensitive to it. I think it’s a consequence of two trends happening. One, people are developing a sense for how to figure out when the thing might be overly assertive in terms of its confidence. It’s like, you’re absolutely right. You’re like, well, okay, I’m going to have to double click on that. So knowing how to use the tool a little bit better. And then second, obviously, the technology has gotten a lot better in terms of being able to check its work, as it were, we actually have this interesting, I love the human factors of Building software. And so we built this feature where you get to decide how quick or slow you want your responses to be. And the way we surface that is, do you want fast answers or do you want thorough answers? And there’s a lot of internal debate as to what people are going to want to do and where people would set those defaults. We’re a very metric company. It’s an overwhelming majority of people who are willing to wait for a higher quality answer than get a fast one that’s wrong, which in retrospect seems obvious. (Time 0:20:00)
  • Context Upfront Improves Plans And Reviews
    • Agents improve planning and reduce left-of-cycle code review drift by understanding standards and architecture up front.
    • Customers use Unblocked to generate better plans so PRs and reviews have fewer surprises. Transcript: Kevin Ball It’s a great question. Like, I think what we’re seeing in the industry is like traditional software development is now progressively relegating the task of generating software to an autonomous agent that Will go and do that for you. How do you make sure that it’s actually correct and what is done through that process, right? So I think that’s where we’re starting to see things that both help the person and I think the agents really get on the same page. So when you think about plan generation, I think that’s really solving two problems, making sure that contract between the person and the agent is well understood such that when the Agent goes and executes the plan, there’s no surprises at the end. So it’s fascinating. We see a bunch of our customers who use Unblocked as part of that planning process. So again, going back to that mental model of like, of course, you would spend more time planning as you go to implement a feature. And I’m intentionally being like vague around, am I talking about a person or an agent? So when people are using agents to create plans, and they augment or context enable that, the plans are dramatically better because it understands the coding standards. It understands some of the architectural decisions. It knows all the things that you would get caught up in your pull request or in your code review process. You just skip all of that left of the software development lifecycle. So you’re not getting caught there. (Time 0:22:10)
  • Source Code Is The Strongest Truth Signal
    • Source code is treated as the strongest signal of truth when reconciling context after work is done.
    • Changes, PRs and diffs provide breadcrumbs to infer indirect context when conversations aren’t codified. Transcript: Kevin Ball User experience to be, I think sometimes you will have those agents go and effectively say, this has been done or this has been redressed or whatever it might be. Depending on the company and their level of comfort around someone going and writing back to their systems, they might or might not allow that. Maybe if you think about from where you end up, when that code gets written and that feature is implemented or what have you, we do place a decent amount of weight in this conflict resolution Strategy on the source code itself, right? Like at the end of the day, source is the truth, right? Of what’s actually happening at that point in time. So if there are discrepancies across these different things, especially as that work has been done, when you go to reconcile them, the source code is a really, really strong hint for What actually went and got done. And increasingly, you can look at, well, what are the changes that got you there? (Time 0:24:30)
  • Code Review Caught Dangerous Secret Use
    • Dennis gave a personal example where Unblocked’s code review flagged insecure secret handling in his demo app.
    • He then asked an LLM to implement the recommended fix, illustrating tighter agent-review loops. Transcript: Kevin Ball I’m building an internal tool. I was using Cloud Code to do it. That internal tool would require us to do some secrets management in production. Cloud, just as an experiment, without knowing how unblocked work, basically said to secure the API key in a way that would absolutely break us into jail. But I was like, oh, let’s just see what happens, right? We’ll just stick it in an environment variable. And I’m sure nothing bad will happen when someone compromises your infrastructure, right? That’s not the first thing they do is dump out the environment variables. Anyway, so I’m like, Okay, great. Off you go. Open the pull request. Unblocked code review looked at it and said, like, No, no, the way that you’re supposed to do this is actually like encrypted and vault, decryrypt it sits in memory, so on and so forth. (Time 0:32:06)
  • Code Review Becomes The New Bottleneck
    • The SDLC bottleneck is shifting from coding to review and maintenance as agent-generated volume grows.
    • Tooling must evaluate organization standards to make AI-generated contributions maintainable. Transcript: Dennis Pilarinos Yeah. Well, and this gets to this mental model question of how do we update our mental models when the agents are doing all of the actual writing of code? Kevin Ball Yeah. It’s cliche, but it is an incredibly exciting time to be building software these days, right? Because a lot of the things that we just took as fundamentals as a byproduct or as a consequence of building software are being completely reevaluated. You know, we started this call by saying, you know, I’ve been building software for the better part of 20 years, same way as you have. And there’s this interesting as a consequence of how the software or how software is being built now where you’re like, well, how do I make sure that I understand how it’s evolving? Yeah. Dennis Pilarinos I mean, I feel like things have changed more in the last three years than in most of that 20 years that I had. It’s absolutely wild. Kevin Ball Yeah, yeah. You know, I’ve seen the evolution basically from like on-premise to cloud, cloud to mobile, and arguably like this wave. And, you know, you see these rates at which companies and people adopt this technology. As of November, December of last year, when Claude became a fun thing for you to hack on over the holidays. Like, again, it’s a stepwise function. Like, I think that will be a significant moment in time where you make it very easy for people to build a lot of software very, very quickly. (Time 0:34:08)
  • 30sec Snip Transcript: Kevin Ball It’s probably an unsatisfying answer, purely from a technology perspective. But like, at the end of the day, like the software is a means to an end to solve a specific problem for someone. So early in my career, I was introduced to this model called BXT, Business Experience Technology, right? So what’s the business problem we’re trying to solve? What’s the experience that best enables that solution? And then what’s the technology that provides that experience? And so that’s how we think about it. (Time 0:44:56)
  • Design Context Tools Starting From Business Needs
    • Start with the business problem and user experience before technology when building context tools.
    • Dennis applies BXT: Business then Experience then Technology for focus and product-market fit. Transcript: Kevin Ball So what’s the business problem we’re trying to solve? What’s the experience that best enables that solution? And then what’s the technology that provides that experience? And so that’s how we think about it. For everything that we’ve done is like almost all the projects that I’ve ever kind of started up or worked on are as a consequence of my frustration with the inability to do something. Right. So I’m just like, I’m like, I have a very specific business need. So like for buddy build, I had a very, very straightforward workflow, get push, a build happens, it gets deployed to my device. Like CI, at the end of the day, right? CI, CD. And it was incredibly difficult to do that in the mobile ecosystem. Well-understood problem. Mobile, completely unsolved, really, in earnest. And so the lack of a very polished experience was what caused us to create BuddyBuild. Increasingly, like when we started to build Unblocked, it’s the same 10 people, almost the exact same 10 people who worked at BuddyBuild who wanted to build Unblocked. So, we literally got the band back together as a consequence of this frustration of trying to get the information we need to get our jobs done. (Time 0:45:15)
  • Tooling Speed Must Be Matched By Craftsmanship
    • Faster app creation won’t guarantee success; craftsmanship and understanding of business goals still separate winners.
    • Dennis compares improved tooling to a nail gun: it speeds work but doesn’t replace judgment about structure and quality. Transcript: Dennis Pilarinos Well, I think this gets back to our question a little bit about like, what is the equivalent of knowing about the door and where it’s swinging and those types of things? Like, what does it mean to be a software engineer in this world? You need to be thinking about that business and experience layer. You can’t just be thinking about, oh, I have a well-specified problem and I get to translate into machine readable code. Kevin Ball Yeah, that’s exactly right. It’s like, you should have an opinion for how wide that door is and which way it swings, you know, like the two kind of dimensions that you’d immediately think about. But like most software is a lot more complicated in terms of the dimensions you need to think about. And so I think that is one of the things that will truly differentiate the ability for it. While apps are being built very rapidly, which ones are going to be commercially successful or what have you, is the ones that have great craftsmanship. (Time 0:47:28)