Podcast
Inferact- Building the Infrastructure That Runs Modern AI
The a16z Show
- Why Inference, Not Models, Is the Hard Problem Now
Inference—keeping large models running efficiently—is now as hard or harder than training models.
Modern LLMs create continuous, unpredictable workloads: prompts vary from a single word to hundreds of pages, outputs are stochastic and stream token-by-token, and agentic workflows introduce long-lived, externally driven pauses. This breaks traditional batching and memory patterns, forcing new solutions in scheduling, KV-cache memory management (e.g., page attention), sharding across GPUs, and state handling for agents.
Infraact (creators of vLLM) argue the solution is a universal, open-source inference layer that abstracts hardware and model diversity so any model can run efficiently on any chip—because production scale, model and chip diversity, and agentic state make inference a systems research problem, not just engineering tweaks.
It is only that vLLM wins, vLLM becomes a standard, and vLLM helps everybody to achieve what they need to do, then our company, in a sense, have the right meaning and to be able to support everybody around it.
— Simon Mo Transcript: Speaker 4 A public story of AI progress is about better models and bigger breakthroughs. But underneath it is a quieter systems problem. How do you schedule chaotic requests efficiently? How do you manage memory when you don’t know when a conversation is actually finished? And what changes when AI systems stop behaving like single-turn tools and start acting like agents that think, pause, and interact with the world over time? This episode focuses on the hidden layer. We examine why inference, the act of running trained AI models, has become one of the most complex and important problems in modern computing, and why Thank you. Open source infrastructure is increasingly central to solving it. Matt Bornstein, General Partner at Andreessen Horowitz, is joined by Simon Mo and Rusa Kwan, co-founders of Infract and creators of the open source inference engine VLLM. This is a conversation about the infrastructure beneath AI and why it may matter more than the models themselves. Matt Levy We are here today with Simon Mo and Wusuk Kwan, lead contributors on the VLLM open source project and co-founders of Infraact, a new AI inference company. Super excited (Time 0:01:59)