20VC20VC: Who Wins the Inference Market | Has Open-Source LLM Inference Peaked? | What Happens to the Cost of Compute | The Real Limits of Post-Training and RL with Erik Bernhardsson, Modal CEO
2 Nov 2026 · 62 min · Episode page ↗The true long-term opportunity in AI infrastructure lies in building a comprehensive, multi-product platform that aggregates and efficiently manages diverse compute demands, rather than specializing in single commodity offerings like raw inference.
Summary
Erik Bernhardsson, CEO of Modal, shares his perspective on the rapidly expanding AI inference market, arguing that its immense size will drive specialization across product types, not just model providers. He explains how Modal aims to capture this growth by building a broad, integrated infrastructure platform, contrasting this with the commoditization risk of single-product offerings. Bernhardsson also touches on the intense competition for GPUs, the changing economics of software margins in AI, and the future of open-source models versus frontier models.
Interesting ideas
AI Infrastructure: From Commodity to Comprehensive Platform. The inference market is a new cloud, but raw 'tokens for dollars' is a commodity race to the bottom. Winning requires building a platform with diverse products like post-training, sandboxes, and agentic workflows. This breadth creates switching costs and allows for greater gross margins, moving beyond the 25-40% common today toward 40-60%, though 90% software margins are largely gone due to value being split across multiple layers of the AI stack.
Aggregating Global Compute for Peak Efficiency. Modal's custom infrastructure (file system, container runtime, scheduler) allows it to treat all global compute as one pool. By aggregating thousands of unpredictable customer workloads, Modal can manage demand with high efficiency, scaling resources up and down dynamically. This eliminates the need for customers to do complex capacity planning or lock into long-term GPU reservations, providing a significant value proposition.
Open-Source LLM Inference Has Not Peaked. While frontier models are becoming cheaper, the guest strongly disagrees that open-source LLM inference has peaked. He argues that both frontier and open-source models will continue to grow exponentially (5x-20x in the next 12 months) because the overall market for AI applications is far larger than anticipated. Open-source models will always serve a need for fine-tuning, control, and customization.
The Strategic Importance of Brand and Community in B2B Tech. In today's competitive landscape, brand, community, and a distinctive company personality are crucial for startups, even in 'boring' B2B tech categories. Drawing from a consumer background at Spotify, the guest emphasizes creating 'magic moments' through developer experience and visual identity. This approach fosters a sense of movement and excitement, differentiating companies from competitors and attracting users through 'coolness' and community engagement.
Limits of Post-Training and Reinforcement Learning. Post-training and reinforcement learning excel in tasks with finite, fast feedback loops, like coding or cybersecurity. However, their efficacy might hit limitations for tasks with longer feedback cycles, which characterize much of advanced human engineering. While AI will boost developer efficiency, the core job of software engineers—understanding human requirements and translating them into complex, long-term systems—will continue to evolve rather than be fully automated.
Questions worth thinking about
If specialization is the future of the inference market, how will players decide when to integrate adjacent products into a comprehensive platform versus maintaining a focused, best-of-breed approach for specific solutions?
Given the 'open-air drug market' for GPUs, what long-term strategies can infrastructure providers like Modal employ to mitigate the financial risk of fluctuating hardware prices while continuing to scale capacity?
As AI capabilities continue to expand, how might the definition of 'software engineering' evolve beyond the guest's 10-20 year outlook, and what new skills will be most critical for human engineers in an agent-augmented future?
Written by AI from the episode's full transcript. It can get things wrong: if something here looks off, tell us at hello@podpings.com.