How Open-Source AI Became Critical Infrastructure
Episode publishedPodcast: The a16z ShowThe a16z Show
Transcript preview
Preview only
The fun thought experiment is if GPUs dropped in price by 99%, then do we get back to a real open source
world? If moderation is never solved, in the future, people will go to OpenWait by default because that is where you know for sure you can control your guardrail for trusted use cases.
And can you talk about where VLLM sits in that stack?
VLLM is a inference engine. It is kind of like databases and operating system and other critical software to power AGI. NVIDIA, AMD, Google, their newest chip will make sure VEON can run on them. And then a lot of cases, they use VEON as a benchmark. We're bridging almost a 10 x gap. For proprietary models, there is a regular mode and fast mode. But for open weight, every provider can offer potentially even 10 different levels of speed.
Five years from now, open source AI models, have they closed the gap with frontier models?
Capability wise, I don't really see a big gap,
not even today, because Open source AI has become one of the most important forces shaping the industry. But making frontier models available to the world requires far more than releasing model weights. It requires an entirely new infrastructure layer. In this episode, Elena Berger and Matt Borenstein are joined by Simon Moe, cofounder and CEO of InfraRack, to discuss how open source inference became critical infrastructure for AI, why enterprises are increasingly turning to open weight models, and what the next generation of AI systems will require. They also explore model licensing, inferenc
Today, we're here with Simon Moe, cofounder of InfraAct and a lead maintainer of VLLM, the open source inference engine now running on half a million GPUs at any moment. We're also joined by Matt Bornstein, an e sixteen z general partner. Simon, Matt, thank you so much for joining us. I think first we should start with open source AI and kind of the more recent history of open source AI. So VLLM actually has its origins kind of back in 2022 pre chat GPT. And your team set out to make a slow open source demo faster and instead just found this pile of unsolved problems. So can you talk about wha
Yep. Good to be here. So serving large language model is a fundamentally different problem because serving it requires to run it on accelerators like GPUs or TPUs and it is a computationally intensive process that will require a lot of engineering and ensuring that for each request, user can see the LM's response quickly and efficiently. So this typically means we need to handle differences in input distribution, how long each request is, output distribution, which is nondeterministic, and batching and scheduling a lot more in the at the core of the inference engine.
Keep exploring with a free account.
A free account preserves the short preview. Upgrade to Pro for the full transcript and episode chat.