Reiner Pope – The math behind how LLMs are trained and served
Episode publishedPodcast: Dwarkesh PodcastDwarkesh Podcast
Transcript preview
Preview only
Today, I'm interviewing Rainer Pope, who is CEO of Maddox, which is a new chip startup. Previously, he was doing TPU architecture and many other things at Google. This is a very different format from my usual interviews. This is gonna be a Blackboard lecture. I'll ring it up in a second. We in fact built this whole new studio with specifically this format in mind. And so it's a pleasure to get to inaugurate it with you. We're gonna be talking about model architecture, ML infra, many other things. And the reason I think it's an important topic is because once you actually understand how trainin
Yeah. Very happy to be here.
Just a heads up, this is a lecture with graphs and equations and all that stuff. So if you can, I would really recommend watching it on a video platform like YouTube? Okay. Full disclosure, I am a angel investor in Maddox, but that's unrelated to this podcast. Reiner, maybe to kick us off, I'll ask this question. So we have a couple of companies like Claude and Codex and Cursor are offering something like, Fast Mode, where for six x the price, they'll give you streaming tokens at 2.5 x the speed. Mechanically, I'm curious what's going on here. Why why is it the case that you can pay more to ge
Great. I mean, to jump to a little bit to jump to the conclusion, the big effect is batch size, but what what we're gonna do now is quantify exactly what that looks like and what its implications are on latency and cost. There's gonna be another effect, which is you can call it speculative decoding or multi token prediction. Can We maybe come back to that later, but I think the first thing that we'll talk through is batch size. So what I'd like to introduce is sort of the two principles of analysis. Firstly, we're gonna look at a roofline analysis of how we run a transformer model on a cluster
Maybe I'll just interrupt from time to time to ask some very naive questions or to clarify some basic points. But just for the audience, you're not serving one user at a time. The batch refers to the fact that you're serving many different users at the same time. Yeah. And that's a whole batch.
Yeah. So I can motivate the batch at least a little bit. I mean, we will see exactly why batch is such a favorable optimization. But what will turn out to be the case is that if you do not batch together many users, the costs and the economics you get can be like a thousand times worse than if you do batch many two users together. And we'll be able to see that quite especially. And then number of active parameters. This is saying, if I look at, for example, a DeepSeq model, the DeepSeq v three model has about 37,000,000,000 active parameters and then 700,000,000,000 total parameters. So this i
And maybe just back in, let's let's just explain what the KV cache is
real quick. Yeah. So when I do a forward pass let me draw actually a how the autoregressive inference works. So this is during decode. So if I think I have a bunch of tokens of text, I'm growing a tensor because, ultimately, the tokens are represented as some, like, tensor of in some embedding dimension. And then in this direction, I have the sequence length. The work of running a decode is I I have to run each token through a through a whole bunch of matrix multiplies over a bunch of different layers. And I have I have in general, I'm gonna have to do that work over all of these tokens. But t
Mhmm.
Keep exploring with a free account.
A free account preserves the short preview. Upgrade to Pro for the full transcript and episode chat.
This episode could not be prepared.
Try again to prepare the episode for reading and questions.