Reiner Pope – Chip design from the bottom up
Episode publishedPodcast: Dwarkesh PodcastDwarkesh Podcast
Transcript preview
Preview only
I'm back with Rainer Pope who is the CEO of Maddox which is a new AI chip company. Last time we were talking about what happens inside a data center. Now I understand what happens inside an AI chip. How does a chip actually work? Full disclosure, by the way, I am an angel investor in Maddox. So hopefully you have designed a good chip. Also, you're listening to this on an audio platform, it's much preferable to watch this Blackboard lecture on a platform where you can see what's happening, so switch over to YouTube or Spotify. So I'll start with sort
of the very smallest fundamental unit of of chip design, and we'll sort of build up into what an overall, like, actual production chip, what are the components of that. Yep. At the very bottom level of a chip, the primitives that we work with are logic gates, which are very simple things like and or not. And then these are connected together by wires that have to be laid out physically as metal traces on a chip. The main function that AI chips want to compute is multiplication of matrices, and really inside that is the fundamental primitive is multiply accumulative, just like of pairs of numbe
And and can I ask a clarifying question? Why is this the natural primitive for, you know, whatever computation happens inside a computer?
Yeah. So there there's a few reasons for this. It it's a little bit more efficient, but the the reason it's natural for AI chips is that if you look what's happening during a matrix multiply, the what is matrix multiply? In very short, it is there's a for loop over I and over j and over k of output I k plus equals to input I j times other input j k. And so multiply accumulate happens at every single step of a matrix multiply.
Makes
sense. And then the other observation is that the precision will almost always be higher in the accumulation step than in the multiplication step. This is maybe specific to AI chips, but you're you're multiplying low precision numbers, but then when you accumulate, errors accumulate quickly, and so you need more precision here. So this is why we've chosen to do a four bit multiplication and an eight bit addition.
Let me make sure I understood that. There's two ways to understand that. One is that the value will be larger than the inputs, and the other is that if it was a floating point number, would be maybe that that part is like less intuitive to me, but it's maybe the same principle?
It it is really the same principle. I guess the sort of I mean, I guess the the separate principle is that as you are summing up this number, you are summing up a whole bunch of numbers, and so you've got a lot of rounding errors accumulating. Whereas in this case, there's, like, there's there's only one multiplication in that chain, so there's not a lot of rounding errors accumulating in the multiplication.
Are you summing up a whole bunch of numbers? Is just two numbers?
Keep exploring with a free account.
A free account preserves the short preview. Upgrade to Pro for the full transcript and episode chat.