Inside AI's GPU Economy & Infrastructure Boom | Sam Hogan, Kuzco
Episode publishedPodcast: EmpireEmpire
Transcript preview
Preview only
95% of all AI compute was used for training and only 5% is used for inference. And we're at this kind of interesting moment right now where the pendulum is swinging the other direction. The models have become useful. So once you produce the model, you actually, you're going to spend compute to actually make that model do work for you, in the real world to power your applications, your chatbots, your data extraction, all sorts of things.
All right, everyone. Welcome back to Empire. We have a very special guest. I've known Sam for quite a bit of time. I'm an investor in Cusco's company. Sam is one of those people that I go to, my lizard brain to wrap my head around what's going on in the world of AI. So, Sam, welcome on the show. Thank
you. Good to be here. So I wanna
start with just general background, who you are, what you're building, so people know, and then that will take us into interesting places.
Yeah. So I'm Sam Hogan. I'm the founder and CEO of Koosko. Koosko is a spot market for AI inference. So we match sellers of idle compute cycles. These are typically, compute cycles in higher end consumer devices and data centers from around the world. And we pair those with developers on the other side who want access to the output of these models. Fairly early stage, we launched in March. We have about 5,000 GPUs connected to the network at any given time. We have a test net right now where people contribute GPUs in order to earn points and working on kind of what the demand side of the netwo
it. I'm pulling up a tweet here, which may or may not be correct, but I think we talked about this during breakpoint. But there was a shortage of, like, h one hundreds for a while there. And then and then the price obviously spiked, and the the Ionets of the world really come out under this premise of, like, oh, you know, we'll we'll allow for idle capacity and we'll make the marketplace more efficient. Crypto's really good at that. Right? We're using stablecoins, transcending, you know, just connecting and creating a marketplace that is just more efficient relative to localized markets. The p
Yeah. So, this was definitely true. There's sort of sort of this like platitude from, you know, early 2023 that GPUs are really scarce and, you know, everybody needs h 100. And that was true at the time. Nvidia had only shipped, you a few 1000 H100s, maybe 10,000 or more, but not that many to meet the demand, that we saw primarily for training. So I think at that point in time, people were mostly talking about these massive training runs. You need these really highly centralized, parallelized compute clusters where you have, you know, a 1000, 10000, 30000 H100s that are all interconnected usin
Let's double click on that, you know, because people may not really appreciate the difference between training and inference. And I know you were major on that. Like, let's just, like, go go go there.
Yeah. So training or or pretraining is a process of, you know, you take a ton of data, you take a ton of compute, and you crunch all this data using this compute kind of over and over again. And in order to train a model, the models that we have today are in the 100 of billions of parameters, Closed source models are in the trillions of parameters. These are huge. You know, a trillion parameter, you know, full precision model is something like, you know, 4 terabytes. So so like pretty large large models and, you know, you need a lot of a lot of RAM to, like, load that thing, into a GPU to to i
Keep exploring with a free account.
A free account preserves the short preview. Upgrade to Pro for the full transcript and episode chat.