Episode 18 - Why AI needs a new kind of supercomputer network
Episode publishedPodcast: OpenAI PodcastOpenAI Podcast
Transcript preview
Preview only
Hello, I'm Andrew Main, and this is the OpenAI Podcast. On today's episode, we're discussing how to make supercomputers better at training models. Joining me are Mark Handley from the Core Networking team and Greg Steinbrecher from Workload Systems. They'll discuss how a breakthrough has made training more efficient so everyone gets smarter models faster.
This has really allowed us to remove one of the key barriers to continuing to scale.
We're talking about a lot of the world's fastest GPUs and making them all work together on a single task.
We know we've won when researchers stop needing to know what network protocol this particular cluster is using.
So tell me a bit about your background.
I started out doing physics and math in undergrad, wanting to basically understand how complex systems work. I always liked the part of physics that's about how do you take this thing that is unknowably complicated and build a simple model that is a complete modern lie, but tells you something about that system, And then build your intuition on that and kind of build more complex models. I ended up doing a PhD trying to build quantum computers.
An ambitious PhD?
You know, little things, little things. Unfortunately, what I liked is big complicated systems, and you'll note that quantum computers don't work, and therefore they don't scale yet. They will someday, but they don't yet work. So I kind of took a look at the chips we were designing to control light for quantum computers, and I went, that kind of looks like a network switch. What if we use this as a network switch? And what I found out pretty quickly was that academia does not know a whole lot about what real, like, data center workloads look like. You get a whole bunch of kind of very toy mode
Mark, you've had a considerable amount of experience trying to get computers to talk to each other and do meaningful work. Tell me more about your background.
Keep exploring with a free account.
A free account preserves the short preview. Upgrade to Pro for the full transcript and episode chat.
This episode could not be prepared.
Try again to prepare the episode for reading and questions.