#459 – DeepSeek, China, OpenAI, NVIDIA, xAI, TSMC, Stargate, and AI Megaclusters
Episode publishedPodcast: Lex Fridman PodcastLex Fridman Podcast
Transcript preview
Preview only
The following is a conversation with Dylan Patel and Nathan Lambert. Dylan runs semi analysis, a well respected research and analysis company that specializes in semiconductors, GPU, CPUs, and AI hardware in general. Nathan is a research scientist at the Allen Institute for AI and is the author of the amazing blog on AI called interconnects. They are both highly respected, read, and listened to by the experts, researchers, and engineers in the field of AI. And personally, I'm just a fan of the 2 of them. So I use the deep seek moment that shook the AI world a bit as an opportunity to sit down
Yeah. So DeepSeq v 3 is a new mixture of experts, transformer language model from DeepSeq who is based in China. They have some new specifics in the model that we'll get into. Largely, this is a open weight model and it's a instruction model like what you would use in chat gpt. They also release what is called the base model, which is before these techniques of post training. Most people use instruction models today, and those are what served in all sorts of applications. This was released on, I believe, December 26th or that week. And then weeks later on January 20th, DeepSeq released DeepSeq
for a model to be open weights, and what are the different flavors of open source in general?
Yeah. So this discussion has been going on for a long time in AI. It became more important since Chat gpt or more focal since Chat gpt at the end of 2022. Open weights is the accepted term for, when model weights of a language model are available on the Internet for people to download. Those weights can have different licenses, which is the effectively the terms by which you can use the model. There are licenses that come from history and open source software. There are licenses that are designed by companies specifically. All of Llama, DeepSeq, Quen, Mistral, these popular names in open weigh
We should also say that this is probably one of the more open models out of the Frontier models. Yeah. So, like, in this full spectrum or probably the fullest open source, like you said, open code, open data, open weights, this is not open code. This is probably not open data, and this is open weights, and the licensing is, MIT license or it's, I mean, there's some nuance in the different models, but it's towards the free in terms of the open source movement, these are the kind of the good guys. Yeah.
DeepSeq is doing fantastic work for disseminating understanding of AI. Their papers are extremely detailed in what they do. And for other teams around the world, they're very actionable in terms of improving your own training techniques. And we'll talk about licenses more. The DeepSeq r one model has a very permissive license. It's called the MIT license. That effectively means there's no downstream restrictions on commercial use. There's no use case restrictions. You can use the outputs from the models to create synthetic data, and this is all fantastic. I think the closest peer is something
Yeah. And, I mean, so it should be stated that one of the implications of deep sea, it puts pressure on LAMA and everybody else on OpenAI to push towards, open source. And that's the other side of open source that, you mentioned is how much is published in detail about it. So how open are you with the sort of the insights behind the code? So, like, how good is the technical reports? Are they hand wavy, or is there actual, details in there? And that's one of the things that DeepSeq did well as they publish a lot of the details.
Yeah. Especially in the DeepSeq v 3, which is their pretraining paper
Keep exploring with a free account.
A free account preserves the short preview. Upgrade to Pro for the full transcript and episode chat.