AI researchers debate how close we are to recursive self-improvement
Episode publishedPodcast: Dwarkesh PodcastDwarkesh Podcast
Transcript preview
Preview only
Today, I'm chatting with three of my AI researcher friends from whom I learn a lot every time we talk and who also happen to be at somewhat open ish, labs and companies so you guys can actually, say things on the record. I'm joined by Baron Millich, who is the CTO of Zyphra, which is developing open source models. John Schulman, who is the chief scientist at Thinking Machines, previously the cofounder of OpenAI, who led the RLHF work that led to ChechiPT. And Charlie O'Neil, is head of model training at Base Ten. The first question I have, if we're in 2036, it's been ten years, and we don't ha
I mean, like, my reason would just be, like, it's got to be this sort of like, there's been a classic thing almost like Marvell's paradox. Right? Where, like, we see, like, you know we think of the AI being like, it can do this, it's going to be amazing. Right? Like, if it can solve these hard maths problems, if it can win the chess, blah blah blah. And then it solves these things, and then it's, like, not that impactful. Obviously, it's somewhat impactful, but, like, not everything. It's like, if somehow that continues and, like, there's never, like, the true, like, spark of generalization th
Yeah. I agree with that. Humans have a lot of advantages over models now, and each time a new model comes out, it'll catch up in some of these areas. But you end up getting bottlenecked by the places where the model is weaker, and where it has worse judgment or the models can't check themselves well enough. Yeah. So there's this cycle that keeps repeating where people think where a new model comes out and people are blown away and they're like, is it. This is AGI, but then they use it a bit and then it starts to feel dumb after a month or so. So that cycle just might keep going and it's hard t
For me it's like a question of how far off like this global optimum of a learner you could have on a chip is like the transformer plus like RL, basically like the current recipe. So I think people imagine that even once you have an agent which is better than all humans at AI research, even if it's like 0.1% better than all humans, then the fact that you can run hundreds of thousands if not millions of these in parallel, you can run them much faster, like chips are gonna speed up. That's gonna outweigh every other bottleneck and you're eventually just gonna hit this very fast takeoff with recur
Sorry. But do do think the discontinuity will be harder than anything that's come since 2012?
If if we had the answer to that, that that we'd kind of have the ability to implement it. But, like, maybe there's the distinguished we should distinguish between the discontinuity which adds to the current paradigm. Again, it's cumulative. There's some thing beyond the RL that we have to discover and maybe they're capable of connecting the dots in that straight line. Or again, how far off the global optimum are we? Do we have to go back and throw out gradient descent and neural nets in general? And I don't think if you continue to scale up the current paradigm, an LLM, no matter how many LLM'
Yeah. The only hope really is if deep learning just can't get us to an AI, which is at least, can dominate human research and human development, including the human ability to come up with new paradigms and so forth. Or, like, I don't know, maybe human maybe humans would never also never have discovered the, the next learning architecture, but, to the extent humans could have discovered it eventually. But it just seems like, I don't know, if if you just look at the progress that's happened in 2012 till
Keep exploring with a free account.
A free account preserves the short preview. Upgrade to Pro for the full transcript and episode chat.