Richard Sutton – Father of RL thinks LLMs are a dead-end

Today, I'm chatting with Richard Sutton, who is one of the founding fathers of reinforcement learning and inventor of many of main techniques used there, like TD learning and policy gradient methods. And for that, he received this year's Turing Award, which if you don't know, is basically the Nobel Prize for computer science. Richard, congratulations.

Thank you, Dwakish.

And, thanks for coming on the podcast. It's my pleasure. Okay. So first question, my audience and I are familiar with the LLM way of thinking about AI. Conceptually, what are we missing in terms of thinking about AI from the RL perspective?

Well, yes. I think it's really quite a different point of view. It can easily get separated and lose the ability to talk to each other. And yeah, large language models have become such a big thing. Generative AI in general a big thing. And our field is subject to bandwagons and fashions. So we lose track of the basic, basic things. Because I consider reinforcement learning to be basic AI. And what is intelligence the problem is to understand your world. And reinforcement learning is about understanding your world. Whereas large language models are about mimicking people, doing what people say

I guess you would think that to emulate the trillions of tokens in the corpus of Internet text, you would have to build a world model. In fact, these models do seem to have very robust world models, and they're the best, world models we've made to date in AI, right? So what do you think that's missing?

I would disagree with most of things you just said. Great. Just to mimic what people say is not really to build a model of the world at all, I don't think. You're mimicking things that have a model of the world, people. But I don't want to approach the question in an adversarial way. But I would question the idea that they have a world model. So a world model would enable you to predict what would happen. They have the ability to predict what a person would say. They don't have the ability to predict what will happen. What we want, I think, to quote Alan Turing, what we want is a machine that

Right.

Where experience is the things that actually happen in your life. You do things. You see what happens. And that's what you learn from. Yeah. The large language models learn from something else. They learn from here's a situation and here's what a person did. And implicitly, the suggestion is you should do what the person did.

Right. I guess maybe the crux and I'm curious if you disagree with this is some people will say, Okay, so this imitation learning has given us a good prior or given these models a good prior, but reasonable raise to approach problems. And as we move towards the era of experience, as you call it, this prior is gonna be the basis on which we teach these models from experience because this gives them the opportunity to get answers right some of the time. And then on this, can train them on experience. Do you agree with that perspective?