John Schulman (OpenAI Cofounder) — Reasoning, RLHF, & plan for 2027 AGI

Today, I have the pleasure to speak with John Chulman, who is one of the cofounders of OpenAI and leads the post training team here. And, he also led the creation of ChadGBT and is the author of many of the most important and widely cited papers in AI and RL, including PPO and many others. So, John, really excited to chat with you. Thanks for coming on the podcast.

Thanks for having me on the podcast. I'm a big fan.

Oh, thank you. Thank thank you for saying that. So the first question I had is we have these distinctions between pre training and post training beyond what is actually happening in terms of loss function and training regimes. I'm just curious, taking a step back conceptually, like, what kind of thing is pre training creating? What does post training do on top of that?

In pretraining, you're basically training to imitate all of the content on, the Internet or on the web, including websites and code and so forth. So you get a model that can basically, generate content that looks like random web pages from the Internet. And, the model is also trained to maximize likelihood where it has to put a probability on everything. So it's, the objective is, basically predicting the next token given the previous tokens. Tokens are like words or parts of words. And, since the model has to put a probability on it, and it's we're training with, to maximize log probability,

Yep. Okay. I think maybe I should take a step back and ask. Right now, we have these models that are pretty good at acting as chatbots. Just taking a step back from how these processes were currently, what would the models release by the end of kinds of things the models released in year. What are capable of doing? What do you see the progress looking like five you know, carry this forward for the next five years?

Oh, yeah. Five years. Yeah. I think, the models will get quite a bit better.

But in what way?

In the first five years. Yeah. So, I mean, I think even in one or two years, we'll find that a lot of, you can use them for a lot of, more, like, involved tasks than they can do now. So you could so so for example, right now, like, you could imagine, having the models to carry out a whole coding project instead of maybe giving you one suggestion on how to write a function. So, you could imagine the model, like, you giving it sort of high level instructions on what to what to code up, and it'll go and, it'll, go and write, many files and test it, look at the output, iterate on that a bit. So ju

And, fundamentally, the unlock is that it can act coherently for long enough to write multiple files of code, or what what what has changed between now and then?