Joe Carlsmith - Otherness and control in the age of AGI
Episode publishedPodcast: Dwarkesh PodcastDwarkesh Podcast
Transcript preview
Preview only
Dwarkesh: Today, I'm chatting with Joe Carlsmith. He's a philosopher, in my opinion, a capital g great philosopher, and you can find his essays at joecarlsmith.com. So we have a g b t 4, and it doesn't seem like a paper clipper kinda thing. It understands human values. In fact, if you help have it explain, like, why is being a paper clipper bad?
Or, like, what would just tell me your opinions about being a paper clipper. It'll, like, explain why the galaxy shouldn't be turned into paper clips. Okay. So what is happening such that we have a system that takes over and converts the world into something valueless?
Joe: One thing I'll just say off the bat is, like, when when I'm thinking about misaligned AIs, I'm thinking about or the type that I'm worried about. Yeah. I'm thinking about AIs that have, a relatively specific set of properties related to agency and planning and kind of awareness and understanding of the world. One is this capacity to plan and kind of make kind of relatively sophisticated plans on the basis of models of the world, where those plans are being kind of evaluated according to criteria. Mhmm.
That planning capability needs to be driving the model's behavior. So there are models that are sort of, in some sense, capable of planning, but it's not like when they give output, it's not like that output was determined Yeah. By some process of planning. Like, here's what'll happen if I give this output. Do I want that to happen?
The model needs to really understand the world. Right? It needs to really be like, okay. Here's what will happen. I'm you know, here I am.
Here's my situation. Here's, like, the politics of the situation. I really, like, kind of, having this kind of situational awareness, to be able to evaluate the consequences of different plans. Yeah. I think the other thing is, like so the verbal behavior of these models, I think need bear no so when I talk about a model's values, I'm talking about the criteria that that, the that kind of end up determining which plans the model pursues.
Right? And a model's verbal behavior, even if it has a planning process, which g p d four, I think, doesn't in many cases, its verbal behavior just doesn't doesn't need to reflect those criteria. Right? And so, you know, we know that we're going to be able to get models to say what we wanna hear. Right?
We, that is the magic of gradient descent. You know? If you, you know, modulo, like, some difficulties with capabilities, like, you can get a model to kind of output the behavior that you want. If it doesn't, then you you crank it till it does. Right?
And and I think everyone admits for suitably sophisticated models, they're gonna have very detailed understanding of human morality. But the question is, like, what relationship is there between, like, a model's verbal behavior, which is you've essentially kind of clamped. You're like, the model must say, like, blah things, and the criteria that end up influencing its choice choice between plans. And there, I think it's at least I'm kind of pretty cautious about being like, well, when it says the thing I forced it to say, or, like, you know, gradient descent to
Keep exploring with a free account.
A free account preserves the short preview. Upgrade to Pro for the full transcript and episode chat.