17:I[22773,["6324","static/chunks/1743cc95-cf85d9d3c941e1e1.js","9640","static/chunks/b11ebc50-4a71c53ad6376343.js","1381","static/chunks/a5c829b0-2b810d87451d7bc3.js","8202","static/chunks/8202-d22828e45db31cd7.js","1158","static/chunks/1158-e7afd83bc0bb7e95.js","2731","static/chunks/2731-6a92cc39ead81676.js","8471","static/chunks/8471-27b2f1bfcc80fa14.js","4995","static/chunks/4995-e3959e973cd91773.js","466","static/chunks/466-fbd8b941a58509fd.js","4404","static/chunks/4404-c62beaa9929a58b5.js","4564","static/chunks/4564-62d6b6f538aa201b.js","8305","static/chunks/8305-8dc2fec6739bebd6.js","6827","static/chunks/6827-1cbf17c0f9af91d0.js","8125","static/chunks/8125-a60a2df0b39490ca.js","7437","static/chunks/7437-ffeed6a63ddf5ef0.js","5598","static/chunks/app/podcast/%5BpodcastSlug%5D/%5BepisodeSlug%5D/page-908156565fd191e4.js"],"TrackEvent"] 19:I[96161,["6324","static/chunks/1743cc95-cf85d9d3c941e1e1.js","9640","static/chunks/b11ebc50-4a71c53ad6376343.js","1381","static/chunks/a5c829b0-2b810d87451d7bc3.js","8202","static/chunks/8202-d22828e45db31cd7.js","1158","static/chunks/1158-e7afd83bc0bb7e95.js","2731","static/chunks/2731-6a92cc39ead81676.js","8471","static/chunks/8471-27b2f1bfcc80fa14.js","4995","static/chunks/4995-e3959e973cd91773.js","466","static/chunks/466-fbd8b941a58509fd.js","4404","static/chunks/4404-c62beaa9929a58b5.js","4564","static/chunks/4564-62d6b6f538aa201b.js","8305","static/chunks/8305-8dc2fec6739bebd6.js","6827","static/chunks/6827-1cbf17c0f9af91d0.js","8125","static/chunks/8125-a60a2df0b39490ca.js","7437","static/chunks/7437-ffeed6a63ddf5ef0.js","5598","static/chunks/app/podcast/%5BpodcastSlug%5D/%5BepisodeSlug%5D/page-908156565fd191e4.js"],"EpisodeMetaCard"] 18:T578,[{"@context":"https://schema.org","@type":"PodcastEpisode","name":"Paul Christiano - Preventing an AI Takeover","description":"Episode details from Dwarkesh Podcast, October 31, 2023. Paul Christiano is the world’s leading AI safety researcher. My full episode with him is out! We discuss: Does he regret inventing RLHF, and…","url":"https://barbellinsights.com/podcast/dwarkesh-podcast/paul-christiano-preventing-an-ai-takeover","datePublished":"2023-10-31T13:56:24.000Z","image":"https://barbellinsights.com/pods/dwarkesh.png","associatedMedia":{"@type":"AudioObject","contentUrl":"https://api.substack.com/feed/podcast/138448008/598c2ea7ddbe4fe5da5c646cde2c898f.mp3"},"publisher":{"@id":"https://barbellinsights.com#organization"},"partOfSeries":{"@type":"PodcastSeries","name":"Dwarkesh Podcast","url":"https://barbellinsights.com/podcast/dwarkesh-podcast","image":"https://barbellinsights.com/pods/dwarkesh.png"}},{"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://barbellinsights.com"},{"@type":"ListItem","position":2,"name":"Dwarkesh Podcast","item":"https://barbellinsights.com/podcast/dwarkesh-podcast"},{"@type":"ListItem","position":3,"name":"Paul Christiano - Preventing an AI Takeover","item":"https://barbellinsights.com/podcast/dwarkesh-podcast/paul-christiano-preventing-an-ai-takeover"}]}]1a:Tacb,

Paul Christiano is the world’s leading AI safety researcher. My full episode with him is out!

We discuss:

- Does he regret inventing RLHF, and is alignment necessarily dual-use?

- Why he has relatively modest timelines (40% by 2040, 15% by 2030),

- What do we want post-AGI world to look like (do we want to keep gods enslaved forever)?

- Why he’s leading the push to get to labs develop responsible scaling policies, and what it would take to prevent an AI coup or bioweapon,

- His current research into a new proof system, and how this could solve alignment by explaining model's behavior

- and much more.

Watch on YouTube. Listen on Apple Podcasts, Spotify, or any other podcast platform. Read the full transcript here. Follow me on Twitter for updates on future episodes.

Open Philanthropy

Open Philanthropy is currently hiring for twenty-two different roles to reduce catastrophic risks from fast-moving advances in AI and biotechnology, including grantmaking, research, and operations.

For more information and to apply, please see the application: https://www.openphilanthropy.org/research/new-roles-on-our-gcr-team/

The deadline to apply is November 9th; make sure to check out those roles before they close.

Timestamps

(00:00:00) - What do we want post-AGI world to look like?

(00:24:25) - Timelines

(00:45:28) - Evolution vs gradient descent

(00:54:53) - Misalignment and takeover

(01:17:23) - Is alignment dual-use?

(01:31:38) - Responsible scaling policies

(01:58:25) - Paul’s alignment research

(02:35:01) - Will this revolutionize theoretical CS and math?

(02:46:11) - How Paul invented RLHF

(02:55:10) - Disagreements with Carl Shulman

(03:01:53) - Long TSMC but not NVIDIA

Get full access to Dwarkesh Podcast at www.dwarkeshpatel.com/subscribe5:[["$","$L17",null,{"event":"page_view","props":{"route":"/podcast/dwarkesh-podcast/paul-christiano-preventing-an-ai-takeover","surface":"web","episode_id":"16350ca1-00a3-45a9-bce3-3b8464da75b3","plan":"free","is_anonymous":true,"podcast_id":"4e3d122c-a06b-45c7-8794-fa36d47d076e"}}],false,["$","script",null,{"type":"application/ld+json","dangerouslySetInnerHTML":{"__html":"$18"},"suppressHydrationWarning":true}],["$","div",null,{"className":"relative min-h-screen","children":["$","div",null,{"className":"mx-auto max-w-7xl px-2 sm:px-6 lg:px-8 pb-5 pt-2 sm:pb-7 sm:pt-3","children":["$","div",null,{"className":"lg:[padding-right:calc(var(--chat-panel-width,0px)+0.5rem)]","children":["$","article",null,{"className":"mx-auto max-w-4xl space-y-8 sm:space-y-10","children":[["$","div",null,{"className":"space-y-2","children":[["$","header",null,{"className":"space-y-3 sm:space-y-4","children":[["$","h1",null,{"children":"Paul Christiano - Preventing an AI Takeover","className":"text-[1.5rem]/8 font-semibold tracking-tight sm:text-4xl/10 text-2xl/8 font-semibold text-foreground sm:text-xl/8"}],["$","$L19",null,{"episodeId":"16350ca1-00a3-45a9-bce3-3b8464da75b3","episodeTitle":"Paul Christiano - Preventing an AI Takeover","canonicalPath":"/podcast/dwarkesh-podcast/paul-christiano-preventing-an-ai-takeover","canonicalPodcastSlug":"dwarkesh-podcast","podcastName":"Dwarkesh Podcast","episodeDate":"2023-10-31 13:56:24+00:00","summary":null,"description":"$1a"}],null]}],"$L1b"]}],null,"$L1c"]}]}]}]}]]