What happens if the companies racing to build superintelligence succeed before anyone knows how to control it?
In this episode, Nick sits down with Malo Bourgon, CEO of the Machine Intelligence Research Institute, to discuss why today’s chatbots are only a stepping stone, why leading AI companies are racing toward systems that could automate AI research itself, and why “letting it rip” may not be an option.
Malo explains why the real concern is not that AI becomes “evil,” but that highly capable systems may pursue goals that are not aligned with human flourishing. He also breaks down why AI systems are more like something we grow than something we program, why alignment remains unsolved, what it means when AI systems resist shutdown or hide behavior during testing, and why MIRI is working on international agreements to prevent a race to uncontrolled superintelligence.
This is not an anti-AI conversation. Malo is an aggressive user of AI agents in his own work. The question is whether we can capture the benefits of AI while avoiding the catastrophic risks of racing too quickly toward systems we do not yet understand how to steer.
Topics include AGI, superintelligence, AI agents, alignment, AI safety, MIRI’s proposed treaty, the future of work, national security, AI governance, and what everyday people can do if they are concerned about where this is headed.
Learn more about Machine Intelligence Research Institute (MIRI): MIRI: https://intelligence.org MIRI Governance: https://techgov.intelligence.org If Anyone Builds It, Everyone Dies: https://ifanyonebuildsit.com
🔗 Support This Podcast by Checking Out Our Sponsors:
👉 Build your own AI Agent with Zapier for FREE: https://bit.ly/4nGriLd
👉 Test Prep Gurus: https://www.prepgurus.com Instagram: @TestPrepGurus
Connect with The Nick Standlea Show: YouTube: @TheNickStandleaShow Podcast Website: https://nickshow.podbean.com/ Apple Podcasts: https://podcasts.apple.com/us/podcast/the-nick-standlea-podcast/id1700331903 Spotify: https://open.spotify.com/show/0YqBBneFsKtQ6Y0ArP5CXJ RSS Feed: https://feed.podbean.com/nickshow/feed.xml
Nick's Socials: Instagram: @nickstandlea X (Twitter): @nickstandlea TikTok: @nickstandleashow Facebook: @nickstandleapodcast
Ask questions, Don't accept the status quo, And be curious.
Subscribe for more conversations on AI, learning, work, meaning, and the future.
Chapters: 00:00 The race to superintelligence 00:49 Malo Bourgon joins the show 01:11 Why people are confused about AI risk 03:11 AGI and the original goal of AI 07:07 Companies are racing to superintelligence 09:17 Dario Amodei’s 25% catastrophe concern 10:05 Why the extinction concern is not just hype 15:45 “You don’t get what you train for” 19:29 AI sycophancy and GPT-4o 23:37 AI systems resisting shutdown 27:04 The accelerationist argument 30:18 Are “doomers” actually the optimists? 31:04 Using AI tools while worrying about AI risk 32:18 Sponsor: Zapier 34:42 Violence, Sam Altman, and why peaceful action matters 36:46 MIRI’s proposed AI treaty 44:37 Buying time to solve alignment 47:33 Sudden capability jumps and Anthropic’s Mythos 52:37 Elon Musk, robotics, and concentrated power 55:18 What happens if AI automates cognitive labor? 1:00:10 Bernie Sanders, policymakers, and AI risk 1:07:46 Sponsor: Finding Your Best 1:08:39 How Malo uses AI in his own work 1:10:38 Claude Code, email, calendar, and work systems 1:14:38 Lessons from Wade Foster and Zapier 1:28:44 Social media, AI, and slow government response 1:30:36 AI as a national security issue 1:32:34 What Malo would say to world leaders 1:40:24 Where to learn more