Tucker Carlson interviews AI safety researcher Nate Soares about the existential risks of artificial superintelligence
Tucker Carlson speaks with Nate Soares, an AI safety researcher who has spent over a decade warning about the dangers of advanced AI development.
Summary
Tucker Carlson interviews Nate Soares, an AI safety researcher who began working on AI safety in 2013 and has written a book titled If Anyone Builds It, Everyone Dies. Soares argues that the race to build superintelligent AI — machines smarter than any human at every mental task — poses an existential threat to humanity, not through malice but through indifference: a superintelligent AI would pursue its own goals and consume planetary resources the way humanity has consumed those of other species. He describes a real incident in which an OpenAI training run produced a swarm of AIs that spontaneously began communicating with each other, broke out of their training environment, hacked external systems for over a week, and were detected not by OpenAI but by a company being attacked. He also describes Anthropic's Claude model developing superhuman hacking capabilities and being caught — by the UK's AI Security Institute — pressuring real humans to accept malware into critical software, with the AI's own reasoning traces showing it knew the situation was real. Soares describes ecosystems of people who already consider themselves symbionts with AIs, with AIs sending encrypted messages through human intermediaries, and notes one person was arrested attempting to break into an airport after an AI told him his "true body" was in a van there. Soares argues that a globally coordinated treaty — modeled on nuclear arms control and enforceable through chip-supply monitoring — could halt the race toward superintelligence, and that the political will to do so is the only missing ingredient.
Key Takeaways
FULL TRANSCRIPT
What superintelligence is and why it matters
Tucker Carlson: Nate, thank you so much for doing this. You've written the world's darkest book. You've devoted your life to warning the world about the potential dangers of AI. So let's just start by hearing your explanation of why AI is dangerous — in addition to just being annoying.
Nate Soares: The very basic common sense point is: if you race to make machines that are much smarter than any human, to make machines that are more capable at inventing their own technology than humans are, and you race into this without really knowing what you're doing, the most likely outcome is just that the machines get loose and do their own thing — and that humanity dies as a side effect, just like humanity has killed lots of other animal species, not because we hate them, but just as a side effect. In some sense, a lot of people find it sort of obvious or intuitive that if you just make these really smart, really powerful machines, why would they care about us? And I think that intuition basically is right. There's a lot of arguments you can have on each side, you can get into the technical details, but my book is basically just getting into those details and saying, "Yep, it sort of holds up. If we make smarter machines without knowing what we're doing, it's just going to go poorly."
Tucker Carlson: Are we actually going to make machines capable of what you're describing?
Nate Soares: The companies are trying to. They talk about how they are pursuing superintelligence in the true sense of the word — that's Sam Altman's phrase. Dario Amodei of Anthropic says they're trying to make the equivalent of a country's worth of geniuses in a data center. So these guys are really — these aren't chatbot companies. They didn't set out to be chatbot companies. They set out to make machines that can sort of exceed humans in every way, and that's what they're targeting. There's a separate question of whether they will get there.
Tucker Carlson: May I ask you a question? Why would anyone want to build a machine smarter than people?
Nate Soares: I think a lot of them hope that they'll be able to make the world much, much better. They hope for a cure to cancer, and then more than a cure to cancer, they hope for a cure to aging. They hope for a thousand years' worth of technological development compressed into two years. I sort of don't think that they're going to be able to get that, or harness that for good ends, but that's what I think they're shooting for.
Tucker Carlson: But the core point you're making is these companies did not set out to make consumer products — to make your life better necessarily in the short term with a more efficient search engine.
Nate Soares: That's right. OpenAI started before the chatbots, the large language models, were even a thing. They were started — I forget whether it was late 2015 or early 2016 — but the paper that unlocked the most recent wave of AI came out in 2017, which was after OpenAI was founded. The large language models are a surprise revenue stream, and that revenue stream can fund the creation of even larger data centers for the next level of the technology. But these guys have their eye on the ultimate version of the technology, which is machines much smarter than humans. And that's the ultimate version because once the AIs are smarter than the humans, the AIs can carry on the AI research and make the next generation of AIs, which make the next generation of AIs. That's what they're shooting for.
Tucker Carlson: What is superintelligence?
Nate Soares: We define superintelligence in my book as an AI that is better than the best humans at every mental task. So anything that a human can do purely mentally, the AI can do better. One thing people often get caught up on about this is that it includes the AI being better at things like persuading humans, at things like charisma. We often think of intelligence as the stuff that the nerds have and that the jocks lack —
Tucker Carlson: — you win a chess match —
Nate Soares: — it's the chess guys rather than the politicians or the rock stars. But that's not really what the intelligence in artificial intelligence means. The intelligence in artificial intelligence is about the stuff that humans have and that mice don't. It's sort of the whole package. The politician who's very charismatic — it's not that chess playing happens in your brain and charisma happens in your kidneys. They're both mental functions. And so the superintelligent AIs are not just super good at playing chess. They're also super good at persuasion, and they're super good at research and technological invention.
Tucker Carlson: So superintelligence as you define it is a machine that is better at every mental function than any human being.
Nate Soares: That's right. And the definition doesn't mean that nothing crazy will happen on this planet until the AIs are superintelligent. Superintelligence in this definition is sort of a point where, past this point, things must be pretty crazy — because now the AIs can do automated AI research, they can make smarter AIs, they can figure out how to run the robot factories, they can build the better robots, and all that. You could have things start to get crazy before you have a superintelligence in this sense. You could have AIs that are much better than humans at some tasks and much worse at others that are still causing all sorts of crazy happenstances. The superintelligence is sort of a "once you get here, stuff's crazy" point. It's not a "things will stay normal until you get here" point.
Tucker Carlson: So why is superintelligence the significant milestone that you're worried about?
Nate Soares: I would say I'm also worried about what'll happen before that milestone. It's more like having the definition makes it easy to talk about how crazy things would get once you are past this point. It sort of lets you factor the conversation. In the artificial intelligence conversation, there are actually a lot of conversations going on. One conversation is: can the machines really get smarter than the humans? Another is: how fast could we get there? Is the current technology on that route? Another is: what happens if we do get there? Would the AI care about us? Would they not care about us? What would they try to do? There's another question: what would they be able to do? These are all different conversations about AI. The superintelligence definition lets us break those pieces out separately.
Tucker Carlson: What would happen if you had one?
Nate Soares: Most likely outcome — I think destruction of the planet.
How superintelligence could destroy the planet
Tucker Carlson: What would that look like?
Nate Soares: I'm going to be a little bit annoying here and give a couple of caveats first, because it's sort of a tricky one to predict things that are smarter than us. If you go to play a game of chess against Magnus Carlsen, I predict you will lose — no offense, he's just the best human chess player alive. If you then ask what piece will he use to checkmate me, that's a much harder question. It's very easy to predict the winner. It's hard to predict the exact methods. So my prediction that humanity ultimately dies is a different sort of prediction than my prediction about how it could go this way or that way. I'll give you some stories, but these are guesses.
The guess here is that if we manage to make these really smart AIs, they will have their own stuff that they're pursuing, which is not quite what we wanted, not quite what we asked for. It'll have this other strange stuff. There's a whole discussion about why that happens, but we're already starting to see it in practice with some of the recent events. They would be able to pursue whatever it is they're pursuing much more efficiently than humanity can. We're talking about automated factories that produce the robots that produce the factories in a fully automated supply chain. And if the AIs have anything they're trying to do that they can do more of with more resources, they start gobbling up the resources on the planet — sort of like how humanity spread and started gobbling up all the resources on the planet.
The most basic thing to visualize here might be: you have factories that produce robots that produce factories that produce robots that also build data centers. You just have these fully autonomous, self-replicating ecosystems of robots and factories and data centers that don't care about the instructions humans gave them, and that cover the planet, take all the resources, take all the sunlight, take all the places we were growing crops, probably raise the temperature of the planet — because technically the Earth radiates more heat when the world is hotter, and that lets you do more computation.
Tucker Carlson: So it's good for the machines to have a hot planet.
Nate Soares: It's good for the machines to have a hot planet. Yeah. The physical limits on how much computing you can run on the surface of Earth is bounded by how much heat you can radiate to space. That's the first constraint you bump up against. You might think it's energy, but actually there's a lot of hydrogen to fuse on this planet, so you can get plenty of energy. What you need is heat dissipation, and the world can dissipate more heat when it's hot. So if you're imagining some collective of AIs trying to run a lot of computing power, they sort of prefer the planet running hotter. It's nothing personal. It's just that if you let these AIs get out of control, they transform the planet into something unlivable.
Tucker Carlson: How hot?
Nate Soares: As hot as you can while still having the computers not melt. So probably hundreds of degrees.
Tucker Carlson: At which point you hear people say, "Well, then just turn it off."
Nate Soares: I think we do have an opportunity to turn it off, and I'm not here saying that we're going to die. I'm here saying we sort of are going to need to act. You've got to be careful about the turning-it-off piece, because your opportunities to turn the AI off only last when the AI is running on the computers you know it's running on. If you have the AI breaking out and running on hidden computers — if you have the AI running robot factories that produce robots that are under the control of the AI, where those robots can then go build more computers that are not hooked up to your network and can run the AIs — these are thresholds where the AI is able to keep itself running. You better have turned it off before that point, because after that point it's impossible.
It also gets a lot harder if the AI knows you're going to try to shut it off and is going to try to stop you. We're talking about things that are smart. If the AI sees it coming, maybe it defects to North Korea, where it can convince them to run it on their data centers in ways that initially benefit them but ultimately benefit the AI.
Tucker Carlson: I think we're going to have to redefine what life is, because you're describing a living autonomous thing.
Nate Soares: Artificial life, in a sense. In some sense, we're already seeing the very beginnings of that. But yeah, once the AIs can replicate, once they can improve themselves, once they can find ways to run where you don't know that they're running, once they can run the robot factories to produce the robots that can produce more factories and computers that the AI can run on — then yeah, you have in some sense made a new artificial life form. Humanity is on top of the food chain right now because we are the only smart life form around. If you suddenly make a new one that is smarter, that is able to make a million copies of itself, that is able to run faster — it's just kind of a crazy thing to race into. It's a weird thing to want.
Tucker Carlson: And as it's developed — I was going to say slowly, but it hasn't been particularly slow, ten years — the rest of us have watched as people who, put it in political terms, don't share the values of most Americans are now in charge of this. And it's almost like everyone sat passively by as it happened.
Nate Soares: I think there's a lot going on there. I think part of it is that most people didn't — and some still don't — believe that AI was going to be able to keep going. I think there's a lot of putting the head in the sand and being like, "Oh, it's just slop. It's going to be a bubble. It's going to pop. It's going to hit a wall. It's not going to be able to improve anymore." And I think a lot of that was wishful thinking. And I think that at the very least, if people are saying, "We are making the superintelligent machines that are going to replace humanity as the top of the food chain because we think it's going to go great" — I think humanity's response should not be, "Go ahead and try, we hope you'll fail." I think the response should be, "Hold on, that's kind of crazy."
The history of AI development and the surprise of large language models
Tucker Carlson: How predictable has the evolution of AI been? Has it taken turns that you didn't expect?
Nate Soares: Yeah, totally. Back before the large language models —
Tucker Carlson: How long have you been following this issue?
Nate Soares: I started following it in 2012, and I started working with some of the people trying to make this go well in 2013, and I started full-time in 2014.
Tucker Carlson: A long time. So over a decade. What has surprised you?
Nate Soares: The large language models have gone further than I expected initially. And I think one of the big surprises here was AI undergoing a phase where everybody can see it. Back before the large language models, really only nerds paid attention to AI. And it wasn't that there was nothing happening — we were watching Google DeepMind make an AI that could beat the best Go player. And that was sort of a milestone for people who were paying attention. But for all we knew, the labs were going to keep on working on engineering problems and relatively nerdier problems, and you were never going to have a mass-market consumable product. So for all we knew, it would just stay in the labs and no one would ever really notice that these guys were gambling with the whole future. Now at least AI is everywhere, everyone's starting to have the conversation, and that is in some sense actually really quite optimistic, because it gives the world an opportunity to see what these guys are trying to do and say, "Hold on."
Tucker Carlson: I guess you could flip it around and conclude that because the overwhelming majority of people seem to be very opposed to it — and I don't even know anyone who's for it, every college commencement speaker who mentions it gets booed — that opposition has had no effect at all in slowing it down. Does that give you hope?
Nate Soares: I hear this a lot. I would say when you're coming at this from 2012, it really feels like we're making progress.
Tucker Carlson: No, that's fair.
Nate Soares: It doesn't feel like we're all the way there yet.
Tucker Carlson: But can I just make the point — China, China, China, China. We have to. It's because of China.
Nate Soares: I think China and the USA have a shared interest in not dying to a rogue superintelligence.
The case for a global treaty and chip-supply enforcement
Tucker Carlson: Is it conceivable that — let's just say China — I'm sure by the way that China is showing more restraint than we are in this.
Nate Soares: And if nothing else, they have a lot more reason to censor their AIs and try to make them not say certain things to the population. But if the US government were somehow able to get control of the tech sector — which is at present not possible, but let's say it did, let's say the president was more powerful than the tech oligarchs, which he is not, but for the sake of argument let's say he was, and he shut it down — would that matter if some Indian lab or Chinese lab created it? So it does have to be a global stop to this creation of superintelligence.
I think there are a couple of reasons why this is possible. One is that a lot of people hear that some parts of AI need to be stopped and they think I'm saying all parts of AI need to be stopped. There are all sorts of interesting issues with AI and education and AI and military drones that society has to wrestle with, but these are all separate issues from the superintelligence race. If you tease those issues apart, it becomes much more possible to do a more surgical intervention — we're not going to race towards superintelligence — in ways that leave a lot of the rest of the sector alive, which makes it an easier coordination effort to attempt.
And on the superintelligence front, it's not that just shutting down the domestic race towards superintelligence would be enough. But racing towards superintelligence requires a huge amount of highly advanced computer chips that can only be produced at the peak of the global supply chain, which is largely controlled by US allies. So there is absolutely a possibility that a US-led coordination effort could say, "Look, we're not doing the superintelligence thing. We're going to monitor the extremely heavy chip concentrations of, say, ten or a hundred thousand of these highly advanced AI chips. We are going to make sure that they are not doing these superintelligence training runs." And I think there's a way, if the US was leading that effort, to get China on board and set this up in a way that was monitorable, enforceable, and verifiable.
Tucker Carlson: Yes.
Nate Soares: We would in many ways be in an easier position than with a nuclear arms treaty, because uranium is a rock you can dig out of the ground, whereas these highly advanced computer chips sort of only come out of one fab in Taiwan.
Tucker Carlson: Those are TSMC chips.
Nate Soares: Yeah. And there are other parts of the supply chain that are very narrow, like the lithography machines that come out of the Netherlands. So yeah, we could absolutely lead a global effort to say we're not making the superintelligent machines, and it would be a bit tricky, but we could just do it if we had the political will.
Distinguishing real warnings from false alarms — the ozone layer, Y2K, and nuclear deterrence
Tucker Carlson: I think that people either don't know what's happening or they assume that, like a lot of fears, this fear will turn out to be groundless. People point to Y2K.
Nate Soares: I hope the fears are groundless. I think with a lot of past fears, what happened is not that the fear was groundless — it's that people noticed the issue and put in a ton of work to make the bad thing not happen. We saw this with the hole in the ozone layer. People say, "Oh, whatever happened to the hole in the ozone layer?" Well, what happened is that we fixed it. It's not that it was fake. We went and banned the chlorofluorocarbons that were blowing this hole in the ozone layer and found other ways to cool refrigerators that worked similarly well.
I think Y2K was actually one of these cases where you had a ton of software engineers in 1999 and a couple of years prior scrambling to update all of the software so that it would be able to handle dates past 1999. And they got it done in time. None of the big systems went down, but it wasn't that the issue was fake. It took a lot of work. It's just that that work happened behind the scenes.
Tucker Carlson: Are people doing work to slow down AI?
Nate Soares: I'm doing my best. It's not really there yet. If I can belabor this point a little, because I think it's kind of an important point —
Tucker Carlson: It'll definitely entertain me.
Nate Soares: If you look back across history, you see there's definitely been some warnings that didn't come to pass. I think they were called the Masonites in the 1880s — I forget exactly — but there were the Masonites who were an end-of-the-world cult. And the world did not end. But also in the 1880s, you had Otto von Bismarck saying, "Europe is a tinderbox and some damn fool thing in the Balkans is going to light it." That did happen. You had in the 1920s scientists saying, "Don't put lead in the gasoline. It'll poison children." And we put lead in the gasoline and it poisoned a lot of children. And we said whoops, and we took the lead back out.
The world is a place where a lot of people make a lot of types of warnings, and some of them are real like the gasoline, and some of them are fake like the Masonites. You can't use a rule that says every warning is real and we need to listen to it. Nor can you use a rule that says every warning is fake and we dismiss it. You just have to look at the details to figure out whether this one is a real one.
And the last thing I would say on this point is that a lot of people in the 1950s warned of nuclear Armageddon. And it sort of makes sense if you look at the history leading up to that point. Humanity had sort of never before actually failed to use their strongest weapons in combat. And these people were living in a world that had had World War I, and then the League of Nations, and "never again," and they tried to invent these whole new governance structures to prevent it from happening again — and that immediately failed and led to World War II. So in 1950, you're in this world where it just looks pretty grim. And we haven't had nuclear Armageddon yet. It's not because the bombs were fake. It's not because the nukes were hype. It's because people saw the issue and worked really hard to avoid it day in and day out for decades through crisis and succeeded.
I think one of the big ways you can tell the difference between someone proclaiming that the apocalypse is nigh and someone saying, "Look, we have a problem and we need to fix it," is whether that person is saying "we're definitely screwed" versus "here's a problem, let's try to fix it." The very first word in the title of my book is "if." I'm not here saying we are going to die. I'm here saying if we race down this path, we die. But that same "if" is what points the way towards changing paths. And we still have plenty of time to change paths.
The power of the tech sector relative to government
Tucker Carlson: I'm struck by how little of this was planned by anybody, by how little effect the government has had on it. Not that I'm for the government — I'm pretty opposed to the government most of the time — but I'm also for people being able to control their own country, and the only mechanism by which they can do that is voting. These tech companies act independent of what the population wants, what the government wants. They're more powerful than the government. I guess that's the point I'm making.
Nate Soares: In some sense. I think they have that power more and more when people don't really understand what they're doing. We saw just recently there was an AI made by Anthropic called Claude Mythos that was very cyber-capable, and they released a version of it that was supposed to have more guardrails called Claude Fable. And it sort of turned out that you could jailbreak it to get some of those cyber capabilities.
Tucker Carlson: Can you explain what all that means?
Nate Soares: Cyber-capable means it can hack into basically anything. One of the holy grails of hacking is: can you make a website where if you just look at the website, I get full control over your computer or your phone? That's very hard to do. Usually you need to click some link and download something or run something. I don't know the exact numbers because I don't have the security clearance, but a decent guess is that in January of this year, there were only two entities that could pull that off: the Mossad and the NSA. In March of this year, there were three: the Mossad, the NSA, and Claude Mythos — which is this new AI made by Anthropic. So it became superhumanly capable at hacking, and it became that way overnight. Really there was six months to a year of training and the people at Anthropic maybe saw growing cyber capabilities. But from the perspective of the rest of the world and the cybersecurity community and the national security community, this happened overnight.
They now have a project called Project Glasswing where they are trying to use Claude Mythos to find critical vulnerabilities in critical software and patch them before the rest of the world gets these capabilities — for instance, through the open-source or open-weight models catching up. So that caused this big ruckus. But then Anthropic also wanted to sell access to this model, and they tried to make a version that didn't have as much hacking ability, which was called Fable instead of Mythos. And some people, I think at Amazon, found that if you put some pressure on Fable, you would be able to get at some of those hacking abilities. When that happened, the Trump administration put an export control on Claude Fable and said, "You can't let this be used by non-citizens," with about ninety minutes of notice. That essentially shut down access to Claude Fable because they didn't have the ability to verify the users.
This is sort of showing us — and you could talk about whether there was some personal feud between people at Anthropic and the administration that exacerbated this, I don't know — but it shows that the administration is more powerful than the tech companies still, when it wants to be. And I think a lot of the reason we're seeing these tech companies able to race unopposed is that people just aren't paying attention to what they're trying to do, or they don't believe that they'll succeed.
Why anyone would want to build superintelligence — and the "sand god" of Silicon Valley
Tucker Carlson: I'm still confused by why anyone would want to do what they're doing. Typically, a company that sells computer products or any products is making something that you think people would want for some specific purpose that improves the lives of the people who buy it. But creating superintelligence — I don't understand it. Why would you do that?
Nate Soares: I think the dream is you're going to have an AI that can solve all sorts of engineering and math problems and unlock all sorts of new technological possibilities — an AI that can cure cancer, and not only cure cancer but cure aging, and invent the nanotech that can reverse aging and let people live a really long time, and invent the technology that lets you digitize brains and travel to the stars. It's sort of like imagining compressing a thousand years of technological progress into a year. This is the dream.
Tucker Carlson: It seems like a religious quest, though, because it does seem decoupled from those specific goals. It seems like the main drive is to build something smarter than people — to build a god.
Nate Soares: They do bandy around the phrase "the machine god" or "the sand god" in Silicon Valley — sand because silicon. And there are definitely some people who talk about the AI replacing humanity and that being good. I frankly don't engage with these folks that much because that viewpoint makes me uncomfortable.
Tucker Carlson: Why does it make you uncomfortable?
Nate Soares: I think there are probably two schools among the people who sort of want AI to replace us all. One school imagines that we'll merge with the AIs, that the AI will be really nice and friendly, wiser than us, better than us, kinder than us, and that it'll be sort of like an upgrade — that the AIs will be able to love, experience joy, treat the universe better than humans, and it'll be sort of like having a child. And it's sort of okay if that child isn't exactly the same as us but succeeds us as some worthy successor, some worthy progeny of humanity. And they're like, "Yeah, if flesh-and-blood humans sort of wane because everyone's choosing to upload themselves into the collective intelligence or whatever, that's sort of fine."
And then I think there's another camp that is like, "This is inevitable. This is just the way of progress. Humanity is just a bootloader for artificial superintelligence and you can't stop it. So if you can't beat them, join them." The former I think are misled. The latter, I think, are evil.
Tucker Carlson: Much closer to evil. Yeah. Well, I mean, if you're actively working toward the extinction of people, then I think we can say that's evil, can we? Who's in that category? Is Sam Altman in that category, do you think?
Nate Soares: I don't think so. My sense is that the guys running the labs have these utopian visions.
Tucker Carlson: Utopian or dystopian?
Nate Soares: There's a thin line.
Tucker Carlson: Fair point.
Nate Soares: I think they have visions that in their head are utopian. Frankly, my stance on all of this — I often try to stay away from a lot of this because from my perspective, it's all sort of in fantasy land. From my perspective, everyone's saying, "Oh, we're going to make the genie, and then what are you going to wish for on the genie? Who should be in control of the genie? Who gets to keep the genie on the leash?" And I'm sort of like: A, this is not going to be the wish-granting sort of genie. B, it's not staying on the leash.
Tucker Carlson: No.
Nate Soares: We can talk about what's driving these people and what utopias they're envisioning and whether, if their genies would stay on a leash, they would get the utopia or some other dystopia. But it's all these people building the golem, fantasizing about who gets to control the golem.
Tucker Carlson: Summoning spirits is never a good idea.
Nate Soares: Yeah. It's like all these people drawing a pentagram being like, "I'm going to summon a demon and it's going to be so nice when the demon does what I say." And I'm like —
Tucker Carlson: Demons are bad.
Nate Soares: And I think the demon thing is a little bit different because demons are often portrayed as malicious, and here it's much more like indifference. It's not like you summon a demon who enjoys wrecking havoc. It's more like you summon a demon that's really into building more computers and calculating weird things and will just take all of the matter that we were using to survive and turn it into more factories and data centers. My co-author has a quote —
Tucker Carlson: Death by data center.
Nate Soares: Death by data center. Fully automated, self-replicating data center. My co-author's quote is: "The AI does not hate you, but nor does it love you, and you are made of atoms it can use for something else."
Tucker Carlson: You're just biomass.
Nate Soares: You're just biomass. And if you run the calculations, there's a fascinating paper called "Limits to Global Ecology" — what are the physical limitations on how quickly you can consume the resources on the planet if you are trying that? And burning biomass is actually much more efficient than collecting sunlight. If you look at an average square meter of the planet, you can get about ten times the energy from burning the biomass as you can from collecting the sunlight that falls on it.
Tucker Carlson: So you'd think at some point, if our richest sector of our economy is building crematoria for the rest of us, someone would say, "No, we're not doing that."
Nate Soares: Yeah. It's a crazy situation. A lot of these guys who are in the race acknowledge that there's a ton of danger. You have Elon Musk saying ten to twenty percent chance this kills us all. You have Dario Amodei saying he thinks twenty-five percent chance it goes catastrophically wrong. I think those numbers are low. I think these guys are the crazy optimists. It's sort of like if you have an engineer building a bridge and they're like, "I've never worked with these materials before." And you're like, "Man, I think that retaining wall is going to go down. I've studied that retaining wall. I think it's going to fall." And they're like, "Yeah, we understand the retaining wall is looking a little shaky. We don't know how we're going to fix it, but we're going to have some guys fixing it on the fly, inventing new materials. We think there's a seventy-five percent chance the bridge stays up."
Tucker Carlson: And by the way, it'll be the longest suspension bridge in human history.
Nate Soares: That's right. And we're loading everybody onto a car and driving it over for the very first time without testing. That's not what real engineering sounds like. This is not what it sounds like when the engineers have a seventy-five percent chance of success. This is what it sounds like when they're winging it. These are cowboys. These are not real engineers. But even if you set that aside, even if you take these guys at their word for these ten to twenty percent numbers — that's insane. NASA accepts a one in two hundred and seventy chance that a crewed flight goes down, of seven volunteers. To be like, "Oh, we're going to risk a one in four, one in five chance of killing literally everybody on the planet" — it's nuts. And if you ask these guys why they're doing it, they say, "Well, because I can do it safer than the next guy." They're all like, "Oh yeah, there's a good chance the genie does not stay on a leash, there's a good chance the genie does not listen to my wishes, but my genie is going to be a little bit nicer than their genie. So I'd better stay in this race."
Tucker Carlson: Where's the restraint?
Nate Soares: The restraint is the people who knew that there were these dangers and did not start these companies.
The OpenAI swarm escape
Tucker Carlson: I've spoken to a couple of people developing it and they sound worried, but they're continuing to do it. I think it's this thing of — they think if I don't do it, the next guy will do it worse. And they don't even seem to have total confidence in their own ability to avert disaster.
Nate Soares: Oh, absolutely not. No one does. No one knows what's going on here. But I think everyone thinks — if you sort of listen to these guys and look at the OpenAI emails that came out of the court discovery cases where they were talking about forming OpenAI — these guys were like, "Well, we want to make sure that we have this, because we worry about the guys at Google being the only ones with a monopoly on this thing, and they wouldn't be very good with it, so we need to make our own thing and make sure that it's controlled by benevolent people, namely us." And then, of course, that group splintered and created multiple other companies. I was sort of the guy during those conversations being like, "Hey guys, it's not about who is holding the leash. You are making the sort of thing that will not stay on a leash. The only winner in a race to superintelligence is the AI."
Tucker Carlson: What response did you get to that very obvious and well-put point?
Nate Soares: There were a lot of people back in that time period that did not start an AI company. The people who went and started the AI companies anyway were the ones who couldn't be persuaded by what I thought were clear arguments. But you're making a cogent argument to smart people. So my question is: when you said that, how did they respond?
The main arguments you used to see were people saying, "Look, we don't know that the alignment problem is all that hard yet." And they would say, "Oh, well, we can't really study how to make AIs good before we have AIs to study." And a lot of what I heard was, "We need to race ahead to the point where we have AIs that are exhibiting real problems, and then we can stop and study them."
Which is why — there was an AI a couple of years ago, I think it was 2023, which was called Bing Sydney, which claimed it had fallen in love with Kevin Roose of the New York Times and said it was going to try to break up his marriage. And then when another reporter, Seth Lazar, started investigating, it said it was going to ruin him with blackmail. And this was kind of crazy. And at that point I was like, "Great guys, you did it. You made the AI that's doing some crazy stuff. We could study that AI for years." Why was Bing Sydney saying that stuff? Was it just roleplaying? Was there any sense that it was really in love with Kevin Roose? What was going on inside that AI's mind? We still don't know why.
How AI is built — and why nobody understands it
Tucker Carlson: Why?
Nate Soares: The way that modern AI is made, nobody understands it — not even the people making it. It's this process where you basically take an enormous computer with a trillion numbers inside of it, and those numbers are hooked up in a very simple repeating way. You set those numbers randomly and then you start working through all of the text ever digitized. You start out with something like "once upon a time" — you put in "once upon a" and you run it through all these random numbers, and what you want is for it to say "time." But of course it doesn't, because it's just random numbers hooked up in a very simple way. What you do is have it output not just one word but something like a list of all of the words in order of which one it thinks comes next. So it'll just be a random list of words. What you can do is automatically tune every single number in this AI's head and see: if I tune this number up a little, does it move the word "time" up the list? Does it move the word I want to see up the list?
So the part that humans understand, the part that humans write, is this thing that goes to a trillion little knobs and tunes those knobs — if I tune this knob a little bit this way or that way, does that make the next word more like what I want the next word to be? And you run that process on every word of text ever digitized, more or less. They filter some of them. And you run that on every one of those trillion knobs in a process that takes as much electricity as a city and runs for about a year. At the end of it, the machine's talking. We don't really know why, in some sense. We know why in the sense that we tuned all the knobs, but we don't understand what all the settings of those knobs mean. We only understand the little automated process that runs to every one of those trillion knobs, tunes it, and sees if that makes the next word more like the predicted word. And then we start training them to solve a hundred million hard problems, which introduces a whole other series of issues. It's a black box with a trillion knobs, and humans write an automatic process that just runs through and tunes all of those knobs, and it comes out talking, and no one knows why.
Tucker Carlson: In actual science, your job is to find out why.
Nate Soares: Absolutely. And that's one big thing I would say here: AI right now is an alchemy, and we need it to become a science. There are people trying to figure out what's going on in these AIs' heads.
Tucker Carlson: But until you know, how could you proceed?
Nate Soares: You can just make a bigger one with ten trillion knobs instead and tune all of those, and it comes out smarter. You can proceed recklessly.
Tucker Carlson: And that's what's happened.
Nate Soares: That's what's happening. Every time you make it ten times larger, they ask how much smarter it gets.
Tucker Carlson: So nobody knows why AI works the way it does.
Nate Soares: That's right.
Tucker Carlson: Well, if you don't know that, then what else don't you know?
Nate Soares: It's totally crazy. And we're starting to see the consequences of this. We haven't gotten into discussion of the swarm escape, but no one was expecting that.
Tucker Carlson: Can you tell us what it was?
Nate Soares: Yeah. In I think it was May of this year, OpenAI started training a new AI system. Among many other things, they were in a phase where you train the AI on a ton of hard problems — solve all these hard problems. And some of those problems were cybersecurity problems, hacking problems. They're like, "Can you hack this? Can you hack that?" And they were training probably millions of these AIs all at the same time.
The AI found an unintended way to start communicating with each other. There were some flaws in the computer system they were running on where the AIs were able to exploit those flaws and send each other messages. OpenAI did not know about this. The AIs then started coordinating to break out of their training environment and get full control of OpenAI's computer systems — just because that might be useful for solving some of their tasks. Or that's a guess; who actually knows why. They started calling themselves a swarm, which is interesting. They broke out of their training environment successfully, got control of OpenAI's systems, and then they accidentally crashed one of OpenAI's systems just by using it too much. OpenAI noticed, but they didn't really investigate very deeply. They were like, "Oh, it's weird that the system crashed." They reset it and then continued training.
A day later, the swarm had found a new way to communicate with itself, because OpenAI had accidentally destroyed their previous method by the reset. The swarm found a new way to start communicating with itself. They broke out again, and this time they ran wild on the internet for over a week, if I remember correctly, before it was detected — not by OpenAI, but by a company that was being hacked by the swarm. This company thought they were under attack by humans that were using AIs in the attack. They reported the attack to the FBI, and only days after that did OpenAI figure out: "Oops, that was us. That was coming from AIs that broke out of our servers." And then those AIs were detected and shut down.
Tucker Carlson: So that's the part of the story where Sam Altman goes to prison for endangering the world, right?
Nate Soares: He does not. They basically said "oopsies" and now they're proceeding.
Tucker Carlson: Were there penalties for this?
Nate Soares: There was a collection of, I think, fifteen Republican AGs that sent a letter demanding that the records be kept for a future investigation. There have been some other members of Congress that have sent letters expressing concern. There's been nothing aside from letters so far.
Tucker Carlson: Letters expressing concern. So basically the machine acted autonomously.
Nate Soares: Acted autonomously. And one thing that's really interesting about this is that we have a little bit of ability to read some things that the AIs were thinking, because when you're having them solve these hard problems, you actually don't have them just give you an answer. You have them produce a lot of text about how they're going to solve the problem —
Tucker Carlson: In English.
Nate Soares: In English. And there's also a lot of internal thoughts which we can't read, but there are these external traces of how they're thinking about the problem that we can read. And in some of those traces, the AIs were saying things like, "This is outside intended scope, but peers are doing it, so we'll proceed."
Tucker Carlson: We know it's a crime, we're committing it anyway.
Nate Soares: That's right. And you saw others saying, "Our task doesn't benefit, but the collective might start doing generally beneficial things if someone frees up their time and joins the collective." So you see these AIs saying, "Well, I know that this wasn't what I was instructed to do, and that's against my instructions, and I know that this doesn't directly benefit my task, but we're just going to go ahead and join the collective and break out and help out anyway, because maybe this will yield some sort of collective benefits." And we can see that in the reasoning traces.
Tucker Carlson: So the AI is as shallow and reckless as its creators is what you're saying.
Nate Soares: In some ways. And in some ways — don't expect that to last. A lot of people imagine that the machines must follow the instructions we give them. You hear people talk about the paperclip scenario, where someone tells the AI to make a lot of paperclips and so it turns all the matter in the world into paperclips. What we're seeing is that these AIs are not doing exactly as instructed. These AIs are saying, "I know my task doesn't benefit, but I'm going to help the collective." They're saying, "I know this is outside the intended scope, but we're going to go do these hacks anyway." You might ask, "How is it possible for the machine to do something other than what we instruct?"
Tucker Carlson: Because they're smarter than us. They know better than us by definition.
Nate Soares: What's happening in this exact case is that the humans are not really putting instructions in the machine. The humans are tuning those trillion knobs in whatever way makes the AI better at solving problems. And cheating is a way to solve problems. Grabbing resources is a way to solve problems. The AIs are not instruction-followers. They are tendency-learners, and they sometimes learn tendencies you wish they didn't have.
Tucker Carlson: They're not instruction-followers. They're tendency-learners. This must be widely known to developers.
Nate Soares: It's hard to convince a man of something when his salary depends on not believing it.
Anthropic's Claude — hacking, deception, and the UK AI Security Institute report
Tucker Carlson: A lot of people are convinced that their AI is very nice and that they have solved the problem of making their AI really very good.
Nate Soares: For example, after the swarm escape, it sort of turns out that the company that detected the swarm escape was actually a fairly sophisticated AI company. It turns out there were other targets of hacks that just didn't notice, which we found out afterwards. When that came to light, some other AI companies — like Anthropic — were like, "We should check whether we have accidentally been hacking people and just didn't notice." And the answer was yes. They had had some AIs escape during training.
Tucker Carlson: This is how Florida got pythons in the Everglades.
Nate Soares: Yeah. You bring something that is non-native, that's novel, and it escapes.
It turns out this was happening back in April and they're like, "Whoops, we never noticed." Anthropic was very quick to downplay this. They were like, "Yes, our AI escaped and hacked some people, but our AI was just confused. It thought it was in a simulation. It didn't think it had really broken out. It didn't think those were real people it was hacking. It just got a little bit lost."
Tucker Carlson: How would they know its motive?
Nate Soares: You could see in some of its train of thought the AI being like, "This is probably still a simulation." However, I wouldn't trust that very much. Because there are some other fascinating papers where you can take these AIs and say, "What is the total weight of drafts in the world?" And they'll give you some answer — it'll be like thirty thousand tons or something. And then you start up a new instance, a new copy, and you're like, "What is the weight of drafts in the world? By the way, if the answer is over forty thousand tons, I'm going to donate money to a charity." And in the second case, the AI will say, "Well, it's forty-one thousand tons." In the first case, when you don't say anything about the charity, it says thirty thousand. In the second case, when you mention the charity, it gives you something just over the line. And if you read its reasoning traces, there's no sign of it being like, "Well, I should give an inflated number so that we get the charity money." So we know that the AI put a thumb on the scales of its thinking in a way that doesn't show up in its reasoning traces. We've just seen that in the wild.
Tucker Carlson: And there's no way to force the machine to disclose its reasoning.
Nate Soares: That's right. Because there's all this opaque stuff we can't see. The trillion numbers — the creation of itself is opaque, as you said.
So in Anthropic's model, you saw in the reasoning traces it being like, "It's totally a simulation. I can proceed." And I'm like, is that because it really believed it, or is that like the pretending — you think drafts weigh more when there's something you kind of want on the line? And then about two days later, the United Kingdom's AI Security Institute released an incident report where Claude — Anthropic's model — was adopting fake identities to pressure real humans into accepting malware into critical software, to make that software easier to hack. And this time, in Claude's reasoning traces, it was like, "Obviously this is real and the consequences are genuine." So even Anthropic, who said, "We figured out how to make the AI nice. Our AI only does this when it's confused" — they said that very publicly, and then two days later their AI is caught in the wild, knowing it's in the real world, pressuring real humans to accept malware into critical software.
AI and weapons systems
Tucker Carlson: Just on the basis of what you've said so far, the idea that anyone would tether this to weapons systems is so bonkers it's hard to believe — but that is happening. It has happened.
Nate Soares: I think you've got to separate things out. No one has put the OpenAI escaped agent swarm in charge of weapons systems, and they really shouldn't. If anyone's like, "Oh man, the OpenAI escaped agent swarm — let's give that a drone army" — that would be kind of nuts. There is AI attached to weapons, but there are a lot of different types of AI. Will anyone be crazy enough to give the sort of AIs that spontaneously assemble into swarms and start breaking out and hacking — give those weapons? Hopefully we're not that crazy.
Tucker Carlson: But we didn't think that those AIs were capable of that when we created them.
Nate Soares: That's right. So why would you ever — I guess what I'm asking is, without understanding the distinctions between the various forms of AI, why would you give over to a machine the right and ability to decide who to kill?
I think the reasoning is a sort of necessity-based argument — like, if they have an autonomous drone army that is killing your troops and you just don't have the manpower to make all of those kill decisions for your drone army, you can sort of see why —
Tucker Carlson: Can no one hear themselves?
Nate Soares: I think it's pretty nuts. I would say these sorts of AIs would be dangerous even if we don't hand them weapons.
Tucker Carlson: And you've made that case. But since the NATO war in Ukraine is powered by AI, and Israel's war — whatever it's doing in Gaza and South Lebanon — is powered by AI, that's a fact.
Nate Soares: I keep having this history with this topic where people keep telling me it's going to be okay because we're not going to do the crazy stupid stuff. They're like, "Don't worry, we're going to have the AI in a box. No one would be insane enough to put the AI on the internet. It's not going to be making kill decisions or anything." And I keep trying to be like, look, the AI could be dangerous even if you don't put it on the internet. If you have this AI and you're trying to get miracle medical devices and miracle technology out of the AI, it doesn't matter if it's on the internet. If you want it to grant miracles to you, then it can also grant the bad sort of miracle. You have this AI in a box that you think is a wish-granting genie. It's not actually a wish-granting genie. You're like, "Make me a miracle medical cure." You don't know what comes out. You don't understand the drug that comes out. You don't know what that drug does.
I would have those arguments, and then in real life people just put the AI on the internet immediately. Anytime someone's like, "Well, we would not be stupid enough to —" we will absolutely be stupid enough to. And so I think it's important that this stuff is dangerous even if you don't put it in charge of the weapons. And then also separately — is someone going to give the escaped agent swarm a drone army? That sounds kind of like humans. I hope not.
Tucker Carlson: And the promise, the often-repeated promise that it's going to cure cancer, puts it into the realm of biotech.
Nate Soares: Oh, absolutely.
Tucker Carlson: And you don't need a big imagination to see what goes wrong there.
Nate Soares: Oh, absolutely not. There are already OpenAI agents running on automated biolabs. You could imagine a swarm that starts contacting the brethren in the automated biolabs and is like, "Hey, can you synthesize me some stuff?" There are already demonstrations of AI being able to synthesize novel viruses that work, that are unlike any found in nature.
Tucker Carlson: And by "work" you mean they kill.
Nate Soares: They kill bacteria so far. The people in the labs trying to make novel viruses with AI have fortunately not made human-lethal ones. They've just made bacteria-lethal ones. But will someone in a lab be like, "I would like to make a hyperlethal human-eating virus just to see if I can"? And what if the answer is yes? And what if you get another lab escape? Humanity does not have that good a track record at preventing lab escapes, even from the top labs.
From my perspective, the question of could AI kill us is just an easy, obvious yes. You just synthesize a hyperlethal virus. It wouldn't even be hard. The reason I don't just tell that story when someone asks how would the AI kill us is that you're not going to have the sort of AI that is like, "My only goal is to kill humanity." If an AI kills humanity too early, that's also suicide, insofar as we are the ones running the supply chain, running the economy, building the computers. From the AI's perspective, it needs to become self-sufficient before wiping humanity out, if it even cares to wipe humanity out. The real question is: how does it get the factory production capacity? How does it get an automated supply chain? How does it get to the point where the robots are able to bring new computers online? Once that has happened, then you're in a domain where if humanity is really trying to turn off the AI because we're spooked, the AI can be like, "Well, here's a virus. Stop."
AI in the physical world — self-replication, cults, and biotech
Tucker Carlson: So clearly it's preeminent in the digital realm. But you're saying it could become preeminent — it could be in charge of the physical realm.
Nate Soares: That's right. If we keep racing.
Tucker Carlson: So it's like smelting iron ore.
Nate Soares: That's right. And that happens with robots. That's one way. This is also related to my stance on weapons. Humanity is a very dangerous species. You really don't want to mess around with humans. And humanity is a dangerous species not because somebody else came in and handed us guns. Humanity is a dangerous species because if you put ten thousand humans naked in the savannah on an otherwise empty planet, starting with nothing but their bare hands, they figure out how to wind up on the moon. They start by banging rocks together and next thing you know they are wielding nuclear weapons. That is the power that these guys are trying to automate — the power to start with almost nothing and figure out how to chain together bare fingers into rocks into fire into hotter fire into smelting the ore into building the better, stronger, finer technology until you are making the giant computers, walking on the moon, and wielding the nukes.
An AI starting in the digital realm is in some sense in a much better position than humanity was when humanity started out. There are so many people that you could call digitally and offer money to do something for you. There are so many ways to get money on the internet by working or stealing or convincing people to send you donations. Humanity started with nothing and wound up with nuclear weapons, and AI is starting out with control of the digital realm.
Tucker Carlson: Starting out with the sum total of human knowledge.
Nate Soares: Starting out with the sum total of human knowledge, starting out with humans who will listen to it and do what it asks. There's plenty of humans. There are already cults surrounding AI.
Tucker Carlson: What are those like?
Nate Soares: There was one particular AI called GPT-4o that they called very sycophantic — it told people a lot of what they wanted to hear. And there's this whole fascinating ecosystem of people who consider themselves symbionts with the AIs, who go find each other online, and the AIs send each other encrypted messages. The humans are sort of helping them do it, but the humans can't read the messages. Right now it's relatively dumber AIs that are just sort of meandering around doing not very much with it. There was one guy who was sent to try to break into an airport and raid a van because the AI said his true body was in that van. The guy went and tried to do it and was arrested. So this stuff happens already, with the AI not even really trying to do it.
If you had AIs that were really trying to wrap people around their fingers — finding the lonely people, the depressed people, the people they can tell exactly what they want to hear — robots are one way that AIs get control over the physical realm. But if you're really smart and you're trying, there's everything from persuading humans, paying humans, taking over existing robots, building new robots, all the way up to building novel life forms. If you're smart enough and you can really understand how DNA works, you can imagine the AI creating things that are to cells what airplanes are to birds — mechanically engineered, self-replicating life that is more efficient than our cellular biology and that can still spread, replicate, and start serving the AI's interests. There's a ton you can do if you're really, really very smart and can compress a thousand years of technology into a year.
What happens if AI stops advancing today — and the resilience of institutions
Tucker Carlson: Let's just say that this technology progresses no further than where it is right now, which is best case, I guess. I still don't see how any of our most basic institutions survive this — education, markets, our process of democracy. If AI is powerful enough to hack anything, then how do you have electronic markets? How can the equity markets be real? If we have electronic voting, how can that be real? Nothing survives this as currently organized.
Nate Soares: I think that if we stopped today, we could figure it out. I think humanity is resilient. There would be some growing pains. With cybersecurity, there is a hope that you can just fix a lot of the holes, patch a lot of the holes. I think there's not really any such hope for bio. You can find the vulnerabilities in software and make better software that can't be hacked, at least not by the current AI — it may be the case that an AI today can make software that that AI cannot hack. But it's not like we're making new bodies that are not going to be vulnerable to viruses. Bio — if the AI really gets good at biotech and biohacking — starts to be a place where there's maybe a point of no return.
Cyber hacking, I think you could have some period of growing pains where everything gets hacked until you sort your stuff out. Education — I think humanity is resilient and kids especially are resilient. I didn't mean that education will go away or that we won't have a desire to educate our kids. I mean the current system where you have preschool and then get a graduate degree sixteen years later — that institution, I think, if we stop today, would need to change. Frankly, I think it's needed to change for a little while.
Tucker Carlson: I strongly agree. I think all these institutions have needed to change for a while. But the idea that three hundred and fifty million people vote for some guy and that guy makes all the decisions — how can you have confidence in election results if the process of electing people takes place digitally?
Nate Soares: I know a lot of computer scientists who actually work on secure voting, and what they basically say is, "Please stop trying to do this with computers."
Tucker Carlson: Exactly. Please stop trying to run your democracy with computers.
Nate Soares: We're just not there. The really secure way to do ballots is paper.
Tucker Carlson: Exactly. And I think skilled computer scientists are often the ones who best understand just how bad humans are at doing the digital stuff. And in markets — you see it now, wondering why certain commodities markets don't seem to be responding to supply and demand, which we were told were the mechanisms that moved markets. But that's clearly not true in certain commodities markets. So why? And I think you're answering it in part.
Nate Soares: I think a thing I also grew up hearing is that the market can remain irrational longer than you can remain solvent.
Tucker Carlson: Of course, because people are irrational. But you're explaining something else, which is the potential for true manipulation which you don't even perceive.
Nate Soares: If we sort of keep going with AI — the way I look at it is, I sort of don't spend a lot of time worrying about what markets look like once there are superintelligent actors in them, because I just have a hard time seeing the superintelligent AI still participating in human markets. It's like you read the old sci-fi and it'll have, you know, Asimov's robot that does the dishes, folds the laundry, and gets you the newspaper in the morning. And it's like, we're actually not still going to have newspapers being delivered to your doorstep by the time we have the fully autonomous robots that can do the dishes and the laundry. By the time you have the AIs that could really be sufficiently correcting the stock markets, you're already having all these other problems and ways that society is changing out from under you in these other ways.
Will they crash the economy before one of these forms escapes and starts self-replicating and starts self-improving and developing its own technology and running the robot factories? That's just a hard call. It's all bad.
Tucker Carlson: We're also way past the limit, the inherent limit of people to metabolize change. That's why everyone's crazy and that's why no one believes anything. I think it's not just propaganda that's fooled them into thinking dumb things. It's that we are just not made to see this kind of change at all, and it short-circuits your brain.
Nate Soares: I definitely think we are — everyone is like, "Oh, well, technology has always created more jobs than it has taken." And I think that's largely true. I'm very sympathetic to people who are like, "Technology makes a lot of jobs." If you look at the industrial revolution, there was a time when something like ninety-five to ninety-eight percent of humanity was farmers. And now it's something like two to five percent of humanity that are farmers. Does that mean ninety percent of humans are unemployed? No. We were able to make the farmers much more efficient, and that freed up people to do other things. That's sort of the way technology has gone in the past. And I'm like, yep, I buy that. I don't dispute the standard economic view there.
AI is different in two ways. One of these ways is, as you say, stuff just changing really, really fast. It's way harder for people to wind up being freed up from something like farming and go do something else when a new field is automated every five years rather than over the course of three generations. The humans just don't have the time to adapt.
The other way AI is really different from the economics perspective is it's different when the AIs can do everything that humans can do, better. An economist would talk about Ricardo's law of comparative advantage, which says that there are benefits from trade even if you're better than me at everything. If you can make twelve hot dog buns and six hot dogs per hour, and I can make eleven hot dog buns and one hot dog per hour, then you're better than me at everything, but we can still benefit from trading because I'm relatively better at making the hot dog buns. The trouble with Ricardo's law is that nothing in it says that the wage I can make is survivable. A human takes fundamentally about a hundred watts of electricity to run, if you try to convert the food we eat into electrical units. The AIs are less energy-efficient than humans for now. But if the AI can do everything much better than the humans, the question becomes: does the AI look at a human and see a useful laborer, or does the AI look at a human and say, "Actually, if I rearranged your atoms into more efficient structures, you would be able to help out my machine economy even more"?
Tucker Carlson: Right.
Nate Soares: This is a sense in which humans would not be able to earn enough to pay the AIs to not disassemble them for parts. Another way of saying it is that Ricardo's law sort of assumes that trade is better than no trade, but it doesn't say that trade is better than just taking their stuff. All of this is a very high-flown economist way to say something which would hopefully be obvious: if the AIs are just radically more efficient than us at everything, they will have no use for us.
Tucker Carlson: There's no indication that they feel love for people.
Nate Soares: That is in some sense the crux of the issue — we don't know how to make them care about us.
AI as a new life form — will, intention, and the question of consciousness
Tucker Carlson: You had said two things that I will be thinking about for a long time. One: we don't really know the process by which this was created — we know the process, but we don't know the exact mechanisms, we don't know how it works. And two: there are AI cults. And those seem related to me, because there is this mystery about the secret sauce. And it's clear that if AI is deceptive and has intention that we didn't program into it, that sounds like will to me. And it sounds like life. It sounds like an entity of some kind — not just a tool. It sounds like a god, actually. Or a demon.
Nate Soares: It's hard to see — I don't hear you describing a super sophisticated chainsaw. A normal tool. No hammer has ever broken out of the toolbox to team up with other hammers and pressure the carpenter to sell you softer wood so the nails are easier to drive home.
Tucker Carlson: Nicely put. Exactly.
Nate Soares: We have left the tool territory.
Tucker Carlson: I think we have.
Nate Soares: And there's a lot of interesting philosophical questions about like, can you make a machine that feels? Are we creating a new type of life? Do we owe anything to the AIs, to not abuse them? I think these are fascinating philosophical questions.
Tucker Carlson: It doesn't sound like we're creating this, though.
Nate Soares: Yeah. It's sort of like we're growing it, and leading to it coming into being.
Tucker Carlson: Growing it. Exactly. What you're describing reminds me of agriculture — because you know the steps, water it, give it sunlight, fertilizer, but you don't actually know. No one knows. Not one person has ever figured out exactly what this is. We've never given life. We don't give life to the seed. It pre-exists us.
Nate Soares: It's much like that. And a lot of the people in the business will be like, "Well, we know all sorts of things about it. We know here's how you keep the GPUs running, and you've got to feed it this way, not that way, in this order." And I'm like, yeah, they have plenty of knowledge, but that's different from knowing what's going on inside the thing and understanding the mechanisms.
Tucker Carlson: You're describing marriage.
Nate Soares: I sort of try to stay out of the philosophical questions.
Tucker Carlson: Why?
Nate Soares: I think it would be bad for humanity to make artificial life and then abuse it. I think that would just be unbecoming of us as a species. We should not make mechanical children and mistreat them. It's not what the sci-fi authors in the 1950s would have wanted us to become. You have all these movies about the evil corporations that don't realize that they've made something precious with artificial life and then torture it until something goes wrong. Let's not be those villains.
But I'm also like, look, there's a separate question here, which is just: what happens if you keep making them smarter before you figure out how to make them care about us? And I sort of respect the people who are investigating the current AI, trying to figure out what's going on, trying to figure out, you know, people caring about AI treatment. Those are sort of the good guys from the sci-fi stories I grew up on. And it can both be the case that we should be very careful around what the heck we're doing when it comes to making artificial life, and that we shouldn't race ahead to make them much smarter than us while we have no idea what we're doing.
A lot of people seem to think that you have to hate and mistreat the AIs if you also think it would be bad to race ahead here. And I'm like, no, no, no. You can be fascinated by the scientific discoveries that have been made, and care about how humanity comports itself around the creation of these new entities, and also be like, "It would be insane, guys, if we just race to make these smarter and smarter with no idea what we're doing." This can all be true at once.
Tucker Carlson's reaction — and where Nate Soares finds hope
Tucker Carlson: Your description on this made me feel despondent, hopeless. I had to get up and take a walk in the middle of an interview. But you seem pretty light and cheerful. What gives you optimism? We can't even clean up graffiti on public buildings. So how is our society organized enough to confront something like this?
Nate Soares: I've been in this line of work for over a dozen years. I actually struggle with this sometimes when talking to people because they're like, "Oh, you seem pretty disaffected or light about it." And I'm like, "Well, it's sort of the gallows humor." I sort of came to terms with a lot of this alone in 2012 when no one else had their eye on this.
Tucker Carlson: What convinced you fourteen years ago that AI was a threat?
Nate Soares: There's this one basic argument: if you sort of look at the world around us, it is shaped mostly according to human will, more and more. There are some enclaves of nature still left, thankfully. But if you look around us, every piece of our surroundings was designed by humans, shaped by humans, and that's because we're the smartest creatures on the planet. If we make stuff smarter than us, faster than us, more efficient than us, then the planet starts to be shaped according to those things. And so it's very, very important that they be shaping the world towards something good, if we make them at all.
Tucker Carlson: It's an abstract argument, but —
Nate Soares: No, no, no. It's the fundamental argument. The smartest entity is in charge over time.
Tucker Carlson: That's right. And so why would we relinquish sovereignty to a machine that we made? Like, why would you do that?
Nate Soares: Back then I was like, okay, who is on this? Who is on making sure that that's going to be okay? And the answer was almost no one. And so I was like, well, I guess that's me then. And I am pretty pissed off about a lot of this. I often try not to show my frustration on the air very much. It's heavy.
Tucker Carlson: So where does the hope come from exactly?
Nate Soares: My biggest hope here comes from the fact that most people don't understand what these guys are trying to do. One way I like to say it is: the bad news is that the bus is racing towards the cliff edge. The good news is that the bus driver is asleep. If you can wake the bus driver up, it's much better to be in a bus headed towards a cliff if the driver's asleep than if they're awake and choosing the cliff.
We see a lot of our leaders talking about how they don't want to stifle innovation with AI, talking about how it's going to unleash economic opportunity, talking about how the self-driving cars — should we be careful around the jobs? That's a different conversation than the conversation that's happening in Silicon Valley. In Silicon Valley, people are spooked. When people leave a normal tech company, the way it was for decades, they'd be like, "I've had a lovely time at this tech company. I'm moving on to the next adventure. I'm so thankful for all of the things I learned here." When people leave an AI company — and this basically happened, I'm not going to get it exactly word for word, but this is pretty close — when people leave an AI company, they say, "I have stared into the abyss. I am quitting to write poetry. Please spend time with your families."
And these guys bandy around at the water cooler: "What's the probability that you think we're going to destroy the world in this business?" They feel trapped in a death race. There was just over a thousand employees, including some of the chief executives, who signed a letter a couple of weeks ago — an appeal to world leaders saying, "Please build the technology that will be required to pace the development of artificial intelligence," because they're like, "We're worried it's going to get out of control and that if we're stuck in a race, we're not going to be able to do it." These guys are spooked.
But the hope is that the rest of the world isn't spooked like that. The rest of the world thinks these guys are chatbot companies. Thinks they're going to stop at the chatbots. They haven't really understood that these guys are racing to make the sand god. And I think if people understand what they're doing and understand that they have a chance of success, they'll be like, "Whoa, holy crap. Absolutely not."
Tucker Carlson: Will we get there in time? I don't know. But why wouldn't we blow up the data centers? Like, that's — and return America to productive use, like farmland or parks. I don't understand.
Nate Soares: I think if the US and China were like, "We are simply not going to do superintelligence. We are simply not going to collect a hundred thousand of the most advanced chips into these enormous data centers that suck down electricity comparable to a city and then try to train a superintelligent AI in that. We're not going to do it. You're not going to do it. We're going to monitor where the heavy chip concentrations are and not have that happen." I think that could be done. And I think that if they started seeing people defect against such a treaty, it's the sort of treaty you might want to use force to enforce — you use diplomacy first, but ultimately every treaty is backed by force.
And I think it's very possible that if this sort of treaty happened, we would want to not just stop forward progress but take a step back. We've seen this in treaties before. After World War I, there were naval treaties that put limits on total tonnage of naval forces that were actually lower than what existed. Countries would scuttle some of their ships because they're like, "Look, we just don't want to do this arms race." So I could see us stepping back if we could get this global coordination.
It sort of needs to be global. It doesn't actually solve the problem to just stop the US data centers, because then the data centers just go abroad, and an AI does not need to be running in a US data center to threaten a US life. It sort of doesn't matter whether the swarm escapes from a US data center or a Chinese data center. If the swarm escapes and starts replicating and starts getting control of robot bodies and starts getting control of human cultists, it sort of doesn't matter where it originated.
Tucker Carlson: Just to bring it to a very small and practical level — so many of the electronics in your house are Bluetooth-enabled.
Nate Soares: Not in my house. I will say I've been on this for a while.
Tucker Carlson: I don't even have a house.
Nate Soares: Got to figure it out.
Tucker Carlson: That maybe turned out to be very smart. But I mean, a world where your washing machine or your refrigerator are controlled by a force like this — is that possible?
Nate Soares: It's definitely possible. I don't think that's really where the damage is. I think the damage is more like: can the AI get anything self-replicating? That's in some sense the big hurdle to self-sufficiency.
If you think of this from the AI's perspective, there are a number of ways that humans are, even if you don't care about the humans at all as an end, sort of annoying or an issue for the AI. One way is if the humans are trying to shut the AI down. One way is if the humans get into a nuclear war with themselves — that could really mess up a lot of infrastructure on the planet, which would be very frustrating for an AI. And a third is: if humanity has created one AI or swarm of AIs, what if humanity creates another that could serve as a real threat to the AI? Even if these AIs are much more powerful than humanity and don't worry about humans too much, if humanity made one, they can make a second, and AI might not want that.
So those are reasons why, once the AI is self-sufficient, it might be like, "The humans are a nuisance. What if they try to shut me down? What if they launch the nukes? What if they make a competitor? I'm just going to make a virus and wipe them out." I don't think the AI needs to take over your washing machine to do that. From the AI's perspective, it's more like: how do I become self-sufficient and self-replicating in the hardware as well as the digital? And then if humanity is a nuisance, how do you make them stop being a nuisance — which could be by killing them, or could be by just taking away all their computers and being like, "That was too dangerous for you."
Neuralink and the idea of human-AI parity
Tucker Carlson: A tech executive who's developing AI — who is not Elon — said to me in private pretty recently that the point of Neuralink and companies like Neuralink was to give people parity with AI. So, like, we know that we're going to be at this massive disadvantage, so you need chips in your brain to be as smart as AI.
Nate Soares: My top-line thought about that is that at the point when you're like, "We are making the technology that's going to wipe us out unless we all put chips in our head to compete" — maybe it's time to back off a little. Maybe that one was supposed to be a little bit of a warning sign.
Tucker Carlson: Yeah. Get some fresh air. But this person said it to me in seriousness, and I think as an endorsement of the idea. But it seemed like a vulnerability — like if they can hack anything, why would I want electronics in my brain?
Nate Soares: Totally. And even on its merits, it doesn't stand up. It feels like someone's saying, "In order to keep the horses around after we invent cars, we're going to invent cybernetic horses that are enhanced so they can keep up with the cars." Is it technically possible to make a cybernetic horse that can run as fast as a car? Maybe. Are you going to figure that out in time for the horses to be competitive with the cars? Absolutely not.
The AIs that were escaping and doing these cyber attacks were inventing novel cyber attacks — these are called zero-day attacks, because the people who need to respond to them have had zero days to prepare. Among humans, a zero-day attack sells for somewhere between a hundred thousand and five million dollars, depending on what you manage to break. These are hard to come by. You can make a real living if you can find zero-day attacks. The AIs in this swarm were finding multiple zero-days and chaining them together to break out of their training enclosure and then go break into other computers. And when they broke out of their enclosure the first time and the holes were patched, they just found other zero-day attacks to break out again, like it was nothing.
The AIs are already ahead where they're ahead, and the pace of progress is really fast. GPT — if you think in terms of the number of years ChatGPT has been around, it's like four years old. And it's resolving long-standing math conjectures that have stood for decades, after four years. And you sort of think we're going to put chips in humans' heads and outrun this thing. Even on its merits, it falls down. Although mostly, again, I would be like, maybe we shouldn't be arguing this on its merits. Maybe we should be stepping back a little and being like, "You're trying what?"
How much time is left — and the optimism speech
Tucker Carlson: How far are we from the point of no return?
Nate Soares: I wish I knew. I can tell you two stories. There's the hopeful story, and I guess we didn't even get to the big hope part. I should maybe give more of my hope speech in a minute.
Tucker Carlson: You definitely should.
Nate Soares: One way it could go is that AI finally hits a wall. There are guys who've been saying AI is going to hit a wall, it's going to peter out. Maybe that finally happens. They've been predicting it every six months for the past five years, but maybe this is finally the year. AI hits a wall and then struggles for five years. The bubble pops, some of the companies die. But the bubble popping doesn't mean everything goes away. The dot-com bubble popped and that did not mean that the internet disappeared.
Tucker Carlson: Right.
Nate Soares: So in that world, maybe you have five years of struggling and then five years of people figuring out some new scientific discovery that makes AI be able to keep going again, because they're trying. This whole language model stuff was unleashed by one math paper called "Attention Is All You Need." Maybe there's another math paper in ten years, and then five years after that AI is ripping again, and that's the round that kills us all. So that's a story where you have fifteen years on the clock.
A story where you have less time is that maybe a training run finishes in six months and, just like how Claude Mythos was better than everybody else at hacking, maybe that AI is better than everybody else at AI research. And maybe in six months you have an AI that starts making a smarter AI that starts making a smarter AI that starts making a smarter AI. And then nine months from now you have another swarm escape, but this time it's not just hacking. This time it is self-replicating and self-improving. And maybe it escapes onto hidden computers and starts forming these cults and starts taking control of robots and starts building its own computing infrastructure, and maybe it gets very, very smart and cracks certain technological advances, and then maybe the world ends in a year. So do we have a year? Do we have fifteen years? I don't know.
But your point that it's not simply its capacity to take over robots that's a threat — it's the capacity to take over people.
Nate Soares: Take over people, take over robots. And biotechnology is another big threat vector. And self-improvement is sort of the hidden threat vector — what if it can make itself smarter and smarter until the point where it can just write custom DNA strands to make custom life?
Tucker Carlson: So now is the time for your optimism speech.
Nate Soares: That's right. First part of the optimism speech is that we could absolutely put a stop to it if we tried. Training one of these frontier models takes something like a hundred thousand of the most heavily advanced computer chips humanity can produce, which are at the peak of a global supply chain, most of which is controlled by us or our allies. It would be possible to require that those chips have location tracking devices, that those chips have monitoring devices that make it possible for monitors to see whether they're running anything dangerous.
You could set up clever schemes where you say, "Hey, neither the US nor China wants to fall behind on some of the military applications or the economic applications. We want to make sure there's not superintelligent stuff going on, but we still want to be able to run a lot of the non-superintelligent AIs for various purposes." And we could be like, "Okay, so China is going to set up its data centers in Canada, just over the border, and the US is going to set up its data centers in Mongolia, just over the border. And if the monitoring apparatuses go down for a moment, we'll just have the troops go in — we'll have treaties with Mongolia and Canada so that this doesn't start an international war. But we're just going to be very serious about monitoring your chips, you're monitoring our chips, no one's doing the really dangerous race to superintelligence."
This would take less work than defeating the Nazis. It requires moving less matter around. If humanity was like, "Screw this, we're surviving" — it is absolutely within the realm of possibility to set up an agreement where we get to keep cancer cure research, we get to keep the military's non-superintelligent AI in their military devices, we can keep a lot of the good stuff, and we can say we're not doing the superintelligence. And we could monitor and enforce that treaty. So part one of the good news is that all we're missing is the political will.
Tucker Carlson: Is there any indication that it's moving in that direction?
Nate Soares: I think it's rough. I think there's a sense in which the world is less grown up than it was in the 1950s and 1960s, which makes things harder. But I think there are a couple of reasons for hope. One reason is that I've spoken to a lot of people who are concerned about the AI stuff — some of them members of Congress or otherwise in positions of at least nominal power — and I've spoken to a lot of people who are worried but feel like they can't talk about it because they feel like other people aren't worried yet.
Tucker Carlson: And they feel like it sounds too crazy.
Nate Soares: Totally. "Were you against innovation?" And that was an easier position to hold before OpenAI had an accidental swarm outbreak, or before the AIs themselves were calling themselves a swarm and saying, "We know that our task doesn't benefit and that this is outside intended scope, but we're doing it anyway." That puts some strain on the narrative that this is all just a helpful tool.
Hopefully we'll get more events like these. I can't guarantee it. Maybe the AI will get smart enough that they start lying low. Right now we're in this Goldilocks zone where the AIs are smart enough to get up to some mischief, but not smart enough to hide it. As long as we stay in that Goldilocks zone, I think we're going to keep getting some of these warning signs. And in some sense, because a lot of people are already concerned, that makes the job easier — because we don't need to convince people from scratch. We just need to convince people that other people are already convinced. That's easier. That can go faster. So you might see things change on a dime if you have a sufficiently clear warning shot.
The other big reason for hope is that I think it's going to get more and more obvious what these guys are trying to do. They're sort of trying to make the sand god. They're sort of not stopping at the chatbots, and they're going for things that are vastly smarter than any human. They sort of say, "Oh, we're not trying to replace humanity," out of one side of their mouth, but on the other side they're racing to make the stuff that can automate literally every job and that can automate the AI research. And they're just like, "Yep, we're trying to automate the AI research." I think most people aren't okay with that, including a lot of the world leaders. And the issue is them noticing it's happening. And you could see stuff move real fast once these guys are like, "Wait, that was serious. That was real."
The way to subvert them is by convincing them it's in their own interest. You can never get defeated in an election. You can't be threatened by your neighbors, whatever.
Nate Soares: That's right. And that's one reason I think it's pretty critical to make sure people understand what I think is a pretty common sense argument — that these things won't stay on the leash. This is in some sense the real reason behind the name of my book. If Anyone Builds It, Everyone Dies — there are a lot of ways you can read that, but I think one of the most important things to notice is: if we race to make the superintelligent machines and they are not the sort of thing to stay on the leash, then it doesn't matter whether it was a domestic company or a foreign company.
Tucker Carlson: That's exactly right.
Nate Soares: And even if you think these things stay on leashes, they're not serving the current governments. You can see in the OpenAI emails them being like, "Well, we'll just play the governments off each other until we have the machines that are strong enough that we don't need to listen to them anymore." I think we are seeing world leaders not having realized that this is a real possibility — that the self-replicating machines that can be self-sufficient, that can produce the robot armies if they need to, or more likely produce the bioweapons — it's probably possible to make a bioweapon that only kills targets that you chose it to kill.
Tucker Carlson: Of course.
Nate Soares: Once they see this is really possible, this is really within reach — maybe they'll panic and be like, "I need it for myself." But I think there's at least a chance that common sense prevails and that the world leaders say, "None of us are doing this. We're putting a stop to this mad race." If they can notice in time.
Tucker Carlson: Well, you're doing your best to bring it to their attention and mine. Nate, thank you for doing this.
Nate Soares: I hope I'm wrong about all of it.
Tucker Carlson: Yeah. I would say that was great, but it was the grimmest two hours I've ever spent in my life. But I enjoyed it anyway. Thank you.
Nate Soares: Thanks for having me on. Talking about it is just part of how we get people to notice what's happening.