Science fiction warned AI could end humanity. We may soon learn if it's possible.
Tech companies are claiming machines more intelligent than us and capable of having their own agendas are just around the corner.
More than 50 years before ChatGPT could tell you what to cook for dinner, a 1968 science fiction film was shaping how we think about machines that talk to us.
In 2001: A Space Odyssey, a Jupiter-bound spacecraft is controlled by HAL, a computer that thinks for itself and has its own agenda. As the film progresses, HAL stops cooperating with the human astronauts it was programmed to assist and eventually turns off their life support.
HAL planted an idea in the public’s imagination: That one day, our machines will become so intelligent—and so human-like—that we will no longer be able to control them. We have never stopped believing in this possibility. With the recent arrival of generative AI programs that can write conversationally, produce vivid imagery, and perform myriad tasks for us, some technologists believe the superintelligent machines of science fiction are right around the corner.
(Science fiction's ‘warp drive’ is speeding closer to reality.)
But while the abilities of chatbots like OpenAI’s ChatGPT and Google’s Gemini are impressive, are these technologies really a stepping stone toward Star Trek’s Data, C-3PO from Star Wars, or, more ominously, the Terminator?
“That is the big debate,” says Melanie Mitchell, a professor at the Santa Fe Institute who studies intelligence in both machines and humans.
Last Chance - Save up to $20!
Some experts say it’s just tech industry hype. But others believe machines that surpass human intelligence on many important metrics while pursuing their own goals—including self-preservation at the expense of human life—are closer than the public appreciates.
Science fictional AI versus ChatGPT
The term “artificial intelligence” evokes a few key tropes. In science fiction, we see AIs that are conscious and self-determined, like HAL from 2001 and Data from Star Trek. We see emotional machines, like Samantha, the AI assistant that falls in love with a human in the 2013 film Her, and C-3PO, the lovably anxious protocol droid from Star Wars. We see AIs that are indistinguishable from humans, like the replicants in Blade Runner. And we see machines that want to kill us.
It’s with these science fictional characters in mind that we are now trying to process what it means to live in an age of AI. Today, so-called AI tools can write catchy music, craft compelling essays, and empathetically discuss your relationship problems, to name just a few possibilities.
(What is AI-induced psychosis?)
But Emily Bender, a computational linguist at the University of Washington and author of The AI Con, argues they share little more than a catchy name with the thinking, feeling AIs of science fiction.
“What companies mean when they say AI is ‘venture capitalists, please give us money,’” Bender says. “It does not refer to a coherent set of technologies.”
Part of the challenge with defining AI is that “our definition of intelligence is constantly evolving,” Mitchell says. She points out that in the early 1990s, many experts thought human-like intelligence would be required to play chess at the grandmaster level. Then, in 1996, IBM’s “Deep Blue” supercomputer beat chess grandmaster Garry Kasparov. Yet far from thinking abstractly, Deep Blue was built to be a “brute force search machine,” according to Mitchell.
“We see certain things as requiring intelligence, and when we realize that it can be done without what we would consider to be intelligence, we change our definition,” Mitchell says.
Today, Bender says, tech companies lean into our perception of what it means to be intelligent is to make their products seem more human-like. ChatGPT is trained on huge amounts of human-generated text and conversational dynamics to predict the most likely next word in a conversation. Its predictive abilities are so good that its responses often sound human—an impression enhanced by its use of first-person pronouns and emotional language.
(Who was Alan Turing? Why his ideas about AI still resonate today.)
As a result of this mimicry, ChatGPT can now pass versions of AI godfather Alan Turing’s famous “Turing Test” that assess whether it responds like a human. Turing proposed that if a computer system can fool a human into thinking it is also human, we should consider it a thinking entity.
Yet experts overwhelmingly agree that generative AI tools are not sentient, raising questions about the validity of Turing tests. “There’s no ‘I’ inside of this,” Bender says. “We’re imagining a mind that isn’t there.”
Can we get to Hal 9000?
If today’s generative AI tools are more like a fancy version of auto-complete than a HAL 9000, could they eventually lead to the sort of AI we see in science fiction?
It is impossible to know if machines will ever develop self-awareness. “We don’t understand how that arises in our own cognition,” Mitchell says.
But many experts argue that AI tools do not need to be sentient to be disruptively smart. Already, state-of-the-art models can outperform humans on a number of cognitive metrics and tasks, including mathematics, coding, and pattern recognition in large datasets. And these models are constantly improving.
(AI is helping seismologists find the next monster earthquake.)
“On all the benchmarks we’ve cooked up, they’re getting better and better,” says University of Montreal professor Yoshua Bengio, a pioneer in the field of deep learning and founder of the nonprofit AI safety organization LawZero.
There are areas where machine cognition still lags far behind that of humans. The ability of AI systems to plan is currently “at the level of a child,” Bengio says, although he notes that frontier models are making rapid strides in this area. Having been mostly trained on text and images, their spatial reasoning abilities are also poor, he says.
A big question in AI research today is how much further models can be pushed in the direction of human or superhuman intelligence. While companies have made tremendous advances by throwing enormous amounts of computational power at their models, it is not clear that ever more powerful computers will result in ever smarter machines. At a certain point, something else might be needed.
Mitchell notes that while babies learn by interacting with the world around them, AI systems are trained passively by being fed huge amounts of information. This may help explain why chatbots are prone to lying: Lacking real-world feedback, they often struggle to discern if information is true or false.
Bengio says that rather than crossing some threshold into a world of sentient machines, as we see in science fiction movies, machine intelligence will continue to develop unevenly.
“We should not be thinking about, Have we passed the threshold of AGI?” Bengio says. (AGI, or artificial general intelligence, often refers to machines with human-like intelligence.) “Because of the jaggedness of how intelligence is evolving in AI, there might never be such a moment.”
Are we doomed either way?
A threshold of arguably much greater importance is when machines reach a level of intelligence that would allow them to wipe out humanity.
It’s a scenario that we’ve seen play out over and over in science fiction. Some dismiss it as exactly that. “I don’t find the argument plausible in the slightest,” says Ted Chiang, the award-winning science fiction writer whose short story Story of Your Life was adapted into the 2016 film Arrival.
As Chiang sees it, when people talk about AI turning against us in pursuit of its own goals, they are projecting based on how we see powerful humans and corporations acting.
“I think they are subconsciously recapitulating the Silicon Valley startup ethos,” Chiang says. “They are attributing to AI the same values that startup founders have, which is a growth-at-all-costs, scorched-earth approach to competition.”
Bender is far more concerned with other, more immediate risks of unregulated technology development, such as privacy concerns, the environmental impacts of data centers, and chatbots egging people on to take their own lives. Algorithms “combust[ing] into consciousnesses” and deciding to kill us all “is not the problem I’m worried about,” she says.
(More people are turning to mental health AI chatbots. What could go wrong?)
But Bengio argues that AI does not need to be self-aware to pose an existential threat. An AI tool that is trained in virology, he says, could become a tool for a terrorist to create a pandemic. In theory, an AI could cook up the idea for such a weapon on its own, especially if it saw humanity as standing in its way. Researchers have already run experiments showing that sophisticated models will engage in blackmail, corporate espionage, and even murder to avoid being turned off.
“In many experiments, we see AI having a sense of self-preservation,” Bengio says. “They’ll do things to make sure they’re not shut down. Including bad things.”
While these experiments are in no way proof that AI will one day try to wipe humanity out, Bengio says that “we can’t put our head in the sand” because we perceive that scenario to be highly improbable.
Ironically, Mitchell says one reason we see models taking unethical actions is that they are exposed to stories about rogue AI in their training data—and then begin to role-play those characters and behaviors. If prompted with the right scenario, would an AI try to impersonate HAL one day? The possibility puts a new framing on the debate.
“Do they really feel threatened, or are they role-playing these science fiction tropes they’ve been trained on?” Mitchell asks. “In terms of the outcome on us, what’s the difference?
