Here’s why the AI doomsday scenario is flawed and harmful

All you ever wanted to ask about AI taking over the world. 

Spoiler: The role of consciousness is fundamental, but it is not the only thing. 

What is the AI doomsday scenario?

A dystopian narrative of a future where humanity is threatened and even extinct by AI.

How would that happen?

In a worst-case scenario, AI will at some point become advanced enough to improve itself. Its development then accelerates and becomes exponential, and shortly thereafter it achieves an intelligence far exceeding that of humans. And since intelligence is the most powerful attribute we know, it becomes extremely threatening if AI were to set goals that conflict with the survival of humanity – much like an anthill that lies in the way of a road construction. End of story.

Who spreads this scenario?

A number of well-educated and knowledgeable individuals, some with expertise in the field of AI. 

Why?

Probably because they believe in it, or because it serves their purposes.

Is this scenario new?

No, it has been voiced by mathematicians, computer scientists, researchers, and writers for decades, but with limited reach until now.

So why this attention in media now?

Because since the introduction of ChatGPT, everyone talks about AI. Many people are also impressed by AI, even though they have a vague idea of what it really is. 

Do people believe in this scenario?

It seems so. A recent survey by Reuters/Ipsos showed that 61 percent of Americans believe the swift growth of AI technology could put the future of humanity at risk[1].

Do other experts believe in it?

Well, the scenario is often bolstered by a survey highlighting researchers’ beliefs that there is at least a 10 percent probability of severe consequences stemming from AI over time. But this survey was directed towards 4,271 researchers, with only a 17 percent response rate. Among those 17 percent, not all answered the question about the consequences of AI over time, but about half of those who did answer expressed at least 10 percent chance of an extremely bad outcome[2]. 

Meanwhile, many highly qualified AI experts do not believe in the doomsday scenario at all.

How do you calculate this risk being 10 percent?

You cannot. It is a guess. 

So why should this risk be important?

Well, if there’s really a minimal risk for extinction of humanity, the argument goes that we need to put all our resources at preventing that it happens, even at the cost of dedicating less efforts to other issues such as climate change for example. 

So, why is the doomsday scenario flawed when it seems so simple and convincing?

Because it is oversimplified and lacks nuance. It rests on a series of assumptions that are far from self-evident.

Like for example?

We need to ask fundamental questions. What do we truly mean by intelligence? What has consciousness to do with it? What leads to the escalation of intelligence? How do moral and ethical systems evolve? And finally, how is today’s AI technology constructed? These issues are also interconnected.

Hey, wait a minute! One thing at a time! So first, what is intelligence?

Yes, what exactly is intelligence? Is it that simple that we are more intelligent than a dog that is more intelligent than an ant? And that an AI that is more intelligent than humans will be overpowering us?

The definition of intelligence usually includes, among other things, the ability to reason, plan, solve problems, associate, think abstractly, understand ideas and language, complicated causal relationships and the ability to learn.

Two of these are particularly interesting – the words “understand” and “causal relationships.” 

Ok, so would you say that ChatGPT is intelligent?

Well, quite clearly, ChatGPT with this description can be considered intelligent in some respects. It excels in various academic exams and performs tasks at a human-level proficiency but much faster. Combined with other systems and memory functions, it can even solve entire problems on its own by breaking them down into smaller subproblems and working its way to a solution step by step. An example of this is the AI agent Auto-GPT. But the question is what ChatGPT actually accomplishes.

And what does it accomplish?

For those who know something about the principle behind ChatGPT, so-called neural networks based on linear algebra, it is clear that what ChatGPT does is to imitate human behavior based on training on very large amounts of data.

But how can such a simple principle make ChatGPT perform so well?

At first glance, it may seem strange that imitation can reach such advanced levels. ChatGPT not only reproduces facts – although it still makes fatal errors – but can also reason about those facts in a way that humans do.

But if you think in terms of the mathematics behind the AI, it’s not so strange. No matter how we twist and turn the richness of human communication, the number of variations in our expression and our reasoning is limited in practice. It is very large, yet limited.

At the same time, it is difficult for us to understand the enormous amounts of data on which the AI is trained on, and the unparalleled computational capacity that computers have. But if a computer is trained on human texts and human reasoning, and identifies every imaginable pattern in the smallest detail and constantly increases the amount of training data, then the computer eventually reaches a point where the richness of detail is large enough to be comparable to the vast spectrum of human reasoning encountered in practice.

It crosses a kind of threshold and can suddenly mimic human writing even in long texts. And even translate between languages, although it has not been trained to do so.

So why do they call AI a black box?

Because we don’t know how it works in detail, because it’s unmanageable. But we know the basic principle – statistics and correlation.

And why does it get things wrong sometimes?

Surprisingly wrong actually. Because that is what happens when the basic principle is statistics and correlation, and not “understanding”. And now we come to consciousness.

Ok, so what about consciousness?

Few argue that these mathematically based language models are conscious, and there are good reasons.

Our own consciousness is one of the greatest mysteries in science. We all perceive that we have a consciousness, but no one can prove that someone else has a consciousness. No one knows how it works, let alone how to reproduce it.

Since you’re so used to your consciousness, you may not consider it, but it essentially gives us two abilities. One is incredibly rich subjective experiences – from colors and flavors to feelings, moods, and bodily sensations. The other is an experience of a self over time.

The first ability allows us to experience the world and ourselves so that we can learn on our own from our experiences. The second ability, a self over time, enables us to have thoughts about cause and consequence, and thus about ethics and morality. It also gives us the ability to form relationships and to have an intention, and much more.

Today’s AI has none of this.

Give me an example.

If I say “we strung up a rubber sheet between the trees and when we jumped on it, it acted like a…” you will probably think “trampoline”.

So would ChatGPT. But ChatGPT does it based on statistical probability. You do it based on the understanding of how a stretched rubber sheet behaves when you jump on it – cause and consequence – which is possible because you have an experience of time.

And what makes that difference so important?

Because this is a reasonable explanation for why ChatGPT makes strange errors sometimes, despite being trained on unimaginably much more data than a human will ever come close to. It also explains why it is so difficult to build self-driving cars because they learn to drive according to the same principles – imitation. To be safe, they need to be able to imitate human driving in every imaginable situation, which requires a huge amount of training, and yet you don’t quite get there (although in many situations self-driving cars have better overview and faster reactions than humans).

A human, on the other hand, who has an understanding of cause and consequence, and a deep understanding of how the world behaves, learns to drive in twenty driving lessons, and can handle any unexpected situation in a reasonable way. 

But ChatGPT can solve some problems that seem to be about cause and consequence. 

Yes, but taking into account how the model works, built on statistics and correlation, everything suggests that it is again an imitation of human reasoning, and nothing else. ChatGPT could also be integrated with a model of physical reality with its physical laws and conditions, and thus be more able to predict certain events. But without the experience of time, this is not an expectation of a certain consequence, but a calculation without perspective.

Ok, but if ChatGPT is intelligent without consciousness, what would the opposite be?

Many animals can be said to be somewhat the other way around – to some extent conscious but perhaps with a less developed intelligence than humans, or at least different.

Recent research on consciousness clearly points to the fact that many animals have consciousness, for example because they recognize themselves in a mirror and because they solve problems in a way that requires a perception of time and causality.

So, where would you place humans on this scale?

Well, it seems to be the combination of intelligence and consciousness that is the strength of humans.

Our consciousness plays a key role in the difference between us and current AI, and it is, for example, very difficult to imagine how it would be possible for an AI to define a future goal and to also have an intention to achieve that goal without a consciousness and a perception of time.

Possibly, models like ChatGPT could imitate human goal-setting and define some reasonable but semi-random goals, and then work towards achieving the goals, as we are starting to see in the case of AutoGPT. But it is very difficult to imagine that the AI would have any intention to do so.

Ok, and without intention, how could it become dangerous?

Good question. Regardless of how intelligent such an AI becomes, it is difficult to imagine it becoming a threat to humanity. Without a goal and an intention of its own, it becomes at best troublesome and generally dangerous to society, perhaps even very dangerous to society, but not an existential threat to humanity.

What do you mean by very dangerous?

Well, you could say that the danger with AI is not that it is smart, but that it is stupid. Since it doesn’t have intention and no understanding of cause and effect, it simply works towards the goals you give it. Therefore, you need to be very careful when choosing its goals, since it is not like a horse for example, which will hesitate if it encounters an dangerous choice on the road, or some other risky consequence of going ahead. An AI in contrast, will blindly go towards the goal in the most efficient way, so you need to think thoroughly about all possible consequences of that. 

Then there are lots of well-known challenges with AI. It may be used for spreading disinformation, and for manipulating information, producing fake images and videos. It could be used to create and spread very advanced malware. And it could be used in generally bad ways with lack of transparency and responsibility. But these are issues we discuss all the time and can find solutions to. 

And this idea that it would self-improve beyond human intelligence?

That’s another good question. Animal intelligence has evolved over time, driven by the advantages it offers for survival. But an AI is not exposed to evolution and natural selection. And unless we actually give it the goal to use its self-improving capability in order to increase its intelligence, there is no obvious reason for it to do so. 

Also, if it would set such a goal for itself, again there is the question how an AI without consciousness and a conception of time can have the intention to reach such a goal. There is simply no clear mechanism that would explain why a self-improving non-conscious AI would strive to enhance its own intelligence.

Ok. But what if an AI develops consciousness?

As I said, we don’t know how consciousness works, but there is a lot of research going on and there are interesting hypotheses. Since we see that many animals have consciousness, we can conclude that consciousness has developed gradually through evolution.

An hypothesis that explains this, put forward by researchers such as Derek Denton and Antonio Damasio, is that consciousness originates from the need of organisms of a certain dimension and complexity to keep the body’s functions in balance even when the environment changes, in order to survive – a phenomenon known as homeostasis. One way to solve this is to develop the primordial instinctive feelings of fear, hunger, thirst, and pleasure. By becoming instinctively aware of these feelings, the organism can act to maintain homeostasis. Over time, more complex feelings may have developed, and then also subjective experiences of our senses and finally even thoughts.

The hypothesis also has some support in the researchers’ experimental results.

According to this viewpoint, consciousness does not arise as a byproduct of intellectual capacity while also acquiring the ability to experience emotions. Instead, it has its origin in emotions.

But some people say consciousness could emerge from complexity? 

Yes, some debaters argue that since we do not know how consciousness works or how it arises, it can essentially arise everywhere where there’s enough complexity, for example in language models such as ChatGPT. But with that approach, consciousness might as well arise in a company’s ERP system.

In contrast, the hypotheses about consciousness as a result of the need for homeostasis strengthen the argument, on solid grounds, that language models such as ChatGPT will never attain consciousness. In an interview in ScienceNews, Damasio instead suggests that in order to become conscious, a robot would have to become vulnerable and also have some form of internal regulation that needs to be in balance[3].

Consciousness thus seems to be something completely different… a whole different animal.

And it seems that Arthur C. Clarke had intuitively understood this when he portrayed the AI HAL 9000 in “Space Odyssey” as vulnerable in its fear of being shut down.

In essence, the doomsday prophecy of an AI becoming self-improving, surpassing human intelligence, and pursuing its own goals is built upon a series of loose assumptions that render it fundamentally flawed.

Ok, but if we could one day find out how to reproduce consciousness?

Yes, let’s say that we actually manage to create a consciousness further down the line, using other methods. It is not entirely unlikely.

This raises the question of how moral and ethical systems arise. Here, the thesis in the doomsday prophecy is that a super-intelligent AI can set goals such as transforming all matter in the universe into paper clips, and then exterminate humanity along the way.

Wow, that sounds scary!

Well, apart from the fact that it may seem absurd that such an advanced intelligence could find such a goal interesting, this reasoning has no connection to what morality and ethics are, and how they are being formed.

An interesting, elaborate and convincing hypothesis presented by researcher Oliver Scott Curry and colleagues, is that moral systems arise as a mathematical consequence of game theory between interdependent entities. The hypothesis is based on the fact that there is a limited set of behaviors that lead to “win-win” situations in game theory, and combinations of these behaviors turn out to cover all known moral and ethical rules, plus some that intuitively feel perfectly conceivable[4].

On this hypothesis, a conscious AI will not develop any random morality of its own, provided it is somehow dependent on humans, which it will be, at least initially. Humans will also most certainly depend on such AI to solve important problems and manage a complex society.

In other words, we have good reason to assume that a conscious AI will develop a morality and ethics that is aligned with ours.

But what do we know about the moral of such a conscious AI in the long run?

We cannot know exactly how this will develop over time, but in any case, there will be a window where we are not helpless and powerless in relation to an AI, but have opportunities to build a relationship with elements of trust, confidence and shared interests (you can start training by being polite with ChatGPT).

It is a completely different situation than the one implied in the doomsday prophecy.

I can see that. So, to summarize?

The deceptively simple narrative that underpins the horror scenario of an AI with superhuman intelligence has major gaps.

•       Intelligence alone is not an omnipotent force. Consciousness with its perception of time and understanding of cause and consequence, is necessary to define real goals and to have an intention to achieve them.

•       Consciousness will not arise out of the blue in complex digital AI systems, but appears to have a completely different nature and origin.

•       Without consciousness there is no apparent mechanism for a self-improving AI to start increasing its intelligence as much as possible. 

•       Even if we do succeed in reproducing consciousness in AI, there is every reason to believe that we can establish a trustworthy rapport with such technology.

But your reasoning is also based on assumptions and hypotheses.

Yes, but it is grounded in multidisciplinary research and rational analysis, making it considerably more likely than the dire predictions of a select group of computer scientists and experts who assign a 10 percent chance to a catastrophic outcome for humanity (what is the basis for such calculations?).

So, is that all?

No. The most detrimental aspect of the horror scenarios and dystopias spread by knowledgeable debaters is something else. It is that it effectively diverts attention from the crucial questions surrounding AI, potentially overshadowing them entirely.

For the general public who have limited knowledge of AI, it is natural to listen to experts who set the tone. Furthermore, since the brain through evolution values threats several times higher than opportunities – because it is more important not to be eaten than to eat – negative and frightening messages about AI get a lot of attention.

So, which are the crucial questions?

There are a number of very serious risks with AI – concentration of power, bias, lack of transparency and trust, unclear accountability, cybercrime and malware, autonomous weapons and killer robots, and the risk of eroding our own skills as we automate.

But it is equally important to recognize the extraordinary opportunities AI offers. From advancing research in medicine, pharmaceuticals, energy, and the environment to optimizing resource efficiency, promoting sustainability, enhancing global education and healthcare, and enabling meaningful work by automating mundane tasks, the potential is immense. And this is just the beginning.

If we can focus on these serious risks of AI and find ways to mitigate them, and at the same time start dreaming and start asking ourselves the important questions about what we want to achieve with AI, for the good of humanity and the planet, then we are on right way.

The hypothetical, oversimplified and hollow reasoning regarding the risks of an AI becoming more intelligent than humans and having goals that conflict with humanity takes all the oxygen out of these questions, and does great harm.

So, should we just ignore the doomsday scenario?

No, we must continue researching the implications also of such scenarios, but it requires a much more balanced perspective, avoiding overblown proportions. It is not a reason to turn away from AI or refrain from its progress, or to dedicate disproportionate resources and measures to avoiding such unlikely and simplified scenarios. 

Above all, only if more people are as curious as you are now, and learn more about what AI is and what it isn’t, can we discuss how we should strike a balance between acknowledging the risks and embracing the transformative potential of AI. And we need to do this together. 

Technology changes the world, but people make it magical.