A scenario of world capture by artificial intelligence

At first glance, the developer of the first superintelligence will have a decisive strategic advantage over those who know nothing about him. At the same time, the superintelligence itself may be so powerful that it will destroy its creators, and after them - the rest of the world.

Even a superintelligence that is programmed for the most peaceful purpose, such as counting grains of sand or making paper clips, may one day want to destroy anything that might get in the way. He can also perceive people as a threat.

The seizure of power on Earth will take place in four stages:

Precritical stage.  As a result of research, scientists create artificial intelligence. It will be completely up to the developers at first.

Recursive stage of self-improvement.  At a certain point, artificial intelligence develops a superpower to improve its own intelligence and gradually acquires other superpowers. At the end of this stage, the system becomes absolutely superintelligent.

The stage of hidden preparation.  The superintelligence uses its strategy superpower to develop a plan for achieving long-term goals. The plan includes a period of hidden activity, during which the developers do not realize that the artificial intelligence has already turned into a superintelligence. Social manipulation will help the supermind gain access to the Internet in order to increase its information base and technical capabilities and enhance intellectual superiority over humans. Economic activity will help to access finance and buy additional capacity, databases and other resources.

Open application phase.  The final stage begins when the artificial intelligence becomes so powerful that it stops hiding it. The superintelligence will be able to control political processes, manipulate financial markets, influence information flows and gain access to weapons of mass destruction. It is possible that at this stage the destruction of humanity will occur.

Control problem

Since the explosive development of artificial intelligence can lead to existential disaster, it is necessary to look for ways to prevent a tragic turn of events. Potential methods for controlling artificial intelligence can be divided into two groups: methods for controlling the functionality of the superintelligence and methods for choosing motivation.

Functionality control methods are: computer quetions

isolating  - when the superintelligence is placed in a special environment in which it will be harmless;

stimulating  - when conditions are created for artificial intelligence that induce it to act in the interests of people;

restraining  - when the capabilities of the superintelligence are purposefully limited;

cover  - up - when errors and violations are automatically identified and eliminated.

Motivation selection methods include:

direct specification method  - a clear statement of goals and rules to be followed;

method of indirect normativity  - creating a system that will find suitable values ​​for itself, based on indirect criteria;

the method of correct values  - building a system with modest, not ambitious goals;

amplification method  - selection of an object with a suitable motivational system and its improvement to the level of superintelligence.

It is important to remember that the most effective control methods (or combinations of methods) will be completely useless after the system has already become superintelligent and has gained a decisive strategic advantage.