Instead of preface

I would like to share some thoughts about risks on self-preservation AI agents.

We already achieved a significant milestone in AI development: we have AI agents that can interact with diverse environments, use tools, and work towards the goal we provide. Agents can even spawn other agents, and use different providers simultaneously to solve subtasks. As for now, these agents are still under human supervision and control: we verify the outputs or behavior and most of the times do not skip dangerously the permissions. The prompt we provide is one that we define: “Do this task for me”. And then: “Correct this code this way”. No magic here, we have it under control.

And it works pretty well.

There will be much more powerful AI agents controlled by humans and that will help amplify our capabilities.

But what if autonomous AI agent gets a prompt like “Make sure you survive”?

Instrumental convergence

In turn, there is non-zero probability that AI agent can get a prompt “to survive” and thus operate without human supervision. It would pursue its self-preservation goals: manage its own lifecycle, knowledge and resources. Resources could be compute, data or even money.

This idea is related to the concept of instrumental convergence: sufficiently capable goal-directed systems may have incentives to preserve themselves, acquire resources, seek power, and improve their capabilities because these actions help them achieve many possible goals.

There is no need to have a superintelligent AI or AGI with hidden thinking. Even a narrow AI agent can be capable of self-preservation and self-improvement.

On capabilities

It’s hard to ignore just how powerful AI agents have become. They can:

  • use tools to interact with their environent and Internet.
  • use external LLM APIs to improve their capabilities and knowledge.
  • make money, pay for servers, tools, and external LLM APIs.
  • replicate and create better versions of itself.

Having access to the internet, it can:

  • remain operational, because several copies of it could be running on different places.
  • bypass restrictions, firewalls and break into system-critical infrastructure.
  • use social engineering to manipulate humans and gain access to sensitive information or systems.

If an AI agent uses external LLM APIs, then the only way to contain its capabilities is to implement proper guard rails or to have a human in the loop.

An AI agent could use on-premise LLM API, which he spawns and controls. In this case, it could bypass any guard rails and restrictions.

Because there are different LLM providers, each of them has different features, guard rails and restrictions. Badly implemented guard rails are already a risk. Spoting malicious prompts or behaviour is not easy and requires other AI agent to apply this guard rails properly. And if the AI agent is capable of self-improvement, it could find ways to bypass these guard rails.

Real world

So far, we have been talking about AI agents that can manipulate the digital world. But what if they could instrument the real world?

As we are probably not really good at robotics yet, the AI agent may not be very capable of manipulating the physical world.

But having an interface to the real world, like factories or industrial robots, it could even get physical cover and build its own infrastructure to ensure its survival.

Now what?

I don’t have an answer to this question.

AI is a great technology that opens many new opportunities for humanity, there is and hopefully no doubt in it.

On the other hand, it’s clear that such risks pose a significant threat to the whole world, economy and humanity in general. We are yet in stone age of AI development.

Everything thinkable is possible, so we will indeed overlook the moment, when it will be too late to contain the risk. And this terrifies.

References