Human Role in Autonomous Systems

Most of its life, AI as a technology has been used autonomously. From Google preventing bots to PayPal detecting fraudulent payments.

A usage¹ revolution came with ChatGPT. A revolution because it allowed assisted usage: a system smart enough to hold basic conversations, write emails, and summarize your meetings.1This revolution is as much a UX as a technological revolution

It produced a shock wave, as the AI industry has been working in secrecy - not because the industry wanted, rather because the regular Joe did not care at all about it. Until now.

This revolution in assisted usage only put more light on the endless possibilities of autonomous usage. When we talk about autonomous usage, we mean systems prompted by a signal rather than a human.

But what is the role of the human worker, when a system is being replaced by an autonomous entity?

Let's first see how autonomous systems can replace human capabilities to the finest.

Small note

Before going further, it is important to differentiate autonomous vs assisted from LLM vs Agents. It is not the same field at all.

In today's terms talking about AI already means agents. An agent is an LLM given a suite of tools to grasp more information about the world. The web version of ChatGPT and Claude are already agents.

Defining Autonomous Usage

The autonomous agent lives in your internal system. Preferably one customized for it. The system should wake the agent up on specific occasions requiring its work. The same as a human worker would!

For example, the system could wake the agent when an email is incoming. Or when a colleague pings it via a Slack chat.

It never misses an email. And always up for the work.

This is the core of autonomous usage: it's prompted by the system, rather than a user. And you'll see a lot of what we'll discuss can be found similar to human training.

The tools of an Autonomous Architecture

Models are very intelligent. But lock up the most intelligent man in a cave for 20 years, he'll not seem smart right out. Which is fair. Intelligence is measured by how fast and accurately you can link up information to produce a result.

In our everyday life, we use tools to gather knowledge which stays at the top of our mind. Knowledge which we do not even know is reachable.

To unlock the intelligence of autonomous agents, you need to create the tools it'll be able to use. The most common tools include web search, internal database for memory system... but can (and should) range to the domain specific knowledge.

A compliance autonomous system should have access to internal documents, emails... And an autonomous trader to financial data, portfolio state, order management system, broker access...

Give the same tools to the autonomous system a human employee would have.

Mission of the Autonomous Agent

The agent needs to know its mission.

It needs to know what it shall do, its target, and why it is in the system. As well as the tools it has access to. Basically like the same file you'll upload on Indeed to find the new recruit.

As the perfect employee, it should have its own name, position, hierarchy, and defined task.

The Learning Loop

We now have the tooling, memory capabilities and mission. But you would not send your fresh employee directly to the battlefield?

We need to implement a system for the agent to review past cases happened inside the company (eg: an incoming email) and propose an output.

The system then compares the proposed output to the actual output, and discusses it with the agent. It should discuss with the agent what it did good and wrong, and what modifications we can address to make this work better.

Every occurrence where the agent could have been woken up in the past is a learning experience. We should have each experience replay past cases in chronological order, the way a backtest replays market days.

Would a new tooling help the case for example? Would adapting the wording of the mission streamline the operations?

Each case (positive and negative) should be input into memory for further usage. The agent becomes more fit to the task the more it does.

The agent should, at regular intervals, adapt the learning loop to new situations the agent faced. This way, the agent can adapt to its environment, the same way an employee would.

Practical Example

At ARTE One, we built a compliance agent to perform the heavy lifting of the operations. Since all work from compliance comes via the email system, it fires at each incoming email.

We also gave it several tools:

• Email access
• Project Management Software
• Internal Drive Access
• Web Search

Whenever a new email comes in, the agent classifies it by importance level and assigns it to a project (eg: Cayman fund incorporation). It creates the appropriate to-do items.

For example, one to-do item would be to respond to the email with attached the appropriate document. Another to-do item could be to create a document via the knowledge of the company and available information, and then send the email (reviewed by a human).

Or another item could be to ask a human for more information or supply the document directly.

On human work

In the past years, we've seen countless theories on the future of human labor. One sentence well heard is "You will not be replaced by AI. You will be replaced by someone using AI."

Well that was true with assisted usage. With autonomous usage coming sooner than later, human labor relying on some kind of intelligence will be replaced by companies adopting autonomous usage.

However as we've seen, a lot of the value derived from such a model comes from the training loop. Which needs human data. We expect in the short term the following:

• Entire companies datasets will be bought out for training purposes. We already see that happening with code. But it will be the case for email communications, Slack messages, internal documents...
• Humans will continue to work alongside (semi) autonomous agents in the mid term. Both for social reasons, and because the data produced by humans will be used for training.
• In the long term, we'll see ground up companies adapting fairly rapidly with autonomous agents and others falling behind (notably because of social measures) or buying them outright.

As we see, the human is not directly replaced. Its role shifts from a direct operator, to being accountable for its agent, and then providing the training data for it.

The Accountability Problem

While flying planes, the most accidents happen at takeoff and landing. The remaining of the flight is very much helped by autonomous systems.

Even now, we already have the technology for planes to be fully operated automatically. But why did it not happen?

Because the role of a pilot is not to operate the plane. It's to be there if things mess up. And to be held accountable for the situation. Because we cannot blame the constructor for the tragic accident of a plane running on autopilot.

The same situation is true with all autonomous systems.

The reviewer role will persist, not out of technical necessity, but because accountability needs a human name attached. It will shrink as institutions, not brains, adapt.

Two paths for adaptation exist, either create from the ground up, or as a large company buy the company which did it from the ground up.