On a boat, the autopilot should be autonomous — but not from the skipper.
Autonomous agents act on their own. And teams don’t want to approve every API call, database query, browser click, or standard email.
But there’s a problem hiding inside the word autonomous.
Autonomous from what?
On a long passage, the skipper doesn’t steer the boat all the time. The autopilot does that. It can
- hold a course
- compensate for wind and waves
- make constant small corrections
- steer for hours
That’s autonomy. But the autopilot is not autonomous from the skipper.
The skipper still determines:
- where we’re going
- what route we’re taking
- what risks are acceptable
- when conditions have changed
- when the autopilot should be disengaged
The autopilot is autonomous in execution, not independent in purpose. That’s the distinction we need to make with AI agents. AI agents increasingly can:
- create plans
- use tools
- access systems
- execute multiple steps
- evaluate results
- change their approach
- continue working without waiting for approval
The human is moving from operator toward supervisor. And that’s a good thing.
We shouldn’t build agents that constantly interrupt us:
“Would you like me to execute step 7.a? Yes / No / Yes to All”
But the opposite extreme is equally problematic:
“I’ve completed the task. Here’s the answer.”
We need to understand the reasoning that got the agent there.
Imagine the autopilot says:
“I’ve changed course.”
And the skipper asks:
“Why?”
And the autopilot responds:
“I determined it was the optimal course.”
That’s not enough.
The skipper needs to understand and influence the reasoning, not necessarily every calculation. We need the plan at a level and in a form designed for humans — with the help of software tools — to understand, challenge, and influence the reasoning.
Imagine if the autopilot showed the skipper
- Destination — Where are we trying to get?
- Course — How are we planning to get there?
- Evidence — What information are we relying on?
- Assumptions — What are we assuming to be true?
- Constraints — What are we not allowed to do?
- Changes — What changed that caused the plan to change?
- Decision points — Where might the skipper’s judgment be required?
Now scale this to an enterprise
An organization may have many agents operating across:
- CRM
- ERP
- databases
- code repositories
- financial systems
- customer systems
Each agent can potentially make dozens or hundreds of decisions. Teams need to know:
- What did the agent reason from?
- Why?
- What information did it use?
- What assumptions did it make?
- Can someone review the reasoning?
- Can the organization reuse what it learned?
- Can a human take over when necessary?
This doesn’t mean putting a human in the loop for every action. That defeats the purpose of autonomy. It means keeping the human at the helm.
Let the agent execute autonomously, make small corrections, work for hours on simple tasks without asking for permission.
But keep the skipper connected to the plan, reasoning, and important decisions.