The Smartest AI Is Not the One You Should Trust Most
As AI’s biggest champions warn about its dangers, business leaders need to rethink where trust belongs: not inside the model, but in the system that keeps it accountable.
The world’s smartest employee has just joined your company.
It can analyse customers, write code, design campaigns and recommend business decisions. It works around the clock, never asks for an increment and can produce a strategy before you finish explaining the problem.
Impressive.
Now, would you let it set its own permissions, approve its own decisions and investigate its own mistakes?
Probably not. Even the most brilliant employee does not get to become the finance department, compliance officer and internal auditor simply because they performed well in the interview.
Yet that is the temptation with enterprise AI: to mistake extraordinary intelligence for a license to act.
Microsoft Chairman and CEO Satya Nadella’s post, “Models as Insider Risks in the Super Intelligence Era,” challenges precisely this assumption. His argument centres on the risks of granting models sensitive access and operational authority without sufficient independent controls.
It also offers a powerful starting point for a different conversation about trust:
The most trustworthy superintelligence system will not be the one with the model we trust most. It will be the one that enables us to trust the model the least.
The implication is not that we should stop using powerful AI.
It is that we should stop making faith in the model our primary safety mechanism.
The warning is coming from the people building the future
Concern about AI is no longer confined to its critics. Some of its most influential builders are publicly discussing the risks.
They are not delivering one identical message. Their concerns range from misuse and cybersecurity to concentration of power and loss of human control. Their views on acceptable risk also differ.
But those differences make the conversation more useful—not less.
Anthropic CEO Dario Amodei’s essay, “The Adolescence of Technology,” examines risks associated with increasingly powerful AI, including unpredictable behaviour, misuse and the concentration of power. In “We Must Pace the Frontier,” he also identifies loss of control, cyberattacks, bioterrorism and economic disruption as serious concerns.
Google DeepMind CEO Demis Hassabis has highlighted biosecurity and cybersecurity risks, emphasising the need for reliable systems, stronger defences and international cooperation.
OpenAI CEO Sam Altman has warned that increasingly capable autonomous systems could outpace institutions and make decisions people no longer understand or control. At the same time, he has argued that society should accept some risks to capture AI’s benefits, distinguishing those from catastrophic dangers.
Nadella brings this debate directly into the enterprise. His focus is on what happens when organisations place models inside agentic systems, give them access to sensitive information and allow them to take consequential actions. The deploying organisation, he argues, cannot outsource its responsibility to the model provider.
The useful conclusion is not “all AI is dangerous.”
It is that intelligence, reliability, security and accountability are different things. A business needs to evaluate each—not assume that excellence in one guarantees the others.
From “What did it say?” to “What did it do?”
Consider an illustrative marketing scenario.
An AI assistant recommends an unsuitable retention offer. A marketer reviews it, rejects it and moves on.
Now imagine an agent that selects the audience, calculates the offer, changes the campaign and sends the communication without review.
The original mistake may be similar. Its consequences are not.
The difference is delegated authority.
This is the transition Nadella’s post addresses: AI moving from generating outputs to acting on an organisation’s behalf.
For business leaders, that changes the most important question.
“How accurate is the model?” remains relevant.
But it must be accompanied by:
“What can this system do when it is wrong?”
A model’s capabilities determine how useful it might be. Its permissions determine how far a mistake can travel.
That is why an impressive demonstration is not sufficient evidence of readiness for unrestricted deployment. The demonstration shows what the system can accomplish. The deployment design must establish what it cannot do.
Trust should survive a mistake
There are two ways to think about trustworthy AI.
The first asks us to believe that the model will behave correctly.
The second asks whether the surrounding system remains dependable when the model behaves incorrectly.
The second is the more demanding standard.
Nadella’s argument calls for independent controls, auditability and separation between the model and its operating environment. He also cautions that model-generated reasoning is not consistently faithful and that placing one opaque model under another can create “nested black boxes” rather than genuine accountability.
A persuasive explanation is therefore not the same as an independent check.
A second AI saying “that looks fine” is not automatically an audit.
And a model promising to follow policy is not equivalent to a system preventing an unauthorised action.
The operative principle is containment: allow the intelligence to contribute while limiting the authority it can exercise.
This does not mean treating every model as malicious. It means refusing to make the organisation’s safety dependent on the model being infallible.
We already understand this elsewhere.
An employee can be trusted and still need approval to exceed a spending limit. A senior executive can exercise judgment and still face an audit. A supplier can have an excellent reputation and still operate under a contract.
Controls do not negate trust. They make trust less fragile.
For CX leaders, the stakes are customer promises
In customer experience, an AI action can become a promise.
An offer can create an expectation. A service response can commit the brand to a remedy. An eligibility decision can determine how a customer is treated.
Consider a hypothetical retention agent optimising for reduced churn. Without appropriate boundaries, it might recommend increasingly generous offers or make commitments outside approved policy.
The problem is not necessarily that its objective is wrong. The problem is that the objective is incomplete.
“Retain the customer” does not, by itself, specify acceptable cost, approved terms or permitted commitments.
For CX and marketing leaders, I would translate Nadella’s separation of intelligence and control into three layers.
1. Recommendation
What may the agent analyse, propose or draft?
Examples include suggesting an audience, drafting a response or recommending an offer.
2. Execution
What may it actually change?
This requires explicit decisions about data access, campaign activation, pricing changes, customer communications and financial commitments.
3. Escalation
Which conditions require additional approval or suspension?
Examples might include an unusual discount, a sensitive complaint, conflicting customer records or an action outside the agent’s defined scope.
These are design choices, not a universal checklist. The important point is to decide them before deployment—not after the first incident.
Build boundaries outside the model
“Follow our policies” is a useful instruction.
“You cannot execute outside these permissions” is a different kind of protection.
An enterprise needs both.
For a customer-facing agent, a bounded operating environment might specify:
Which data sources it can access and for what purpose.
Which customers, products and channels it can act on.
What financial or operational limits apply.
Which actions require approval.
What events trigger a pause.
Who can intervene and revoke access.
These are practical applications of Nadella’s call for independent access and action controls.
The distinction matters because the model should not be able to reinterpret its way into greater authority.
A retention target should not become permission to ignore an offer limit. A request to personalise should not become permission to access every available customer record.
The intelligence can operate within the boundaries. It should not own the boundaries.
Make recovery part of the experience design
Most AI demonstrations focus on successful task completion.
A mature deployment also needs a credible failure story.
If an agent makes an unauthorised change, can the organisation reconstruct what happened? Identify affected customers? Stop further actions? Reverse what is reversible? Explain the issue without relying solely on the model’s own account?
These questions bring Nadella’s emphasis on observability and containment into the customer-experience operating model.
They also expose an important distinction: stopping an agent is not the same as repairing the consequences of its actions.
Some actions can be rolled back. Others—a message already delivered, a promise already made—need remediation.
The operating design should therefore assign ownership not only for intervention, but also for investigation and customer recovery.
“Someone will handle it” is not a recovery plan.
The boardroom needs a different scorecard
Model performance should remain part of the evaluation. But it should sit alongside five questions derived from Nadella’s control principles.
Question | What leaders need to establish |
What can the agent access? | A defined identity and justified permissions. |
What can it change? | Explicit limits on customer, financial and operational consequences. |
What checks its decisions? | Validation that does not depend solely on the model’s own judgment. |
What evidence remains? | Records that support independent investigation. |
Who can stop it? | Clear ownership and a tested intervention mechanism. |
The strongest test follows:
Would these protections still work if the model were mistaken, manipulated or compromised?
That question shifts governance from a statement of confidence to a demonstration of control.
We do not need superintelligence to need safeguards
The headline debate concerns superintelligence. The enterprise decision is more immediate.
OpenAI’s governance essay describes superintelligence as future systems dramatically more capable than artificial general intelligence. That framing should not be mistaken for proof that today’s enterprise agents meet the threshold.
Nor should frontier-scale warnings imply that every AI workflow carries the same risk.
Controls should be proportionate to the authority being delegated and the consequences of misuse or error.
But a system does not need to surpass human intelligence to make an unauthorised commitment. It only needs sufficient access and permission.
The governance question begins when we delegate power—not when we agree on what to call the intelligence.
The smartest model may win the demo
The warnings from AI’s leading CEOs should not push businesses into paralysis. They should push businesses toward better design.
Nadella emphasises independent controls. Amodei describes a broad spectrum of potential harms. Hassabis stresses defensive capability. Altman acknowledges both transformative benefits and the possibility of losing human control. Their prescriptions differ, but none supports treating capability as a substitute for accountability.
For enterprise leaders, the opportunity is to build systems that can use extraordinary intelligence without granting extraordinary discretion by default.
Let the model recommend.
Let permissions govern what it can execute.
Let independent evidence show what happened.
And let accountable people retain the ability to intervene.
The smartest model may win the demo.
The system that does not require blind faith deserves the business.
