Designing Human Oversight That Works

Human oversight is supposed to be the safeguard behind every AI system. In practice, it is often a signature, not a safeguard.

Every major AI governance framework rests on the same premise: a human stays in control. The EU AI Act requires human oversight of high-risk systems. The GDPR restricts decisions taken by machines alone. Most corporate AI policies promise that a person reviews an output before it reaches anyone who matters.

In practice, that promise is frequently hollow. Reporting on Cigna's claims-review process found that medical directors were rejecting more than 300,000 payment requests over a two-month period at an average of 1.2 seconds per case, a pace that makes genuine medical review impossible. When a subset of those denials were appealed to federal judges, roughly 80 to 90% were overturned. Fewer than 0.2% of affected patients ever appealed at all.

This is not a story about one insurer. Deloitte's 2026 State of AI in the Enterprise report found that while 74% of companies plan to deploy AI agents within two years, only 21% describe their model for governing those agents as mature. Gravitee's 2026 State of AI Agent Security report found that 48% of AI agents already running in production operate with no meaningful security oversight at all.

This whitepaper sets out what has to be true for human oversight to be real rather than symbolic: what people need to understand about the systems they are overseeing, what they need to be able to see in real time rather than after the fact, and what authority, time and protection they need to be genuinely empowered to act on what they see.

A person cannot meaningfully oversee a system they do not understand well enough to doubt.

The EU AI Act is explicit that presence is not the same as competence. Article 14 requires that individuals assigned to oversee high-risk systems be specifically trained on the system's capabilities and limitations, on automation bias, and on the conditions under which its output should be overridden or disregarded. Article 26 goes further, requiring deployers to ensure those individuals have the necessary competence, training, authority and support, not simply a badge that says "reviewer."

Understanding a system also means understanding a well-documented psychological reflex: automation bias. Classic flight-simulator research found that even professional pilots failed to act on problems an automated system missed, or followed faulty prompts without cross-checking, with error rates of around 55% recorded in scenarios where the information needed to catch the fault was readily available to check. The same reflex transfers directly to knowledge work: a radiologist whose AI flags anomalies with very high accuracy will, after the hundredth scan, gradually shift from evaluating the image to confirming the machine.

A July 2026 World Economic Forum analysis by Isabell Steidel and Benedikt Gieger names the structural version of this problem the "oversight paradox": the competence a person needs to oversee a system is built and maintained through practice, the same practice the system is now doing instead of them. On the GPQA benchmark of doctoral-level science questions, human PhD experts score roughly 65%; the strongest model scored 39% on release in late 2023, and roughly 94% by 2026. On SWE-bench, real open-source coding problems, the best system resolved under 5% at launch in 2023 and now resolves well over 90%. The tasks moving fastest to AI are precisely the expert, judgement-heavy ones that oversight depends on.

The practical implication is that "understanding" cannot be a one-time induction delivered before a system goes live. It has to be deliberately maintained over time, precisely because the system doing the work is simultaneously eroding the hands-on practice that would let a person judge that work competently.

An overseer cannot act on a problem they have no way of seeing.

A 2026 Harvard Business School field experiment, published as Working Paper No. 25-001, tested exactly what kind of "seeing" actually helps. Across 228 evaluators screening real innovation submissions, human reviewers given plain AI recommendations aligned with those recommendations 19.4 percentage points more often than an unassisted control group. When the same recommendations were paired with a narrative explanation, alignment rose a further five points, concentrated specifically on rejection decisions. Decision quality, however, did not improve. The researchers term this the human-AI oversight paradox: under real cognitive load, an explanation that reads as more convincing increases reliance on the recommendation rather than strengthening independent judgement.

That finding matters because it separates two things organisations routinely conflate: a persuasive rationale is not the same as a signal to check. Real visibility means surfacing uncertainty, disagreement between models or evaluators, and specific edge cases, not presenting a confident-sounding narrative for the reviewer to accept.

Cigna's claims process illustrates the same gap from the opposite direction. The failure was not the absence of a policy requiring a doctor to review each flagged denial. It was that reviewers working through claims in bulk had nothing in the interface to actually look at, no distinguishing signal between a routine, correctly flagged case and one the algorithm had likely got wrong. Without that signal, 1.2 seconds is not a shortcut around review. It is the only amount of time review can possibly take.

The design implication follows directly: oversight interfaces should surface the specific basis for a system's output and flag the system's own uncertainty, rather than simply presenting a fluent, confident narrative optimised to be approved.

The test is not whether a human is in the loop. It is whether that human could still do the job without the AI.

The World Economic Forum's oversight paradox analysis closes with three practical directions worth treating as a test rather than an aspiration. First, substance over form in human-in-the-loop rules: oversight provisions should require a demonstrated ability to override the system, not merely the procedural presence of a reviewer. Second, deliberate maintenance of competence: knowledge workers in AI-augmented roles need structured practice without the system, in the same way pilots must periodically fly without autopilot, treated as an institutional requirement, not an individual's good intention. Third, bridge actors: people who understand what a system actually does and can translate that into accountable judgement, of whom the analysis notes there are currently far too few.

That framing yields a genuinely practical test any organisation can apply to any oversight role today. Could this person still do the job without the AI, and is that capacity being actively kept alive? And can you point to a specific, documented instance where this reviewer actually overrode the system, not a policy that permits it, an occasion where it happened?

An approval rate that never dips, echoed in the pattern behind both the near-90% reversal rate on appealed Cigna denials and the 48% of production agents already running with no oversight logged at all, is not evidence of a well-functioning system. It is evidence that the review trigger is too broad, too rushed, or too disempowered to ever produce a genuine no.

Oversight that has never said no is not evidence a system is safe.

Cigna's 1.2-second reviews, the 90% reversal rate they produced on appeal, and the fact that only a tiny fraction of affected patients ever appealed at all describe the same underlying failure that Deloitte and Gravitee's 2026 research finds spreading across enterprise AI more broadly: organisations are deploying faster than they are learning to oversee what they deploy.

None of this is inevitable, and none of it requires slowing adoption to a halt. It requires treating oversight as something designed rather than assumed: people trained to understand the specific system in front of them, interfaces built to show them what actually needs checking, real authority to say no without penalty, time genuinely budgeted for the review being asked of them, and deliberate protection of the unassisted practice that keeps their judgement sharp enough to be worth having.

The two questions this whitepaper keeps returning to are the ones worth asking of any oversight role today: could this person still do the job without the AI, and is that capacity being actively kept alive, and can you point to a real, documented instance of them saying no. Where the honest answer to either is no, what exists is a signature. Where the answer to both is yes, oversight is doing the job it was designed for.