AI Governance · Research Report · 2026

Human Oversight That Works

Why human involvement alone is not enough, and what meaningful authority and intervention actually require.

Executive Summary

Human involvement is not the same as human oversight. A person in the process is only a safeguard if they can understand, challenge and change what the AI system does.

On 7 May 2026, the US District Court for the Southern District of New York ruled against the federal government over its termination of more than 1,400 research grants in a case that exposed a fundamental weakness in how organisations think about human oversight of artificial intelligence.

Government personnel had used ChatGPT to assess grant descriptions against a short instruction and terminated grants based on the resulting classifications. The court found no meaningful evidence that reviewers had examined the underlying applications, challenged the model's interpretation or disagreed with a single AI-generated rationale.

The case makes concrete a much broader governance problem. Organisations, regulators and policymakers increasingly require a human to remain "in the loop" of AI-assisted decisions. Yet human presence alone provides little protection if the person involved lacks the authority to intervene, the understanding needed to challenge the system or the practical competence to recognise when it is wrong.

Those conditions can also deteriorate over time. Automation bias can turn active review into passive confirmation. Workload pressure can make meaningful scrutiny operationally impossible. As AI performs more of the underlying task, the human expertise required to supervise it can weaken through disuse.

This report examines why human oversight fails in practice, what emerging research, regulation and case law suggest meaningful oversight actually requires, and how organisations can distinguish between a human who genuinely controls an AI-assisted process and one whose approval merely accompanies it.

01 · The Case That Made the Difference Concrete

A decision-maker cannot delegate judgement to an AI system and then treat the system's output as someone else's responsibility.

The 2026 ruling in American Council of Learned Societies v. National Endowment for the Humanities provides an unusually clear example of what nominal human oversight looks like when examined from the outside.

Government staff used a short prompt to assess whether research grants related to diversity, equity and inclusion, without clearly defining the term for the model or establishing how its interpretation would be validated. One reported output classified research concerning the history of whaling as relevant to the criteria being applied.

More importantly, the staff processing the results reportedly did not examine the underlying grant applications and the court could identify no example of a reviewer disagreeing with an AI-generated rationale and choosing a different course of action.

The government's attempt to separate itself from the model's classifications exposed the central accountability problem. An organisation chooses the system, defines the task, establishes the workflow and decides how much authority the output will carry. Responsibility does not disappear because an AI system sits between the instruction and the outcome.

The absence of documented disagreement also matters. A process may technically contain a human approval step while operating in practice as an automated decision system if reviewers consistently accept the system's recommendation without independent judgement.

The principle extends far beyond public administration. The same question applies wherever AI influences consequential decisions: can the organisation demonstrate that a human understood the decision, had genuine authority to challenge it and actually exercised independent judgement when necessary?

02 · The Oversight Paradox

The more work an AI system performs, the fewer opportunities the human overseeing it may have to maintain the expertise that effective oversight requires.

A growing body of research describes a structural problem at the centre of human oversight: the competence needed to identify an AI system's mistakes is not a permanent asset that can simply be placed in reserve.

Expertise is developed and maintained through practice. Yet when an AI system takes over more of the underlying task, the person assigned to supervise it receives fewer opportunities to perform that task independently.

The oversight role can therefore weaken precisely because the system it is supposed to constrain becomes more capable.

This creates a deskilling cycle. The AI performs the work. The human moves from practitioner to reviewer. But effective review still depends on the expertise that was previously developed by doing the work directly.

A lawyer who spends years approving AI-drafted contracts rather than drafting them, or a specialist who repeatedly confirms AI-generated assessments rather than reaching an independent judgement first, may gradually lose some of the practical experience needed to recognise unusual or novel errors.

Meaningful oversight therefore cannot assume that human competence will remain constant while the system performs an increasing share of the task. If independent expertise matters to the control, maintaining that expertise has to become part of the control itself.

The Oversight Challenge

AI capability is advancing faster than traditional assumptions about human supervision.

As AI systems become more capable, organisations face a difficult question: how do humans retain enough independent expertise and judgement to supervise systems that increasingly outperform them on parts of the underlying task?

Human Expert Performance

Approximate accuracy of human PhD-level experts on the GPQA doctoral-level science benchmark.

GPQA benchmark

Leading AI Performance

Approximate 2026 performance reported for leading AI systems on the same doctoral-level science benchmark.

2026 capability reporting

Automation Bias

Error rates reported in classic automation-bias scenarios where trained professionals failed to detect or appropriately challenge automated errors.

Automation-bias research

Benchmark performance and experimental conditions vary, but the governance problem is durable: effective oversight becomes harder when system capability rises, human practice falls and repeated accuracy encourages reviewers to trust automation by default.

03 · Automation Bias

Humans do not stop challenging automated systems because they become careless. They often stop because repeated accuracy teaches them that checking rarely changes the answer.

Even where a reviewer retains the expertise needed to identify an error, another predictable force works against meaningful intervention: automation bias.

When an automated system is consistently accurate, people naturally begin to treat its output as the default answer. Active evaluation gradually becomes confirmation, particularly when workloads are high and the cost of challenging the system feels greater than the apparent benefit.

Research has demonstrated this effect even among highly trained professionals. In classic flight-simulator studies, professional pilots failed to respond appropriately to automation failures despite having access to information that could have revealed the problem.

The same mechanism can emerge in AI-assisted professional work. A reviewer who sees hundreds of correct recommendations may gradually stop approaching the next recommendation as an independent judgement problem and begin treating it as something that merely needs approval.

Time pressure intensifies the effect. If an organisation introduces AI partly to increase throughput and then measures reviewers against that increased volume, it may create a process in which genuine scrutiny is operationally incompatible with expected performance.

Effective oversight therefore has to be designed around actual human behaviour. Telling reviewers to remain vigilant is not enough if the surrounding workflow systematically rewards speed, agreement and reliance on a system that is usually correct.

04 · Formal Oversight Is Not Real Oversight

Regulation increasingly distinguishes between a human who is present and a human who has the knowledge and authority to change the outcome.

The distinction between formal and meaningful human involvement predates the current generation of AI systems and is already embedded in important areas of data protection and AI regulation.

Article 22 of the EU General Data Protection Regulation places restrictions around decisions based solely on automated processing where those decisions produce significant effects. Regulatory interpretation has consistently emphasised that token human involvement is not enough to transform an automated decision into a genuinely human one.

The person involved must be capable of weighing the relevant information and must have genuine authority to change the outcome rather than simply confirming a decision already made by the system.

The EU AI Act develops this principle further. Article 14 requires effective human oversight for high-risk AI systems and specifically recognises the risk of automation bias. Article 4 separately introduces AI literacy obligations, reinforcing the principle that oversight without sufficient understanding cannot be assumed to be effective.

Emerging approaches elsewhere increasingly reflect the same logic through requirements for risk assessment, senior accountability, transparency and rights connected to automated or AI-assisted decisions.

The direction is increasingly clear: organisations will need to demonstrate more than the existence of a human review step. They may need evidence that the reviewer was competent, sufficiently informed, genuinely empowered and operating within a process that made independent intervention realistically possible.

05 · What Meaningful Oversight Actually Requires

Meaningful human oversight is a property of how the system, workflow and human role are designed, not a consequence of placing a person beside the machine.

Real authority to overrule. A reviewer must have genuine permission to reject, reverse or escalate an AI-generated recommendation without being structurally penalised for doing so. Organisations should be able to demonstrate that this authority exists in practice, not merely in policy.

Enough time to conduct meaningful review. If workload expectations allow only seconds to approve a decision that requires minutes to evaluate independently, the process is designed for confirmation rather than oversight.

Enough understanding to challenge the system. Reviewers need to understand the system's purpose, limitations, relevant inputs and the basis on which its output should be questioned. Reading a conclusion without understanding how it should be evaluated is not meaningful scrutiny.

Protected competence in the underlying task. Where oversight depends on professional judgement, organisations need deliberate mechanisms for maintaining that judgement through unassisted practice, training, calibration and periodic independent assessment.

Clear escalation and accountability. Responsibility should be explicit across system design, prompt or instruction design, review, override, escalation and final decision-making. Ambiguous ownership makes it easier for accountability to disappear between the human and the system.

Evidence that intervention is real. Override rates should never be treated mechanically, but an organisation that cannot identify any meaningful instance in which human judgement changed an AI-supported outcome should ask whether the review process is genuinely independent or merely procedural.

06 · Designing Oversight That Sustains Itself

Oversight has to be maintained against the same forces that gradually weaken it: dependence, time pressure and declining independent practice.

Judge oversight on substance rather than form. A policy requiring human review should be tested against what reviewers actually do, whether they understand the decision and whether they can demonstrate meaningful intervention when the system is wrong.

Protect independent judgement before automation begins. For consequential decisions, organisations should define what humans remain responsible for before introducing the AI system rather than allowing responsibility to drift as adoption increases.

Maintain unassisted competence deliberately. Where independent professional expertise is part of the control environment, periodic work without AI assistance can help preserve the practical judgement needed to challenge the system when unfamiliar situations arise.

Design workloads around genuine review. Productivity gains should not be captured by increasing case volumes to the point that reviewers no longer have enough time to perform the oversight function the governance model claims they provide.

Monitor the oversight system as well as the AI system. Organisations should examine patterns such as overrides, escalations, review times, disagreement rates and repeated reviewer behaviour for evidence that active scrutiny is becoming passive approval.

Build translation capability between technology and accountability. Boards, regulators, risk teams and courts increasingly need to understand how AI-assisted decisions were made after the event. Organisations need people capable of translating technical system behaviour into evidence that governance and accountability processes can actually use.

07 · A Practical Test for Whether Oversight Is Real

The strongest evidence of human oversight is not a policy requiring intervention. It is evidence that meaningful intervention remains possible and actually happens.

The first diagnostic question is simple: could the person responsible for overseeing this AI system still perform the underlying task without it?

If the answer is no, the organisation should ask what independent expertise remains available to identify errors the system itself cannot recognise.

The second question is equally direct: can the organisation identify a specific, documented example of a human reviewer disagreeing with the system and choosing differently?

An absence of overrides does not automatically prove that oversight has failed. A highly accurate system may legitimately generate few disagreements. But where no meaningful intervention can be demonstrated over a substantial period, the organisation should investigate whether reviewers are exercising independent judgement at all.

A third question completes the test: does the reviewer have enough time, information and organisational authority to intervene without the workflow working against them?

An organisation that cannot answer these questions with concrete evidence may have a compliance record showing that humans are present in the process. That is not necessarily the same thing as having a functioning system of human oversight.

Conclusion

Human oversight is not a person in a workflow. It is a functioning capability to understand, challenge and change what the system does.

Requiring a human to remain in the loop responds to a genuine governance need, particularly where AI influences decisions with material consequences for individuals, organisations or society.

But human presence alone does not create meaningful control.

Oversight can weaken through automation bias as reviewers learn to trust systems that are usually correct. It can weaken through workload pressure when there is no longer enough time for independent scrutiny. And it can weaken over the longer term as the expertise required to challenge AI deteriorates because the human no longer performs the underlying task.

Meaningful oversight therefore has to be deliberately designed and continuously maintained. Humans need genuine authority to intervene, sufficient understanding and information to know when intervention is necessary, enough time to exercise judgement and continued practice of the expertise on which that judgement depends.

Where those conditions do not exist, the organisation may still have a human approval step. What it does not necessarily have is human oversight. The difference between the two is the difference between a person who controls the system and a signature that follows it.

References & Sources

Research and evidence informing this report.

This report draws on publicly available legal decisions, regulatory requirements, AI governance research and analysis of human interaction with automated systems. Legal and regulatory requirements continue to evolve and should be verified against current primary sources before being relied upon for legal or compliance decisions.

Sidley Austin LLP. When 'The Devil Made Me Do It' Is Not a Defense: Lessons in AI Governance and Organizational Oversight from an SDNY Decision. 2026.

Referenced for legal analysis of the 2026 SDNY decision and its implications for AI-assisted decision-making, human oversight and organisational accountability.

View source

US District Court, Southern District of New York. American Council of Learned Societies v. National Endowment for the Humanities. 2026 WL 1256545, 7 May 2026.

Primary legal authority referenced for the court's examination of AI-assisted grant classifications, human review and decision-maker accountability.

Steidel, I. and Gieger, B. / World Economic Forum. The Oversight Paradox: Why Human Control Over AI May Be Eroding the Very Competence It Requires. 2026.

Referenced for the oversight paradox, the deskilling cycle and analysis of how increasing AI capability can weaken the independent human competence required for effective supervision.

View source

Anthropic. The Anthropic Economic Index. 2026.

Referenced as supporting research on the changing relationship between AI systems, human work and the distribution of tasks between people and increasingly capable models.

View source

European Union. EU Artificial Intelligence Act, Article 14: Human Oversight and Article 4: AI Literacy.

Referenced for regulatory requirements concerning effective human oversight of high-risk AI systems, awareness of automation bias and organisational AI literacy.

View source

European Union. General Data Protection Regulation, Article 22: Automated Individual Decision-Making.

Referenced for the regulatory distinction between solely automated decisions and meaningful human involvement in decisions producing legal or similarly significant effects.

View source

California Privacy Protection Agency. Automated Decisionmaking Technology Regulations. 2026.

Referenced as supporting regulatory context for risk assessment, organisational accountability and governance of automated decision-making technologies.

Case details, benchmark figures and regulatory references in this report are drawn from publicly available sources current as of mid-2026. AI capabilities, regulatory requirements and legal interpretations continue to evolve. Readers relying on this report for legal, regulatory or compliance decisions should verify all relevant requirements and citations against current primary sources. The durable governance principle examined here is that meaningful human oversight requires demonstrable authority, competence, understanding, time and the practical ability to intervene.

Scroll