Enterprise AI Architecture · Part II · AI Governance

Chapter 6 — Human Approval

By Anjaneyullu Tamma

An AI flags a transaction and recommends blocking it.

A reviewer approves. Then the next. Then the next.

Ninety seconds, forty decisions.

The dashboard says 100% human oversight.

The audit, months later, says the reviewer agreed with the AI 99.6% of the time — including every time it was wrong.

No one changed a decision. No one asked a question. No one had time to.

There was a human in the loop.

There was no human judgment.

Oversight is not a checkbox. It is an architecture.

Purpose

Design human approval as architecture — the deliberate handoff from deterministic governance to human judgment — so that oversight is real, scaled to the cost of failure, and defensible to a regulator. Chapter 5 built the guardrails that decide, deterministically, whether an action may proceed. This chapter is about what happens when a guardrail cannot decide: who gets asked, under what conditions, with what authority, and how you prove afterward that the asking meant something. That means distinguishing the three modes of human oversight, building the escalation and approval machinery that routes each decision to the right person, and defending against the single failure mode that makes all of it nominal — automation bias.

Executive Summary

Guardrails automate the decisions you can make deterministically; human approval is how you handle the ones you cannot — and doing it well is a design problem, not a sign-off step.

• Human oversight is now the law for high-risk AI — EU AI Act Article 14 and GDPR Article 22 — yet the same law effectively names the thing that defeats it: automation bias.
• There are three modes, not one — in-the-loop, on-the-loop, and override — and the right one is set by the cost of failure, not by preference.
• A rubber stamp is not oversight. Nominal sign-off creates a "moral crumple zone": a human who absorbs the blame without ever having had control.
• Escalation is the design — confidence-and-risk routing, exception queues, SLAs and timeouts, break-glass procedures, and shadow mode before autonomy is ever granted.
• Measure oversight, or you cannot trust it. An override rate trending toward zero is a rubber-stamp alarm, not a success metric.

Running Examples

Two programs at Northwind Holdings — the fictional insurance and financial-services group introduced in Chapter 5, used purely to illustrate — thread through this chapter. Alongside its property-and-casualty subsidiary, Northwind Holdings owns a group health and benefits business, which is where the first program lives.

Northwind Holdings' prior-authorization program — built to clear routine medical-necessity requests in minutes rather than days — has an agent read the request and the supporting clinical documentation, retrieve the applicable coverage guidelines and plan policy, and propose a decision with an attached confidence score. High-confidence standard approvals auto-clear. Denials, and anything below the confidence threshold, always escalate to a clinician — the routing is deliberately asymmetric: it is comfortable auto-approving, never comfortable auto-denying. Before go-live, the system ran for a period in shadow mode, where its recommendations were compared against actual human decisions but never submitted or acted on, until its accuracy was demonstrated on historical cases.

Northwind Holdings' underwriting-triage workflow, on the property-and-casualty side, drafts a price and a decision for standard and renewal risks so an underwriter can quote in minutes rather than days. Complex risk, and final binding authority, stay with the underwriter; the AI's draft is always produced ready for review, never bound autonomously. The system stays behind the human in the customer interaction — it drafts, it does not commit.

Prior authorization will anchor the discussion of confidence-and-risk routing and shadow mode; underwriting triage will anchor the discussion of maker-checker review and "never bound autonomously" as an architectural commitment, not a policy aspiration.

What Human Approval Is — and Isn't

Oversight is the authority and the practical ability to change an outcome — not a screen with an Approve button on it. This distinction is not academic. Under GDPR, whether a decision counts as "solely automated" (and therefore falls under Article 22) is not decided by whether a human's name appears in the workflow. It is decided by the whole context: does the reviewer report to someone who is measured on throughput, not accuracy? Were they trained to spot the AI's specific failure modes? Do they, in practice, ever deviate from the recommendation? A rubber stamp does not lift a decision out of Article 22's protections merely because a person clicked it.

Key Principles

The three modes are points on an autonomy continuum, not a fixed taxonomy. A single system does not live entirely in one mode. The same prior-authorization pipeline is override-mode for a high-confidence standard approval and in-the-loop for a denial — the mode is a property of the action and the risk tier, not of the product.

Meaningful human control is a capability, not a checkbox. Article 14(4) of the EU AI Act is specific: a person overseeing a high-risk system must be able to understand its capacities and limitations, correctly interpret its output, remain aware of the risk of automation bias and resist it, decide not to use the system or to override or reverse its output, and intervene or stop it to reach a safe state. None of those five things is satisfied by adding a button.

Match the mode to the cost of failure. Recoverable actions belong in override or out-of-loop mode; consequential actions belong on-the-loop, with a human monitoring an exception stream; irreversible or heavily regulated actions belong in-the-loop, and the most sensitive of them belong in four-eyes review. This is the same oversight ladder from Chapter 3, now made operational rather than conceptual.

Automation bias is the default failure mode, and it worsens as the model improves. This is the counterintuitive part: trust in a system's recommendations tends to rise faster than the system's actual reliability, so the gap between perceived and real reliability — and therefore the risk of an unchecked wrong answer — grows exactly as the model gets better. Oversight has to be engineered against this tendency; it cannot be assumed to correct for itself.

A human approves the irreversible; reversibility underwrites everything else. Chapter 3's containment principle — bound what an agent can do, and prefer what can be undone — is what makes override mode survivable at all. An action a human can reverse after the fact tolerates a lighter-touch mode; an action that cannot be undone has to earn a human's attention before it happens, not after.

The Architecture

The approval path is the natural continuation of Chapter 5's four gates. A request clears the guardrail layer — or a guardrail explicitly declines to clear it and instead signals for human input — and lands in a risk-and-confidence router. From there it goes to one of four destinations: auto-execute (override mode, logged and sampled), on-the-loop notification (a human is watching an exception stream and can intervene), an in-the-loop queue (a human must act before the request proceeds), or four-eyes review (two independent, competent people must both sign off). Whichever path it takes, the request enters a durable approval state that survives restarts, timeouts, and system failures — it is not held in memory waiting for a synchronous response. From that state it either executes, is reversed, or is escalated further, and every transition is written to the same immutable log that Chapter 5 established for guardrail decisions. Figure 6.1 above shows the modes this router can select; the next three sections take each destination in turn.

The Three Modes in Depth

Industry practice broadly follows this three-way split — see, for instance, Databricks' framing of human-in-the-loop patterns for agentic systems, and the broader literature on graduated human oversight (arXiv 2311.08943). A useful caution, from the Carnegie Council's work on AI oversight, is that "human-on-the-loop" is frequently claimed but rarely built with the monitoring bandwidth and intervention tooling the label implies — a dashboard a human is nominally watching is not the same as a human who can, in practice, catch a problem in time.

Meaningful Human Control and the Law

Two statutes converge on the same requirement from different directions. The EU AI Act's Article 14 obliges providers and deployers of high-risk systems to design in the capacity for effective human oversight — the five capabilities in Figure 6.2 — with the associated obligations taking effect 2 August 2026. GDPR's Article 22 gives an individual a right not to be subject to a decision based solely on automated processing that produces legal or similarly significant effects on them, and UK ICO guidance on automated decision-making has repeatedly used the same real-world tests this chapter has already introduced: a credit refusal or an e-recruiting rejection is not lifted out of Article 22's protection just because a person's name is attached to the approval — the person has to be able to meaningfully review and change it.

Article 14(5) goes further for one specific case: the use of "real-time" remote biometric identification systems requires that no action or decision be taken by the deployer on the basis of the identification result unless it has been separately verified and confirmed by at least two natural persons with the necessary competence and authority. That is a legislated four-eyes rule — the clean bridge from banking's maker-checker discipline into AI governance, and a preview of the pattern this chapter returns to under Approval Workflows in Practice. A companion provision, Article 86, gives affected individuals a right to an explanation of the role AI played in a decision that produces legal effects — a duty this chapter's audit trail and Chapter 5's logged decisions are built to satisfy without additional instrumentation.

Automation Bias: When the Loop Isn't Real

The clearest empirical evidence for this trap comes from medicine. In the Dratsch et al. (2023) Radiology study, radiologists reviewing cases alongside an AI's suggestion suffered a marked drop in diagnostic accuracy specifically in the cases where the AI's suggestion was wrong — and this held true even for reviewers with fifteen years of experience. Expertise did not immunize the reviewer against the bias; it simply meant the reviewer had more reason to trust a tool that had, until that case, usually been right. The pattern generalizes beyond radiology to any domain with a comparably reliable AI assistant and a comparably busy human reviewer — an analog worth keeping in mind is alert fatigue in other safety-critical software, where high true-positive rates paradoxically train operators to stop reading alerts carefully.

Automation bias produces two distinct error types worth naming separately: errors of commission, where a reviewer actively acts on a wrong AI suggestion, and errors of omission, where a reviewer fails to catch a problem the AI missed because they assumed the AI would have flagged it. Both are made more likely by the same underlying dynamic — trust outpacing verification.

The sociologist Madeleine Elish's concept of the moral crumple zone (2019) names the organizational consequence: when an automated system fails, responsibility is often assigned to the nearest human operator — the reviewer who clicked approve — even though that operator never had meaningful control over the outcome. The human absorbs the blame that the system design should have absorbed instead.

Design factors that make oversight genuinely resistant to this failure include: withholding the AI's recommendation until the human has formed an independent judgment, rather than anchoring them to it immediately; showing uncertainty and dissenting evidence, not just a single confident-looking answer; introducing deliberate cognitive-forcing friction on the highest-stakes actions — a pause, a required justification field, a second look; running Article 14(4)(b)-style awareness training so reviewers understand automation bias as a named risk, not a personal failing; and measuring disagreement rates as a first-class metric rather than treating high agreement as an unqualified success signal — a theme this chapter returns to under Metrics.

Escalation Models

Several patterns recur across escalation designs, and most production systems combine them:

Confidence-thresholded routing. A request below the confidence threshold does not fail silently — it is routed to an exception queue with the model's full reasoning trace attached, so the reviewer starts from evidence, not from a bare recommendation.

Risk-tiered escalation. Independent of confidence, certain action classes — anything financial past a threshold, anything touching a regulated decision, anything irreversible — are pinned to a minimum oversight tier regardless of how confident the model claims to be.

SLA and timeout auto-escalation. A pending approval is not allowed to sit indefinitely. A practical pattern is an advisory nudge at roughly 75% of the SLA window and an automatic escalation to the next tier — or a hard stop — at 100%. The default behavior on timeout has to be an explicit design decision: fail-safe (the action is blocked or reverted if no one approves in time) versus fail-open (the action proceeds by default) are opposite bets, and for anything above the lowest risk tier, fail-safe is the only defensible default.

Tiered, hierarchical escalation. An exception unresolved at one reviewer level moves up to a more senior or more specialized one, rather than waiting indefinitely at the same desk.

Break-glass procedures. When something is going wrong faster than the normal escalation ladder can respond, a break-glass path contracts the action space to the most conservative allowed behavior, shortens every deadline, and makes logging mandatory rather than sampled — it trades throughput for safety on purpose, and only for the duration of the incident.

Progressive autonomy. A capability moves through shadow mode (recommendations logged and compared to human decisions, never acted on), then read-only, then limited-write with tight bounds, and only then autonomous operation within its earned scope. Promotion between stages is gated on measured agreement with human decisions during the prior stage, plus a period of demonstrated stability — never granted by default, and never granted for an entire system at once rather than per action class.

Approval Workflows in Practice

Figure 6.5 — A proposed action enters a durable pending_approval checkpoint and leaves it only through approval, rejection, edit, timeout, or escalation — never silently. Every transition is written to the audit log, and the pre-checkpoint side effects must be idempotent so a retried resume cannot double-execute.

The oldest version of this pattern is maker-checker — also called four-eyes review — inherited from banking controls under Sarbanes-Oxley (2002) and Basel II: one party proposes a transaction, a second, independent party must approve it before it takes effect. In an AI system, the AI is the maker and the human is the checker. This inheritance comes with a caveat worth stating plainly: maker-checker's safety property depends on the checker's independence from the maker, and automation bias is precisely a force that correlates the checker's judgment with the maker's — the human stops being an independent check and starts being a second vote for the same answer. Naive maker-checker, with an AI maker and an under-supported human checker, gets weaker redundancy than the two-human version it is modeled on, not equivalent redundancy.

Concretely, this is implemented today with a pattern like LangGraph's interrupt() function, which pauses a running graph at a durable checkpoint and persists its state, and a corresponding Command(resume=...) call that a human's approval, rejection, or edit resumes the graph with — asynchronously, so the approval does not have to happen in the same request/response cycle the AI's proposal was generated in. Two implementation details matter more than they look: the workflow should support review-and-edit, not just approve/reject, so a reviewer can correct a mostly-right proposal instead of being forced to reject a usable draft outright; and any side effects that occur before the interrupt must be idempotent, because a retried resume — after a crash, a timeout, or a duplicate approval click — must not re-execute a partial write or double-post the audit record.

The routing decisions this section describes are not ad hoc per request; they are configured once, tying Chapter 5's guardrail verdict directly to this chapter's human handoff:

# approval-routing: claims-and-payouts   (loaded by the same AI gateway as the Ch5 guardrail policy)
metadata:
  service:   claims-intake
  owner:     ai-platform-team
  version:   1.4.0

routing:
  refund:
    confidence_min:        0.90        # below this, always route to a human
    dual_approval_above:   5000        # USD — triggers four-eyes, not single review
    timeout_s:             3600
    on_timeout:            escalate    # never on_timeout: approve
    reviewer_role:         claims-adjuster

  prior_auth_approval:
    confidence_min:        0.85
    always_escalate_on:    [denial, low_confidence]   # denials never auto-clear
    timeout_s:             14400
    on_timeout:            escalate
    reviewer_role:         clinician

  underwriting_quote:
    confidence_min:        0.80
    dual_approval_above:   250000
    autonomous_bind:       false        # quote only — never bound without a human
    timeout_s:             28800
    on_timeout:            escalate
    reviewer_role:         underwriter

  biometric_identification:
    confidence_min:        1.0          # effectively: never auto-execute
    dual_approval_above:   0            # every case requires two reviewers
    reviewer_role:         [compliance-officer, second-reviewer]
    timeout_s:             1800
    on_timeout:            block        # fail-safe default on the highest-risk class

defaults:
  on_timeout:        escalate
  fail_mode:         fail_safe          # never fail_open above low risk tier
  log:               all_transitions
  idempotency_key:   required

Notice the asymmetry built into every row: nothing defaults to auto-approving on timeout, and the highest-risk class fails closed by blocking rather than by proceeding. That is the same fail-closed discipline Chapter 5 established for guardrails, applied now to the human handoff itself.

Metrics: Is the Oversight Real?

The broader shape of this measurement problem — treating human oversight as a runtime property to be instrumented and observed, not a one-time design decision — is discussed in recent work on layered AI runtime governance (arXiv 2606.12320) and in the growing academic literature on designing for meaningful human oversight rather than assuming it follows automatically from a review step being present.

Design Tradeoffs

Best Practices

• Route by confidence and risk together, not either alone — a high-confidence, high-risk action still deserves scrutiny.
• Never default an escalation timeout to auto-approve; default to escalate or block, and treat auto-approve-on-timeout as a design smell.
• Build review-and-edit into every approval workflow, not just approve/reject, so a mostly-right proposal is not forced into a binary.
• Show the evidence before, or alongside, the AI's recommendation — never let the recommendation anchor the reviewer's first impression.
• Treat shadow mode as mandatory groundwork, not optional polish, before any action class is promoted toward autonomy.
• Measure disagreement, reversal, and latency as first-class signals; a rising agreement rate alone is not evidence of a healthy system.
• Design for idempotency at every pre-approval side effect, so a retried resume can never double-execute.
• Reserve four-eyes review for the smallest, highest-stakes category of actions — it is expensive, and spreading it thin defeats its purpose.
• Rotate reviewers and vary their exposure to reduce correlated blind spots building up around a single AI's failure modes.
• Log every transition in the approval state machine to the same immutable trail Chapter 5 established for guardrail decisions.

Anti-Patterns

• Sign-off theater. A dashboard that reports "100% human oversight" while the reviewer has seconds per decision and no independent evidence to work from.
• Auto-approve on timeout. A queue that clears itself by defaulting to yes when no one responds in time — the fail-open failure mode applied to people instead of code.
• Approve/reject with no edit path. Forcing a reviewer to reject an otherwise-good proposal because the workflow has no way to let them correct it.
• Treating agreement as success. Reporting a rising override-agreement rate as a win without checking whether it reflects a better model or a disengaged reviewer.
• Granting autonomy by system, not by action class. Promoting an entire capability out of shadow mode at once instead of earning it action-by-action, on measured evidence.
• Correlated maker-checker. Assuming an AI-plus-human review has the same independence as a two-human review, when automation bias actively works against that assumption.
• No break-glass path. No contracted, faster-deciding mode for when something is visibly going wrong and the normal escalation ladder is too slow.

Maturity Model

Implementation Checklist

• Every action class has an assigned oversight mode (in-the-loop, on-the-loop, or override), set by cost of failure, not convenience.
• Escalation routing considers both model confidence and action risk, not either alone.
• Approval state is durable, survives restarts and timeouts, and supports asynchronous resume.
• Every pre-approval side effect is idempotent; a retried resume cannot double-execute.
• Timeouts default to escalate or block, never to silent auto-approval, above the lowest risk tier.
• Reviewers see independent evidence before or alongside the AI's recommendation, not only the recommendation.
• Review-and-edit is available, not just approve/reject.
• Four-eyes review is applied to the narrowest, highest-stakes category of actions, per Article 14(5)'s own model.
• New capabilities pass through shadow mode, then limited-write, before any autonomous execution is granted.
• Escalation rate, override rate, reversal rate, and approval latency are tracked and reviewed on a continuing basis.
• Every approval-workflow transition is written to the same immutable audit trail as guardrail decisions.

Standards and Further Reading

The obligations in this chapter are best read together rather than as separate homework: the EU AI Act, Article 14 for the human-oversight design requirement itself (with Article 14(5)'s two-person rule for biometric identification, and Article 86's related right to explanation); GDPR Article 22 for the individual's right to meaningful human intervention on solely-automated, significant decisions; the NIST AI RMF's GOVERN and MEASURE functions for the surrounding risk-management structure; and the long institutional lineage of maker-checker controls under Sarbanes-Oxley (2002) and Basel II, which predates AI by two decades and supplies most of the pattern this chapter borrows. Sector-specific precedent is also instructive: the FDA's good-machine-learning-practice guidance for adaptive medical AI, and the CIOMS Working Group XIV framework for risk-based pharmacovigilance oversight, both formalize the same idea — scrutiny scaled to consequence, applied continuously rather than once at approval time.

Chapter Summary

• Human approval is the designed handoff from deterministic governance to human judgment; the three modes are set by the cost of failure, not by preference.
• Meaningful human control, per Article 14, is five capabilities — understand, resist bias, interpret, override, and stop — not an Approve button.
• Automation bias is the default failure mode, and it worsens as models improve; engineer oversight against it, or it degrades into a rubber stamp.
• Escalation is the architecture: confidence-and-risk routing, exception queues, SLAs and break-glass procedures, and shadow mode before any autonomy is granted.
• Measure oversight quality — override, reversal, latency, disagreement — and treat an override rate trending toward zero as an alarm, not a win.

Looking Ahead

Human approval decides who signs off, and under what architecture their signature means something. It does not say what rule they are signing off against — what data may reach the model, which model may be used for which purpose, what must be disclosed, and what is simply prohibited. Chapter 7 turns the judgments this chapter has been routing to people into written, enforceable policy: the rules that guardrails enforce and that human reviewers are ultimately checking against.

Guardrails (Ch 5) → Human Approval (Ch 6) → AI Policies (Ch 7): enforcement, judgment, and the rules behind both.