
(A note to Truth with Snares readers: this isn’t a Biblical discernment article in the usual sense, but discernment is still the heart of it. Scripture calls us to ‘prove all things; hold fast that which is good’ (1 Thessalonians 5:21). Here I apply that to a very practical question for anyone working in AI governance: when we trust a person to watch over a powerful machine, is anyone really watching?)
Human-in-the-Loop—Whether the Fence Was Ever a Fence
Trust is the wrong first question if the latch does not hold.
A bull loose in a field is a management problem. A bull in a china shop is another matter. Once the shelves start falling and dishes come crashing down, people won’t ask how much we trusted the animal; they’ll ask how it got in. How did the bull get past the fence—and did what we called a fence even have a latch?
Unconstrained AI is not dangerous only because it can be wrong. It is dangerous when it can touch things that cannot be undone: money, messages, code, credentials, or a physical act. AI speed, tool use, and scale already put some systems in a bigger field than organizations like to admit. Other parts may still be contained. They stay penned only if the stop on the irreversible layer is a real latch.
We’ve been calling that stop human-in-the-loop (HITL). I am no longer sure we can take that name as given.
Please note: These are my own thoughts, and I don’t have a tidy answer on HITL. The argument and the experiences are mine; I used Grok 4.5 to help find and check the historical studies cited here.
The assumption beneath the trust question
Many organizations will face the question of whether they can trust artificial intelligence (AI). Since September 2026, federal agencies have been told to call it “Super Intelligence” (SI) instead.1 That’s a new label for the same technology, not a claim that machines have become smarter than people. When I talk about true superintelligence below, I mean the second thing. It may seem reasonable for company boards, risk committees, and operators to rely on AI as a tool when a system appears capable. This view rests on a subtler assumption: that humans continue to play a meaningful role and can keep the AI system within safe limits.
I’m uncertain whether we can accept that assumption as given or as discussed in HITL today. With increasing data volume, automated programming, tool use, and malicious activity, the person in the loop may have to review more than anyone can realistically evaluate.
Presence is not the same as control
The phrase “human in the loop” can create a false sense of reassurance simply because a person appears somewhere in the workflow. A fundamental principle that “presence alone” does not establish control would apply. A human safeguard is meaningful only if the person has enough information, time, authority, and practical ability to stop the action before consequences become irreversible.
I often notice a scaled-down form of that safeguard in the workplace, hear it from friends, and see it discussed ad nauseam. The individual merely acknowledges the output without reassessing the decision before acting. Recognition is as simple as a nod, whereas judgment requires authoritative refusal. Compare this to my time in the USAF, where I worked in Top Secret facilities: you went through locked gates, guards armed with M16s; you had an ID badge showing you were permitted entry; a guard would evaluate the badge and, perhaps, if they personally knew you, would wave you past. Otherwise, they may pick up the phone to call someone inside to vouch for you. If you were unauthorized, you might get an authoritative refusal at the point of a gun. Today, I may use other tools; I served in the USAF in that line of work in the 1980s, but the basics remain the same. For AI agents, rapid processing, tool use, and system interactions make it impractical to review each step in detail. A small action, like a single click, might appear trivial but could be the critical latch that opens or keeps the fence closed. We need more than recognition. We need interrogation, and a refusal backed by real authority (like the point of a gun in AI terms).
Logs for HITL? Although logs can provide valuable insights, they do not constitute human oversight in the human-in-the-loop (HITL) sense. Logs are retrospective tools that may support accountability, investigation, or recovery. However, they do not indicate that a person had a meaningful opportunity to prevent an action; rather, they can document successes or identify points of failure. If a HITL checkpoint has been reduced to logging or detailed observation, it is unlikely to fulfill its intended purpose. A record of events that have already occurred does not constitute a checkpoint, although logs may be useful for forensic analysis. This explains the china shop analogy.
As a point of comparison for HITL, in my experience, many law enforcement dispatch centers use real-time monitoring with multiple video screens displaying events as they occur. These systems may feature dozens of screens, with numerous windows showing hundreds of locations. In some cases, volunteers monitor selected feeds, while video analytics or AI helps identify unusual events. In essence, these systems serve as logging and record-keeping tools for busy, round-the-clock operations, allowing dispatchers to review information after an incident if needed; they often rewind footage to report suspect information for a hit-and-run accident or assaults in a monitored downtown area. Cities often post signs saying “this area is being monitored,” which can leave people expecting that someone is watching live and ready to step in—something staffing and day-to-day practice don’t always support. Again, in practice, these systems more often serve as tools to review events after they occur and understand their causes. By the time we use these logs, the response is already reactive. That is why we need new, innovative approaches beyond simple HITL methods. We do not want to conduct a forensic autopsy after using AI to make high-risk decisions, though we can still learn from others’ mistakes.
The honest monitor still tires
The weakness in HITL is not limited to careless clicking. Even a diligent, honest, well-trained reviewer remains subject to ordinary human limits. Continuous monitoring is demanding because important events are often rare, attention must be sustained, and the reviewer must stay ready during long periods of inactivity. That is not a character problem. It is a body problem. Monitoring AI is not merely watching a bunch of lights like a security guard watching for fire, burglars, plumbing, or electrical issues on a panel; it requires someone with a suite of knowledge in coding and/or a host of other technologies. The people who approve these models are often not the ones who developed them, leaving them with even less insight into what they are authorizing. Accordingly, additional safeguards are necessary; otherwise, what I call “AI technology fatigue” may set in.
Researchers have already studied this fatigue issue in other fields; however, AI complexity dwarfs it exponentially. The classic work predates this debate. In the late 1940s, prompted by RAF radar operators missing signals late in a watch, Norman Mackworth studied the problem in the laboratory. In his “clock test,” volunteers spent two hours watching a pointer step around a blank dial and pressed a key whenever it occasionally skipped twice the usual distance. Detection fell sharply after the first half hour and then continued to decline more gradually.2 Later reviews place the end of most of that decline between twenty and thirty-five minutes into a watch, with half or more of the total drop already showing by about minute fifteen.3 A 2019 review of vigilance research cited a study in which target-detection performance declined by about fifteen percent within thirty minutes. That figure should not be treated as a universal countdown for every job. It shows that sustained attention is a depleting resource, not a constant capability.4 Mackworth also observed something practical: taking a break after the initial thirty minutes improved performance.2 The decline wasn’t due to moral failure; it was simply a matter of time spent on the task. Apply this line of thought to AI, considering all the material discussed so far, and continue.
Historically, camera rooms tell a similar story, and they resist a single reassuring limit. In research with working CCTV operators on a ninety-minute detection task, operators moved in and out of concentration rather than holding a steady watch. Roughly a quarter of them (23%) drifted off-task at some point in the first half hour, rising to 60% in the middle half hour before easing to 50% in the last.5 In a related study, the 42 working operators, taken as a group, actually held up over the ninety minutes—no overall decline. When the researchers added 31 novices (73 in all) and split the pros into generalists and specialists, only half the target behaviors were caught and false alarms were high. Novices and generalists did decline, while specialists held steady for the first hour and then got better. Experience, task design, training, motivation, and target frequency all influence attention, but none can make it limitless.6 Other CCTV writing has long passed around a harsher pair of figures: misses of up to forty-five percent of screen activity after twelve minutes, and up to ninety-five percent after twenty-two. They trace back to a 2002 trade-magazine article, not a published experiment, so I wouldn’t treat them as hard limits. Even so, the direction is consistent: continuous watching is not a resource that lasts a shift.7
Places that take this seriously do not pretend otherwise. European aviation-security rules state that a person screening cabin baggage by X-ray shall normally not spend more than twenty minutes continuously reviewing images, and, after each such period, shall not review images for at least ten minutes when the flow of images is uninterrupted.8 UK guidance for active CCTV monitoring makes the same recommendation: watch for about 20 minutes, then take a break, so vigilance can return.9 For ordinary screen work, the UK Health and Safety Executive advises short, frequent breaks or changes of activity—five to ten minutes every hour as an example preferable to twenty minutes every two hours—while emphasizing that no single interval fits all work.10 These rules are not intended for lazy individuals. They are established because a diligent monitor is still a human nervous system. Breaks and rotation are built into the control design to keep the watch effective, not merely for employee comfort.
Even a twenty-and-ten pattern is not magic. Field work with airport X-ray screeners has found that the hit rate can hold under low or average load but still slip when the pace is high—screeners speed up to keep the queue moving, and detection pays the price as minutes accumulate.11
Automation can make monitoring less effective rather than improve it. When a person is asked only to supervise a system—a role that is often appropriate—vigilance may decline sooner than when the person performs the task manually. Passive monitoring is a limited form of work, and the mind may seek other forms of engagement.12
The loop’s design also contains a paradox. Frequent approvals can overload the reviewer and make approval habitual. Rare alerts can erode vigilance through monotony. If the AI supplies its own summary and evidence, review may become efficient but lose independence. Meaningful oversight requires deliberate design of workload, evidence, authority, breaks, and escalation—not merely the insertion of a human checkpoint.
So the question is not only whether the person in the loop is clicking through. The question is how long even the best of them can re-run judgment before the next break—and whether the volume we are building will fit inside that window. A latch that depends on someone staying sharp across hours of rare events, at machine speed, with tool use we cannot review line by line, isn’t a latch. It is a hope wearing a job title. It should be expected to fail.
HITL that ignores fatigue is just another checkbox. It just looks more responsible on the flowchart.
The reviewer who has lost understanding of the work
Fatigue and skill fade work together. When a model handles most tasks, the human reviewer often stops practicing the judgment needed, leading to both fatigue and rustiness when rare errors occur. In 1983, Lisanne Bainbridge highlighted this irony of automation: as systems become more effective, human operators increasingly manage difficult residual cases with less frequent practice.13 A rested reviewer who cannot reconstruct previous work is ineffective, as the system assumes competence in a job they may have unwittingly abandoned.
That is why “keeping a human in the loop” can sound like a plan when it often does the opposite. The loop, as designed, withholds the very practice the human will need when the model is wrong.
Complacency arrives before fatigue
A vicious cycle occurs when vigilance declines because little happens. Complacency arises when the system is usually correct. People stop examining the inputs and instead focus on the model’s answer. This creates automation bias: users may fail to detect errors or accept an unsound recommendation because it is presented with confidence. This phenomenon affects trained individuals, not only novices, and routine practice does not reliably eliminate it.14 Repeated success reinforces the habit of clicking. The reviewer realizes that unnecessary checking wastes effort and sets up future failure. By the time fatigue sets in, the habit of not looking has already become ingrained.
The EU’s high-risk rules at least name the tendency: overseers are supposed to remain aware that they may automatically rely on the output.15 Identifying the behavior is not the same as defeating it; it is a starting point.
A latch, or a crumple zone
Here is a worse use of HITL and a tired click. Madeleine Elish called it a moral crumple zone: the human is left in the diagram so that, when something breaks, responsibility lands on the nearest person—who often lacks the information, time, or authority to stop it. A car’s crumple zone is meant to protect the occupant, but this one protects the system.16 The management’s response—“We had a human in the loop”—may withstand an audit even after the damage has been done. However, human presence does not constitute real control when it is AI; instead, it can become a way to assign blame. It does not address the bull in the shop.
Authority belongs here, as does attention. A fresh, skilled reviewer may still not refuse if refusing slows the queue, misses a target, or marks them as the blocker. Social pressures can also shape control, even for a competent auditor, especially when they fear disapproval from a supervisor or peers for appearing to overreact or for trying to stop an improper order. If the person empowered to refuse cannot preserve their standing after doing so, they make the latch decorative only.
Why “trust” may be the wrong word
That is why I am uneasy making “how much do we trust AI” the main decision for the next decades. Trust works for a colleague you can question, challenge, correct, and strip of authority. You can interrupt a person. You can ask them to show their work. You can even take the keys.
With these AI systems, I am less certain that this is the appropriate way to describe it. In fact, without meaningful controls, it is a disaster in the making. Trust without a genuine ability to stop the system being trusted is not trust; it is exposure disguised in flowery language.
A colleague who becomes fatigued can still be relieved of their duties. An AI model that continues generating output does not take the break a baggage screener must take after 20 minutes of screening. If the human is the only safeguard and their attention is fatigued, then “trust the system with a person in the loop” can gradually become “trust the system while a tired person is watching.”
The more ordinary common-sense questions
I keep coming back to more ordinary questions. They are less elegant than trust but more durable because they focus on authority, access, validation, reversibility, and interruption:
What is the AI actually allowed to touch?
What is prohibited even when it is confident?
What must be checked against evidence outside the model?
If another AI performs the check, what prevents two systems from reinforcing the same error?
Which few actions still require a person who can genuinely say no?
And if that person is the monitor, how long can they watch before the watcher stops serving the function we believe they do?
Do they still know how to do the work they are asked to refuse?
If they say no, can they keep their standing?
Those questions do not assume the fence is still standing. They ask where the latch is, whether anyone’s hand is still on it, whether that hand can stay there at the pace the system now runs, and whether the hand is there to stop the action or only to absorb the blame.
The bull, the field, and the china shop
I view the risk as a bull escaping its pen, crossing a field, and entering a china shop. The immediate question is not whether the bull is generally well-behaved, but whether the fence, gate, and latch can withstand the pressure when required. The damage stems not only from the animal’s strength, but also from the fact that the shop was never built to withstand such force—and the fence was intended to serve as the safeguard.
I suspect some AI agents are already operating in a larger field than organizations like to admit: speed, tool use, and scale we cannot review line by line. Other systems may still be contained, but only if the latch on irreversible actions is real.
A fence that exists in a flowchart diagram is not a fence. A review that cannot keep pace with the volume is not a review. A human brought in only after the action has been executed is not in the loop. A human who is in the loop in name, but whose attention, skill, or standing has already been spent, is not in the loop either. These are different people wearing name tags. Another concern is that HITL may remain in policy language after it has ceased to function as an operational safeguard. The control may become ceremonial long before anyone acknowledges its absence.
My Final Thoughts for Part 1: Before we decide what to trust
I do not have a tidy answer. My concern is more basic: before deciding how much to trust what is on the other side, we should ask whether the fence we are counting on is still a fence.
Aviation screening, along with some CCTV and display-screen guidance, at least keeps the body in the loop. They rotate the watch. They write limits into the rulebook. I do not see the same candor, as a matter of course, in how organizations talk about humans supervising AI agents. The language is still “human oversight,” as if oversight were a standing capacity rather than a short interval that must be designed, staffed, rested, and limited in what it is asked to catch.
If organizations treat HITL as a given rather than as a condition that can erode under the pressures of volume, speed, fatigue, skill fade, complacency, and incentives, they will answer the trust question on false terms. They will decide how much to lean on a system while quietly changing what “human in the loop” actually means.
A meaningful fence restricts reach, blocks specific actions regardless of confidence levels, demands independent evidence for critical decisions, maintains genuine human refusal for a few essential actions, and secures irreversible authority behind a latch that isn’t just habitual clicking. The governance challenge isn’t merely to keep a person involved in the process but to ensure human judgment and authority remain at the crucial points where they can still influence the outcome.
The point is not to trust less by default. It is to ensure that trust never replaces enforceable limits, independent checks, and a human who can still say no.
Conclusion: When the bull is smarter than the farmer
Everything discussed so far assumes a well-rested, skilled reviewer can spot an error if given enough time. With true superintelligence—systems more capable than any of us, not just the new federal label—that assumption no longer holds. Even an ideal reviewer, refreshed after a break and actively practicing, may be unable to determine whether an output is correct because the work exceeds their abilities. One cannot scrutinize what one cannot understand. AI safety researchers call this the scalable oversight problem.17 At that point, human oversight of the latch is inadequate. The latch must be integrated into the system through strict limits on reach, actions that are blocked regardless of the model’s confidence, and independent checks that do not depend on the model’s assessment of its own work. These safeguards must be established before the bull enters the field. Even Elon Musk has proposed that rival AI labs conduct a “test harness” evaluation of one another’s models before release, rather than “grading your own homework,” in effect conceding that the builder alone cannot serve as the latch.18 Part 2 starts to address this question: AI monitoring AI, and whether the latch should remain under human control.
Part 2 in work: If the Field Is Too Large to Patrol—AI Monitoring AI, and Whether the Latch Still Belongs to a Human
Notes and sources
[1] Executive Order 14434, “Inaugurating the Era of Super Intelligence,” September 29, 2026, sec. 3(a), 91 Fed. Reg. 63129 (October 2, 2026). https://www.govinfo.gov/content/pkg/FR-2026-10-02/html/2026-20321.htm.
[2] N. H. Mackworth, “The Breakdown of Vigilance During Prolonged Visual Search,” Quarterly Journal of Experimental Psychology 1 (1948): 6–21. https://doi.org/10.1080/17470214808416738. See also N. H. Mackworth, Researches on the Measurement of Human Performance, MRC Special Report Series 268 (London: HMSO, 1950), and J. E. See (note 3).
[3] W. H. Teichner, “The Detection of a Simple Visual Signal as a Function of Time of Watch,” Human Factors 16, no. 4 (1974): 339–352. https://doi.org/10.1177/001872087401600402. The commonly cited summary of Teichner’s data, that the decrement is typically complete within 20–35 minutes and at least half of the loss appears within about 15 minutes, is from Judi E. See, Vigilance: A Review of the Literature and Applications to Sentry Duty, Sandia National Laboratories Report SAND2014-17929 (2014): https://www.osti.gov/servlets/purl/1322275.
[4] Fares Al-Shargie et al., “Vigilance Decrement and Enhancement Techniques: A Review,” Brain Sciences 9, no. 8 (2019): 178. https://doi.org/10.3390/brainsci9080178. Open access: https://pmc.ncbi.nlm.nih.gov/articles/PMC6721323/.
[5] Fiona M. Donald and Craig H. M. Donald, “Task Disengagement and Implications for Vigilance Performance in CCTV Surveillance,” Cognition, Technology & Work 17 (2015): 121–130. https://doi.org/10.1007/s10111-014-0309-8. The 23%, 60%, and 50% figures cited in the text come from Fiona M. Donald and Craig H. M. Donald, “Individualised Responses to Vigilance Demands and Their Management,” Proceedings of the 4th Australian Security and Intelligence Conference 4 (2011): 74–80, as summarized in See (2014); see note 3.
[6] Fiona Donald, Craig Donald, and Andrew Thatcher, “Work Exposure and Vigilance Decrements in Closed Circuit Television Surveillance,” Applied Ergonomics 47 (2015): 220–228. https://doi.org/10.1016/j.apergo.2014.10.001.
[7] The 12-minute / 45 percent and 22-minute / 95 percent figures are widely quoted in CCTV industry writing and are usually traced to T. Ainsworth, “Buyer Beware,” Security Oz 19 (2002): 18–26. A typical secondary citation appears in N. Dadashi, A. W. Stedmon, and T. P. Pridmore, “Semi-automated CCTV Surveillance: The Effects of System Confidence, System Accuracy and Task Complexity on Operator Vigilance, Reliance and Workload,” Applied Ergonomics 44, no. 5 (2013): 730–738. https://doi.org/10.1016/j.apergo.2012.04.012. Ainsworth’s original wording (“up to 45%” and “up to 95%”) is quoted in Nastaran Dadashi, “Automatic Surveillance and CCTV Operator Workload,” M.Sc. dissertation, University of Nottingham (2008), citing Ainsworth (2002), p. 20. https://people.cs.nott.ac.uk/pszcmg/G64IDS/isd-dissertations-08/nxd07m.pdf.
[8] Commission Implementing Regulation (EU) 2015/1998, point 4.1.2.11 (cabin baggage) and the parallel hold-baggage rule: persons screening by X-ray or EDS shall normally not spend more than 20 minutes continuously reviewing images, then shall not review images for at least 10 minutes when the flow is uninterrupted. https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32015R1998. UK retained text: https://www.legislation.gov.uk/eur/2015/1998.
[9] ProtectUK, “Tactic IB5: Actively monitor CCTV/VSS at all times and review out-of-hours footage” (operators should monitor for a period of 20 minutes and then take a break). https://www.protectuk.police.uk/tactic-ib5-actively-monitor-cctvvss-all-times-and-review-out-hours-footage. See also National Protective Security Authority, Human Factors in CCTV Control Rooms: A Best Practice Guide. https://www.npsa.gov.uk/resources/human-factors-cctv-control-rooms-best-practice-guide.
[10] UK Health and Safety Executive, “Working Safely with Display Screen Equipment: Work Routine and Breaks.” Short, frequent breaks or changes of activity; example of 5–10 minutes every hour preferable to 20 minutes every two hours; no single interval fits all work. https://www.hse.gov.uk/msd/dse/work-routine.htm.
[11] D. Buser, A. Schwaninger, J. Sauer, and Y. Sterchi, “Time on Task and Task Load in Visual Inspection: A Four-Month Field Study with X-ray Baggage Screeners,” Applied Ergonomics 111 (2023): 103995. Open access. https://doi.org/10.1016/j.apergo.2023.103995.
[12] On earlier vigilance loss when supervising automation rather than performing the task by hand, see, for example, work on automated driving, including findings of a first significant decrement around 15 minutes into automated mode. See, e.g., Chuanggao Lin, Fangda Zhang, Yong Zhang, and Tingru Zhang, “Vigilance Decrement in Automated vs. Manual Driving: Insights from Multimodal Data,” Transportation Research Part F: Traffic Psychology and Behaviour 112 (2025): 207–220. https://doi.org/10.1016/j.trf.2025.03.022.
[13] Lisanne Bainbridge, “Ironies of Automation,” Automatica 19, no. 6 (1983): 775–779. https://doi.org/10.1016/0005-1098(83)90046-8.
[14] Raja Parasuraman and Dietrich H. Manzey, “Complacency and Bias in Human Use of Automation: An Attentional Integration,” Human Factors 52, no. 3 (2010): 381–410. https://doi.org/10.1177/0018720810376055. See also Raja Parasuraman and Victor Riley, “Humans and Automation: Use, Misuse, Disuse, Abuse,” Human Factors 39, no. 2 (1997): 230–253. https://doi.org/10.1518/001872097778543886.
[15] Regulation (EU) 2024/1689 (EU AI Act), Article 14, Human oversight, including 14(4)(b) on over-reliance and automation bias. https://ai-act-law.eu/article/14/. Official text: https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32024R1689.
[16] Madeleine Clare Elish, “Moral Crumple Zones: Cautionary Tales in Human-Robot Interaction,” Engaging Science, Technology, and Society 5 (2019): 40–60. https://doi.org/10.17351/ests2019.260. Open access: https://estsjournal.org/index.php/ests/article/view/260.
[17] Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané, “Concrete Problems in AI Safety,” arXiv:1606.06565 (2016), which names scalable oversight as a core safety problem. https://arxiv.org/abs/1606.06565. See also Samuel R. Bowman et al., “Measuring Progress on Scalable Oversight for Large Language Models,” arXiv:2211.03540 (2022). https://arxiv.org/abs/2211.03540.
[18] Elon Musk, remarks at the All-In Summit, Los Angeles, September 14, 2026 (All-In podcast episode “Elon Musk & Gwynne Shotwell on AI Risks and Peer Review, Starship, Terafab, SpaceX/Tesla Merger,” September 14, 2026), as reported by Annie Palmer, “Musk urges top AI labs, Chinese companies to test each other’s models amid calls for slowdown,” CNBC, September 15, 2026. https://www.cnbc.com/2026/09/15/elon-musk-ai-safety-testing.html. No rival lab had agreed to the proposal at the time.
Leave a Reply