What “human in the loop” actually means

The phrase has a legal meaning, and most people using it don't know it

In the United States, whether clinical decision support software is a regulated medical device turns on section 520(o)(1)(E) of the Federal Food, Drug, and Cosmetic Act, as interpreted in FDA's Clinical Decision Support Software guidance. FDA reissued that guidance twice in January 2026 — the operative version, issued on the 29th, supersedes one issued on the 6th, which in turn superseded the 2022 version that most vendor material still cites. (FDA, 2026)

Software escapes the device definition only if it meets all four criteria:

1. It is not intended to acquire, process, or analyze a medical image, a signal from an in vitro diagnostic device, or a pattern or signal from a signal acquisition system.
2. It is intended to display, analyze, or print medical information about a patient, or other medical information such as peer-reviewed studies and practice guidelines.
3. It is intended to support or provide recommendations to a health care professional about prevention, diagnosis, or treatment.
4. It is intended to enable that professional to independently review the basis for the recommendations, "so that it is not the intent that such health care professional rely primarily on any of such recommendations" to make a decision about a patient.

Criterion four is the one that carries the weight, and it is stricter than it sounds. It is not satisfied by a clinician clicking "accept." It requires that the software be designed so the clinician can inspect why the recommendation was made and reach an independent view. FDA states plainly that software intended for a critical, time-sensitive task does not meet criterion four, because the clinician will not have time to review the basis of the recommendation.

Why FDA wrote it that way: automation bias is real and measured

The guidance names the mechanism directly. In FDA's words, automation bias is "the propensity of humans to over-rely on a suggestion from an automated system," producing "errors of commission (following incorrect advice) or omission (failing to act because of not being prompted to do so)."

The underlying research is uncomfortable reading for anyone who assumes expertise is protective.

That study put 27 radiologists in front of 50 mammograms, showing correct AI BI-RADS suggestions first and incorrect ones later. Experience helped — but it did not come close to immunity. (Radiology, 2023)

A randomised vignette study of 457 hospitalists, nurse practitioners and physician assistants found the same shape. Standard AI raised diagnostic accuracy from 73.0% to 75.9%. Systematically biased AI dropped it to 61.7%. The finding that should concern every product team: adding model explanations did not rescue performance — the improvement was 2.3 points and not statistically significant. (JAMA, 2023)

And training does not fix it either. A 2026 randomised trial of physicians who had completed formal AI-literacy training found diagnostic reasoning scores falling from 84.9% to 73.3% when the model's advice was deliberately flawed. (NEJM AI, 2026)

What separates supervision from rubber-stamping

If review can be defeated this reliably, "a clinician reviews it" is not a safety architecture. The design has to make independent review cheap, because a review that costs more than accepting is a review that stops happening by Thursday afternoon. In practice that means four things.

Show the basis, not just the output

FDA's criterion-four recommendations ask for a plain-language description of the approach behind a recommendation, the input data it relied on, and the known limitations. At the level of an individual note, the practical version is provenance: every drafted statement should trace to the moment in the encounter that produced it. Verifying a sentence against the transcript takes seconds. Verifying it against memory takes minutes, and so does not happen.

Draft, never commit

The distinction that matters is whether the software's output is a proposal or an action. An order that appears in a queue awaiting signature and an order that has been placed are different products with different regulatory status, whatever the marketing says.

Surface what's missing

FDA specifically asks that output identify missing, corrupted or unexpected input values. A confident note built on a transcript with two inaudible minutes is more dangerous than an obviously incomplete one.

Don't design for speed at the gate

An interface that lets a clinician approve twelve items with one click has technically kept a human in the loop and functionally removed them. Batching approvals is where good intentions go to die.

Transparency is also a certification requirement

Separately from FDA, certified health IT carries its own obligations. The HTI-1 final rule created the Decision Support Interventions criterion at § 170.315(b)(11), which took effect for Certified EHR Technology on 1 January 2025. For predictive interventions it requires thirty-one "source attributes" to be available for review — intended use, intended patient population, the decision-making role the intervention was designed for, cautioned out-of-scope uses, training data and its representativeness, fairness approach, external validation, quantitative performance, and the schedule for ongoing monitoring. (45 CFR 170.315)

Worth knowing: a December 2025 proposed rule would scale much of this back, reporting that developers found supporting predictive interventions across the standardised data set "burdensome and costly to implement, with questionable value." As of this writing it remains proposed, and the source attributes remain required. (90 FR 60970) Our view is that the disclosure is worth making whether or not it is compelled — a practice buying clinical AI should be able to read what the model was built on without filing a regulatory complaint.

Our position

ChartVoyant is built to sit on the non-device side of that line deliberately, not incidentally. The software drafts; a clinician approves; high-stakes actions pause at an explicit gate; every drafted statement links back to the moment in the visit that produced it. We would rather be slower at the gate than defensible only in the marketing copy.

The automation-bias literature is the strongest argument an EMR vendor can make for its own restraint. It says, with unusual clarity, that the failure mode of clinical AI is not the model being wrong. It is the model being wrong while a competent clinician agrees with it.