Nine questions to ask before you give a vendor your patients' data

Why the accuracy question is the wrong opener

Physicians are clear about what they need before adopting AI: 88% cite validated safety and efficacy, 86% cite data-privacy assurances, and clear liability frameworks rank as the top regulatory priority. (AMA, 2026) Those are the right instincts. But in a sales conversation, accuracy is the question a vendor is most prepared for and least able to prove. The questions below have answers you can check against a document.

1. Will you sign a business associate agreement — covering which specific services?

If a vendor creates, receives, maintains or transmits PHI on your behalf, it is a business associate. There is no serious argument otherwise for an AI system processing clinical encounters. HHS guidance on cloud services is explicit that even a provider who cannot decrypt the data it holds is still a business associate — "lacking an encryption key for the encrypted data it receives and maintains does not exempt a CSP from business associate status." (HHS OCR)

The trap is scope. Major model providers will sign BAAs, but each carves out specific products and endpoints. OpenAI offers a BAA for API healthcare customers but excludes certain products and states that web search with live internet access "is not HIPAA eligible and is not covered by a BAA." Anthropic covers the Claude API and Claude Enterprise but excludes the Console, Workbench and several other surfaces. Google's HIPAA covered-products list enumerates named services and is not a blanket. AWS lists Bedrock as HIPAA-eligible and requires a BAA before any PHI touches it.

So the question is not "do you have a BAA." It is "which of your subprocessors are covered, on which endpoints, and can I see that in writing."

2. Does anyone in the chain train on our data?

For paid API tiers the answer across the major providers is currently no, by default, and each says so in writing. That default is worth verifying rather than assuming, because there is at least one significant exception hiding in plain sight: Google's unpaid Gemini API tier states that Google uses submitted content and generated responses to develop its products, and that human reviewers may read it. The paid tier does not. (Google Gemini API terms)

A vendor that built on a free tier during development and never migrated is not a hypothetical.

3. How long is data retained, and by whom?

Retention is where "we don't train on it" stops being the whole answer. Abuse-monitoring logs are standard and typically run to 30 days. Zero-retention arrangements exist but come with conditions — Anthropic, for instance, states that HIPAA-covered models require 30-day retention and are not available with zero data retention enabled. Some grounding and search features retain for fixed windows that cannot be disabled.

Ask for the retention period at each hop: your vendor's own storage, the model provider's logs, and any third-party transcription or search service in between.

4. What happens to the source audio?

This one is specific to ambient documentation and it is not academic. The AP reported in October 2024 that a widely used transcription model hallucinates — inventing text that appears in no form in the underlying audio. The peer-reviewed study behind that coverage found roughly 1% of transcriptions contained entirely hallucinated phrases, and that 38% of those hallucinations included explicit harms such as invented violence or false authority. Hallucination was more frequent for speakers with longer non-vocal pauses — a symptom of aphasia. (Koenecke et al., FAccT 2024)

The compounding failure is deletion. The AP's reporting noted that one widely deployed ambient scribe erases the original audio for stated data-safety reasons — which means a disputed note can never be checked against what was actually said. (AP, October 2024) Retention and deletion are both defensible choices with different risks. You should know which one your vendor has made before an incident, not after.

5. Can a clinician see why the software said that?

This is the substance behind "human in the loop," and it is a design property rather than a policy. Every drafted statement should trace to its source in the encounter. If review means re-reading the whole note against memory, review will decay into acceptance — and the automation-bias evidence says that decay happens even to experts who have been warned about it.

6. Show me the source attributes

Certified health IT with predictive decision support has to make thirty-one source attributes available for review under 45 CFR 170.315(b)(11): intended use and population, the decision-making role the tool was designed for, cautioned out-of-scope uses, training data and its representativeness, fairness approach, external validation, quantitative performance, and the ongoing monitoring schedule. A proposed rule would scale this back; as of now it is required.

Even where a specific product falls outside the certification criterion, asking for those categories is a fair test. A vendor who cannot describe the population its model was validated on has told you something.

7. What is your policy when the model changes underneath you?

FDA's Digital Health Advisory Committee flagged exactly this in its generative-AI session: these systems are probabilistic rather than deterministic, and the underlying foundation models "will change over time." A vendor should be able to say what it pins, how it tests before moving, and what it tells customers when behaviour shifts.

8. Which disclosure laws apply to us, and does your product support them?

State law here is moving quickly and is genuinely operational.

• California AB 3030 (effective January 2025) requires a prominent disclaimer on generative-AI-produced patient communications about clinical information, plus instructions for reaching a human — with an exemption where a licensed provider has read and reviewed the communication. Note how neatly that exemption rewards a review-before-send design.
• Texas, under HB 149 effective January 2026, requires providers using an AI system in relation to health care service or treatment to disclose to the patient no later than the date the service is first provided. Texas SB 1188 additionally requires practitioners to review all AI-generated records and prohibits offshoring electronic medical records, including by vendors.
• Utah requires licensed occupations to prominently disclose GenAI interaction in high-risk contexts, with mental-health chatbot rules carrying penalties up to $2,500 per violation.
• Illinois restricts AI in therapy and psychotherapy outright, with civil penalties up to $10,000 per violation.

Two of those — the Utah chatbot rule and the Illinois therapy ban — are behavioural-health specific and will not reach a general ambulatory EMR. The California and Texas disclosure duties will.

Worth watching: Colorado repealed and replaced its 2024 AI Act in May 2026 with a narrower regime that exempts HIPAA covered entities from many of the developer and deployer obligations — and did so after a federal court in Colorado halted enforcement of the predecessor law in a constitutional challenge arguing its disclosure requirements were compelled speech. (Ropes & Gray, 2026) This area will not look the same in two years.

9. What does the audit trail actually record?

The last question is the one that matters in a dispute. Does the record show what the model proposed, what the clinician changed, and who approved it — as distinct entries, retained, and exportable? A system that stores only the final signed note has thrown away the evidence that the human was ever in the loop.