Charting, coding, and triage all look like text problems. Until you read the case law. A view from the floor on where LLMs work in healthcare workflows, and where they should not be allowed near a patient.
Where they work
- Ambient documentation. Listening to the encounter and drafting the note for the clinician's review. The clinician edits, signs, owns. Massive time savings; clinicians love it; the legal posture is clean.
- Coding assist. Suggesting ICD-10/CPT codes from the documentation. The coder approves. Throughput up; coder satisfaction up; audit trail intact.
- Inbox triage. Drafting responses to non-urgent patient messages for clinician review. Frees up clinician time for the messages that need them.
- Prior authorization. Drafting the appeal letter from the chart and the denial reason. A surprisingly high-leverage application.
Where they break
- Autonomous triage. The cost of missing one acute case dwarfs the savings of triaging a thousand routine ones. The model goes in front of a clinician, not in front of the patient.
- Diagnosis. Even with a clinician in the loop, raw diagnosis is too high-stakes for current models without specialty-specific evaluation. Use it as differential generation, not as the answer.
- Medication orders. Hard no, in our practice. Even with HITL, the failure mode is too dangerous and the regulatory exposure is severe.
- Direct-to-patient symptom checkers without a clinician backstop. A liability problem disguised as a product.
What HIPAA actually demands
HIPAA does not say "no LLMs." It says: a Business Associate Agreement with any vendor that touches PHI, audit logs, access controls, encryption in transit and at rest, breach notification. Most major model providers will sign a BAA in 2026. Read it. Track which models are covered (not all are).
In healthcare AI, the question is never "can the model do this?" It is "who is the responsible clinician for this decision, and have they signed off?"
The clinical AI rollout pattern that works
Pilot with one specialty, one workflow, one site. Measure: time saved, error rate, clinician satisfaction. Iterate for a quarter. Then expand. The hospitals we have seen succeed move slowly and well; the ones that have failed went broad before they went deep.