When Clinical AI Hallucinates or Appears To: A Reflective Account of Human-AI Diagnostic Interaction in General Practice

Sun, Yuhao and Sun, Junsheng (2026) When Clinical AI Hallucinates or Appears To: A Reflective Account of Human-AI Diagnostic Interaction in General Practice. In: Interactive Health Conference (IH '26) :. ACM, pp. 1-6. (In Press)

Text (ih26-61)
ih26-61_1_.pdf - Published Version
Available under License Creative Commons Attribution.
Download (457kB)

Abstract

Large language models (LLMs) are increasingly used as informal decision support in clinical practice, yet strong benchmark performance does not directly translate to real-world work where information is incomplete, evolving, and distributed across heterogeneous records. We present this co-authored reflective case study of in-the-wild LLM use by a senior general practitioner across outpatient and inpatient settings. Analysing three clinical vignettes, we identify how hallucination-like breakdowns can arise from both factual errors and opaque evidence blending: the model synthesises claims across various record types without making provenance visible, leading grounded details to appear fabricated and speculative inferences to resemble chart facts. We show how disagreement triggers verification work, shifting cognitive load from clinical reasoning to auditing sources. We conclude with design implications for clinical LLM interfaces, including typed provenance links, separation of retrieved evidence from inference, dynamic case reconstitution, and workflows for productive disagreement.