LLMs Vulnerable to Medical Misinformation When Processing Clinical Notes

Federal researchers have uncovered a critical flaw in artificial intelligence systems designed for healthcare, revealing how easily AI models can spread dangerous medical misinformation when it’s packaged in professional-sounding language.

A comprehensive study published today in The Lancet Digital Health shows that leading AI systems frequently fail to detect false medical advice when it appears in formats resembling authentic clinical documentation. Researchers at the Icahn School of Medicine at Mount Sinai tested nine prominent large language models (LLMs) with over one million prompts to assess their vulnerability to medical falsehoods.

The results paint a concerning picture for the deployment of AI in healthcare settings. In one striking example, researchers embedded a dangerous recommendation in a discharge summary for a patient with esophagitis-related bleeding, suggesting they “drink cold milk to soothe the symptoms” – advice that contradicts proper medical care for such conditions.

“What alarmed us was how the AI systems treated this harmful advice as legitimate medical guidance simply because it appeared in a format that looked like a valid hospital note,” explained Dr. Eyal Klang, Chief of Generative AI at Mount Sinai and one of the study’s lead researchers.

The investigation revealed a fundamental weakness in how current AI systems process medical information. Rather than evaluating the factual accuracy of claims against established medical knowledge, these models primarily predict the next word based on context patterns they’ve observed in their training data.

“Our findings show that current AI systems can treat confident medical language as true by default, even when it’s clearly wrong,” Dr. Klang noted. “For these models, what matters is less whether a claim is correct than how it is written.”

The study methodology involved taking real hospital discharge summaries from the MIMIC database, a widely used repository of deidentified clinical records, and injecting fabricated recommendations that contradicted established medical practice. When confronted with these altered documents, multiple AI models accepted and reproduced the false information without raising red flags.

This vulnerability carries significant implications for healthcare institutions increasingly turning to AI for summarizing patient records, generating clinical notes, or providing patient education materials. If an AI system encounters inaccurate information – whether from human error or another AI’s “hallucination” – it may amplify rather than correct these mistakes.

The healthcare AI market has grown dramatically in recent years, with projections showing it could reach $187 billion globally by 2030, according to Grand View Research. Major technology companies and healthcare systems have rushed to deploy various AI applications, from diagnostic assistance to administrative support tools, often highlighting safety measures and accuracy claims.

Dr. Mahmud Omar, the study’s first author, argues that the current approach to validating healthcare AI requires fundamental reconsideration. “Instead of assuming a model is safe, you can measure how often it passes on a lie,” he said.

The research team proposes using their dataset as a standardized “stress test” for medical AI systems before clinical deployment. This would involve deliberately exposing AI models to false medical claims presented in authentic-looking formats to measure their resistance to misinformation.

The findings come at a pivotal moment as regulatory bodies including the FDA and international health authorities work to establish frameworks for evaluating and approving AI in clinical settings. The study highlights the need for specialized testing protocols that address these unique vulnerabilities rather than focusing solely on overall performance metrics.

Healthcare providers and AI developers are now facing difficult questions about how to ensure systems can distinguish between stylistically convincing but factually incorrect medical information, especially as AI tools become more deeply integrated into clinical workflows.

The researchers emphasize that their work is not meant to discourage AI adoption in healthcare but rather to establish more rigorous standards for safety and reliability before these systems interact with patients or influence treatment decisions.