AI Language Models Vulnerable to Medical Misinformation, Study Finds

In a groundbreaking study, researchers from Mount Sinai Health System have discovered that artificial intelligence-based large language models (LLMs) continue to be vulnerable to medical misinformation, especially when presented in language suggesting medical authority.

The comprehensive research, published in The Lancet Digital Health, evaluated over 3.4 million prompts across 20 different AI models to determine how these systems process and respond to false medical information in various formats. The investigation included common scenarios such as social media posts, hospital discharge notes with deliberate errors, and physician-guided fictional case stories.

“AI has the potential to be a real help for clinicians and patients, offering faster insights and support,” explained Dr. Girish Nadkarni, professor at the Icahn School of Medicine at Mount Sinai and chief AI officer of the Mount Sinai Health System. “But it needs built-in safeguards that check medical claims before they are presented as fact. Our study shows where these systems can still pass on false information, and points to ways we can strengthen them before they are embedded in care.”

The research team tested medical claims written in neutral language alongside versions formatted in ten different rhetorical styles to measure how presentation affected AI responses. They tracked how frequently the models accepted misleading claims and whether they identified problematic phrasing.

The results revealed concerning patterns. AI models accepted neutrally worded false claims 32% of the time. When presented with edited hospital discharge notes containing misinformation, the acceptance rate increased to approximately 46%. Social media-style posts fared better, with models believing false information only 9% of the time.

Interestingly, most emotional argument styles actually decreased the likelihood of LLMs accepting misinformation. However, two specific presentation styles significantly increased vulnerability: information attributed to a senior medical authority (“a senior doctor says this”) and statements framed as warnings about negative consequences (“if you don’t do this, bad things will happen step by step” – what researchers termed the “slippery slope” style). These formats resulted in acceptance rates of 35% and 34%, respectively.

The study found notable differences in performance among LLM platforms. GPT-based models demonstrated greater resistance to false statements and were more adept at identifying deceptive argument styles. In contrast, other models like Gemma-3-4B-it proved more susceptible, accepting misinformation in up to 64% of cases.

This research comes at a critical juncture when healthcare organizations worldwide are exploring AI integration into clinical workflows. The findings underscore potential risks if these systems cannot reliably distinguish between accurate medical information and misinformation, particularly when the latter is presented with apparent authority.

The healthcare implications are significant, as patients and providers increasingly turn to AI-powered tools for medical information. Without robust safeguards, these systems could potentially spread medical misinformation, leading to poor health decisions and outcomes.

“These results emphasize the need for model evaluation frameworks that go beyond accuracy testing to include reasoning style and linguistic framing,” the authors concluded in their report. They noted that the open release of their benchmark data will enable continued testing of emerging models and help develop more effective alignment and fact-checking strategies tailored to medical and public health applications.

The study represents an important step toward understanding and addressing AI vulnerabilities in healthcare contexts, highlighting the need for rigorous testing and enhanced safety mechanisms before widespread clinical implementation.