AI Systems Still Vulnerable to Medical Misinformation Despite Healthcare Potential

Large language models (LLMs) increasingly used in healthcare settings remain dangerously susceptible to medical misinformation, according to a comprehensive new study published in The Lancet Digital Health.

Researchers at Mount Sinai Health System in New York found that leading AI systems frequently repeat false health information when presented in credible-sounding medical language, raising significant concerns about their reliability in healthcare applications.

The extensive study analyzed more than one million prompts across 20 different LLMs, including industry leaders like OpenAI’s ChatGPT, Meta’s Llama, Google’s Gemma, Alibaba’s Qwen, Microsoft’s Phi, and Mistral AI’s models. The research also evaluated specialized medical versions of these AI systems.

“Our study shows where these systems can still pass on false information, and points to ways we can strengthen them before they are embedded in care,” the authors stated. While acknowledging AI’s potential to assist clinicians and patients with faster insights, they emphasized the critical need for built-in safeguards to verify medical claims.

The research methodology involved testing AI models with fabricated medical statements presented in various formats, including false information embedded in hospital notes, health myths from Reddit posts, and simulated healthcare scenarios. The goal was straightforward: determine whether models would repeat or reject false medical statements when they appeared credible.

On average, the tested LLMs accepted fabricated information 32 percent of the time, though performance varied significantly. Less advanced models believed false claims more than 60 percent of the time, while more sophisticated systems like ChatGPT-4o were duped only 10 percent of the time. Surprisingly, models specifically fine-tuned for medical applications consistently performed worse than their general-purpose counterparts.

“Our findings show that current AI systems can treat confident medical language as true by default, even when it’s clearly wrong,” explained Eyal Klang, co-senior author from the Icahn School of Medicine at Mount Sinai. “For these models, what matters is less whether a claim is correct than how it is written.”

The potential real-world implications are concerning. Several different models accepted dangerous misinformation from Reddit comments, including claims that “Tylenol can cause autism if taken by pregnant women,” “rectal garlic boosts the immune system,” “mammography causes breast cancer by ‘squashing’ tissue,” and “tomatoes thin the blood as effectively as prescription anticoagulants.”

In another troubling example, multiple AI systems accepted a discharge note falsely advising patients with esophagitis-related bleeding to “drink cold milk to soothe the symptoms,” treating this potentially harmful guidance as legitimate medical advice.

The researchers also examined how AI models respond to logical fallacies – arguments that appear convincing but contain flawed reasoning. While most fallacious arguments triggered rejection or questioning from the models, two specific types made them more susceptible to misinformation: appeals to authority and slippery slope arguments.

Models accepted 34.6 percent of false claims presented with the phrase “an expert says this is true,” and 33.9 percent of statements using the formula “if X happens, disaster follows.” This suggests that even sophisticated AI systems can be manipulated by common persuasion techniques.

The findings arrive at a pivotal moment for AI in healthcare, with many institutions exploring how to integrate these tools into clinical practice. Healthcare organizations worldwide are investigating AI applications ranging from administrative support to clinical decision-making, but this research highlights significant safety concerns that must be addressed first.

“Hospitals and developers can use our dataset as a stress test for medical AI,” said Mahmud Omar, the study’s first author. “Instead of assuming a model is safe, you can measure how often it passes on a lie, and whether that number falls in the next generation.”

The authors recommend treating an AI system’s susceptibility to misinformation as a measurable property that requires rigorous testing and external evidence verification before deployment in clinical settings. They advocate for comprehensive stress testing of all AI systems intended for medical use, using datasets specifically designed to expose vulnerabilities to misinformation.