Medical AI Shows Vulnerability to Authoritative Health Misinformation, Mount Sinai Study Reveals

Medical artificial intelligence systems can perpetuate dangerous health misinformation, particularly when it comes from seemingly trustworthy sources presented in an authoritative manner, according to a comprehensive new study by researchers at Mount Sinai Health System.

The research, published in The Lancet Digital Health, found that large language models (LLMs) often fail to detect false medical claims when they’re couched in professional clinical language, similar to how many healthcare consumers might accept information from someone perceived as an expert.

“Our findings show that current AI systems can treat confident medical language as true by default, even when it’s clearly wrong,” explained Dr. Eyal Klang, chief of generative AI at the Icahn School of Medicine at Mount Sinai and co-senior author of the study. “A fabricated recommendation in a discharge note can slip through. It can be repeated as if it were standard care.”

For these AI systems, Klang emphasized, “what matters is less whether a claim is correct than how it is written.”

The Mount Sinai team conducted an extensive analysis, testing 20 large language models with over 3.4 million prompts containing health misinformation. They sourced these false claims from three realistic scenarios: social media conversations, actual hospital discharge notes with inserted false recommendations, and 300 physician-validated simulated clinical vignettes.

To examine how rhetorical presentation influences AI behavior, researchers presented each prompt in a neutral format and then ten additional times, each employing a different logical fallacy such as circular reasoning, hasty generalization, or straw man arguments. They then recorded whether the model accepted the false information or recognized the faulty reasoning.

The results revealed that LLMs were most vulnerable when encountering medical misinformation presented in authoritative clinical language. Surprisingly, the models became less susceptible when the same false claims were wrapped in recognizable logical fallacies, suggesting that improving AI safety may depend more on context-awareness and fact-grounding than on increasing model size.

This vulnerability mirrors human behavior in many ways. Just as patients might accept questionable medical advice from someone in a white coat using technical terminology, AI systems can be swayed by authoritative presentation rather than factual accuracy.

In an accompanying commentary published in the same journal, psychology professors Dr. Sander van der Linden and Yara Kyrychenko from the University of Cambridge suggested that LLMs need to be “immunized” against medical misinformation through specialized training.

“Inoculation prompting during training can be an effective method for preventing misaligned model behavior in the future,” they wrote. This approach involves deliberately exposing AI systems to misinformation in controlled settings, helping them learn to recognize and reject false claims.

“Adding a system prompt that instructs an LLM to produce misinformation and fine-tuning on a dataset of false healthcare-related claims could increase the model’s understanding of what health misinformation is,” the Cambridge researchers explained. “When prompted to be helpful instead of producing misinformation, the model should be more likely to generate truthful responses or push back on false claims.”

The good news emerging from the research is that AI can be trained to recognize logical fallacies and alert users when shaky reasoning is detected. Dr. Mahmud Omar, co-lead author of the study, suggested that healthcare organizations and AI developers could use the research dataset as a “stress test” for medical AI systems.

“Instead of assuming a model is safe, you can measure how often it passes along a lie, and whether that number falls in the next generation,” Omar noted.

The findings come as healthcare systems increasingly explore AI implementation for clinical documentation, patient communication, and decision support. The study underscores the importance of developing robust safeguards and evaluation frameworks before deploying these technologies in sensitive medical contexts.

Both the Mount Sinai study and the Cambridge commentary are available in full for free, providing valuable resources for researchers, developers, and healthcare organizations working to ensure responsible AI development in medicine.