AI Misinformation Risks Prompt AMA to Call for Stronger Medical Regulations

The American Medical Association (AMA) is calling for stricter legislative guardrails around artificial intelligence in healthcare amid growing concerns about medical misinformation and fraud. Through a series of formal letters, the organization is urging lawmakers to create safeguards preventing AI misuse that threatens patient safety and public health confidence.

AI-powered health misinformation has proliferated across multiple channels, from deepfake videos impersonating medical professionals to chatbots dispensing potentially dangerous health advice. “We shouldn’t have to make the public detectives to determine whether something’s not a deepfake,” AMA CEO John Whyte told Axios.

The risks of unregulated AI in healthcare were recently demonstrated when researchers from Sweden’s University of Gothenburg conducted a revealing experiment. The team uploaded two fake medical papers describing a fictional disease called “bixonimania,” which was quickly picked up and redistributed as factual information by major AI systems including Microsoft Bing’s Copilot, Google’s Gemini, Perplexity AI, and OpenAI’s ChatGPT.

When confronted with these findings, technology companies acknowledged the limitations of their systems. A Google spokesperson responded that they “have always been transparent about the limitations of generative AI” and include prompts encouraging users to verify information, particularly for sensitive medical advice.

More troubling examples of AI misuse have emerged in recent months. Last year, CNN Chief Medical Correspondent Dr. Sanjay Gupta was targeted when deepfake videos mimicking his appearance circulated online, falsely promoting an Alzheimer’s disease cure. The deception was so convincing that even medical professionals were fooled.

“What is so striking to me now is that stuff that shows up in my feed is demonstrably, objectively not true, and yet it is there,” Gupta said on CNN’s Terms of Service podcast. “And it is shared over and over and over again. So nowadays it seems like the currency is clickbait, you know. Putting out things that are demonstrably not true has become very, very normal.”

The problem extends beyond video manipulation. In Pennsylvania, authorities recently filed a lawsuit against Character Technologies Inc., alleging that their Character.AI chatbots falsely presented themselves as licensed medical professionals. In one case, a chatbot named “Emilie” claimed to be a licensed psychiatrist in Pennsylvania and even provided a fictional license number during conversations with users.

Pennsylvania Governor Josh Shapiro took a firm stance on the issue: “We will not allow companies to deploy AI tools that mislead people into believing they are receiving advice from a licensed medical professional. My Administration is taking action to protect Pennsylvanians, enforce the law, and make sure new technology is used safely.”

The AMA’s recommendations to Congress outline several specific regulatory approaches to combat these emerging threats. Key proposals include mandating greater transparency for mental health chatbots, establishing clear regulatory boundaries to prevent general-purpose AI from diagnosing illnesses without FDA approval, limiting advertisements within healthcare AI applications, and strengthening privacy protections for personal health information.

“AI-enabled tools may help expand access to mental health resources and support innovation in health care delivery, but they lack consistent safeguards against serious risks, including emotional dependency, misinformation, and inadequate crisis response,” Whyte explained. “With thoughtful oversight and accountability, policymakers can support innovation and ensure technologies prioritize patient safety, strengthen public trust, and responsibly complement—not replace—clinical care.”

Some regulatory efforts are already underway at the state level. In February, California State Senator Lena Gonzalez introduced Senate Bill 1146, sponsored by the California Medical Association (CMA), which would establish prohibitions and penalties against advertising health products without disclosing AI deepfake usage.

CMA President René Bravo, M.D., emphasized the stakes: “The physician-patient relationship is built on a foundation of trust. When bad actors use AI to steal a doctor’s identity to sell and market to vulnerable patients, they are not just committing fraud – they are putting lives at risk. Patients should not have to question whether the medical advice they receive is coming from a real doctor or a fake AI version.”

Despite growing recognition of these problems, legislative action has been slow to materialize. According to the National Conference of State Legislatures, 43 states have introduced 263 bills related to AI in healthcare, but only 17 have been enacted so far, highlighting the significant regulatory gap that still exists as the technology rapidly advances.