AMA Calls for New Laws to Combat Medical AI Misinformation

The American Medical Association (AMA) is sounding the alarm over artificial intelligence’s growing role in healthcare, calling for urgent legislative action to prevent potential harm to patients and medical professionals alike.

In a series of recently issued letters, the AMA highlighted concerning incidents where AI technologies and deepfakes have been weaponized to spread medical misinformation, fraudulently impersonate healthcare providers, and facilitate various schemes targeting the healthcare system, according to reports from The Jerusalem Post.

“We shouldn’t have to make the public detectives to determine whether something’s not a deepfake,” AMA CEO John Whyte stated in comments reported by Axios, underscoring the organization’s growing concern about the blurring line between authentic medical information and AI-generated content.

The AMA’s warning comes amid mounting evidence that even sophisticated AI systems from leading technology companies are susceptible to propagating medical falsehoods. The Jerusalem Post cited a particularly troubling experiment documented in Nature, where researchers from the University of Gothenburg deliberately created and uploaded two fabricated papers describing “bixonimania” – a completely fictional disease.

What followed demonstrated precisely why medical professionals are concerned. The fabricated condition was rapidly absorbed and disseminated as fact by multiple major AI systems, including Microsoft’s Copilot, Google’s Gemini, Perplexity, and OpenAI’s ChatGPT. None of these advanced systems successfully identified the condition as fictional, instead treating it as legitimate medical information.

This experiment highlights the dangerous potential for AI systems to amplify medical misinformation at unprecedented scale and speed. Unlike traditional medical misinformation, which typically requires human intervention to spread, AI systems can instantaneously incorporate and distribute fabricated medical content to millions of users seeking health guidance.

When contacted about these concerns, a Google spokesperson emphasized the company’s safety measures, pointing to in-app prompts that direct users toward qualified healthcare professionals for sensitive medical inquiries rather than relying solely on AI-generated advice.

The healthcare misinformation problem extends beyond just fictional diseases. The AMA has documented cases where deepfake technology has been used to create convincing but entirely fabricated videos of doctors providing dangerous medical advice or endorsing unproven treatments. In other instances, AI has been utilized to generate fake medical credentials or falsify medical records for insurance fraud.

Medical experts worry that without proper safeguards, these technologies could undermine patient trust in legitimate healthcare information and providers. The healthcare sector, which already contends with significant misinformation challenges, faces unique vulnerabilities when AI systems present fabricated medical content with the same confidence and appearance of authority as evidence-based medical guidance.

The AMA’s push for legislative intervention represents a significant shift in how the medical community is approaching AI. While many healthcare organizations initially embraced AI’s potential to improve diagnostic capabilities, streamline administrative tasks, and enhance patient care, there is growing recognition that the technology’s rapid advancement has outpaced regulatory frameworks.

The association is advocating for several specific measures, including mandated disclosure when AI is used in patient interactions, verification requirements for medical content generated by AI systems, and legal accountability for companies whose AI platforms spread harmful medical misinformation.

This development comes at a critical juncture when healthcare consumers increasingly turn to digital platforms for medical information. A recent survey found that over 60% of Americans have used online resources to self-diagnose medical conditions, making the accuracy and reliability of AI-generated healthcare content a public health concern.

As AI continues to permeate healthcare systems globally, the AMA’s warning serves as a reminder that technological innovation in medicine must be balanced with rigorous safeguards to protect patient welfare and the integrity of medical information.