AI in Healthcare: Navigating Opportunities, Risks, and Regulatory Challenges

Artificial intelligence continues to transform healthcare with groundbreaking possibilities in diagnosis, patient care, and operational efficiency. However, as healthcare providers and life sciences companies rapidly integrate these technologies, they face a complex landscape where potential benefits are matched by significant legal and regulatory challenges.

Industry experts point to the significant advancements in predictive analytics, machine learning, and ambient AI products that have bolstered AI adoption throughout the healthcare sector. From AI-assisted surgery to real-time diagnostic tools, these technologies demonstrate promising potential for enhancing patient outcomes and reducing clinician burnout.

Yet improper implementation or inadequate monitoring of these technologies poses substantial risks. Beyond patient safety concerns, healthcare entities could face liability under the False Claims Act (FCA) and other federal and state regulations if AI systems are not properly managed.

Diagnostic accuracy remains a primary concern with AI tools. While innovative, AI-powered diagnostic systems are inherently probabilistic and should not replace clinical judgment. Without continuous human validation and monitoring, these tools may generate erroneous outputs, particularly for rare or complex medical conditions, compromising reliability in clinical settings.

“Failure to monitor an AI tool can result in undetected errors or deviations from its intended function,” notes one industry compliance expert. “This could lead to potential delays in necessary critical interventions.”

A particularly troubling phenomenon involves AI “hallucinations,” where systems generate false or misleading outputs. If such hallucinations influence diagnoses or treatment recommendations, patients could face harmful interventions. Additionally, AI systems may experience performance degradation over time, creating both patient safety and regulatory compliance concerns if not properly maintained.

Data integrity issues represent another significant risk vector. Since AI relies heavily on datasets to function effectively, substandard or biased data can result in coding errors that jeopardize patient safety. For example, an ambient AI tool used to triage emergency patients might under-prioritize certain demographics due to inherent biases in its training data.

Privacy and security concerns have intensified as AI tools collect and process sensitive patient information. The rise in FCA government enforcement actions against contractors with inadequate cybersecurity measures highlights the importance of robust security protocols for healthcare organizations utilizing AI tools that handle patient data.

On the regulatory front, the landscape continues to evolve rapidly. In October 2023, the Biden administration issued Executive Order No. 14110 on “Safe, Secure, and Trustworthy Artificial Intelligence,” emphasizing the development of frameworks for safe AI deployment across healthcare. This order was recently revoked and replaced with a new directive prioritizing AI research and innovation.

State legislatures have also been active in addressing AI governance. California, Virginia, and Utah have enacted laws addressing AI transparency, bias mitigation, and accountability in healthcare and related sectors. California SB 1120, for example, requires healthcare service plans using AI for utilization review to implement safeguards related to equitable use and regulatory compliance, while explicitly mandating that medical necessity determinations be made only by licensed providers.

Recent enforcement actions underscore the heightened scrutiny facing AI applications in healthcare:

• The Department of Justice has subpoenaed pharmaceutical and digital health companies regarding their use of generative AI in electronic medical record systems, investigating whether these tools result in excessive or medically unnecessary care.
• FCA investigations have targeted Medicare Advantage plans using AI for diagnosis identification and coverage decisions.
• The Texas attorney general reached a settlement with a company that marketed a “highly accurate” generative AI tool for patient documentation, leading to allegations of deceptive claims.
• Commercial insurance companies face class action lawsuits for allegedly using AI algorithms to override physicians’ medical necessity determinations, with one major insurer sued for racial bias in an AI fraud prediction tool.

Healthcare and life sciences organizations are increasingly implementing specialized AI compliance programs to mitigate these risks. Effective programs typically include a multidisciplinary AI governance committee, comprehensive written policies and procedures, employee training resources, and routine monitoring and auditing protocols.

As the integration of AI in healthcare continues to accelerate, organizations must balance innovation with compliance. Those that implement structured approaches to AI governance will be better positioned to harness the benefits of these technologies while minimizing legal and regulatory exposure in an increasingly complex landscape.