AMA Urges Congress to Set Safeguards for AI Mental Health Chatbots

The American Medical Association on April 22 urged Congress to put stronger safeguards on artificial intelligence chatbots used in mental health care, in letters to the co-chairs of the Congressional Artificial Intelligence Caucus, the Congressional Digital Health Caucus and the Senate Artificial Intelligence Caucus, according to the association's press release.

The AMA said the rapid rise of mental health chatbots, along with reports of risks such as encouraging self-harm and privacy breaches, shows the need for clear guardrails. It described the safeguards it proposed as a starting point rather than a limit.

"AI-enabled tools may help expand access to mental health resources and support innovation in health care delivery, but they lack consistent safeguards against serious risks, including emotional dependency, misinformation, and inadequate crisis response," said AMA chief executive John Whyte.

What the AMA asked for

The association's proposals fall into five areas, according to the release:

• Transparency: chatbots should clearly disclose that users are interacting with AI, not a human, and should be prohibited from presenting themselves as licensed clinicians, with federal regulators empowered to enforce the standards.
• Regulatory boundaries: chatbots should be barred from diagnosing or treating mental health conditions without appropriate regulatory review, and Congress should direct agencies to build a risk-based oversight framework and clarify when AI tools qualify as medical devices. Developers should be required to build crisis-detection systems that identify self-harm risk and refer users to help.
• Oversight: ongoing safety monitoring, adverse event reporting and stricter standards for tools used by children and adolescents.
• Commercial influence: advertising in mental health chatbots should be discouraged or prohibited, particularly for minors.
• Privacy and security: limits on data collection and retention, clear user consent and safeguards against unauthorized access.

The FDA's role

Healthcare Dive, which reviewed the letters, reported that the AMA said chatbots that do diagnose or treat mental health conditions should be reviewed by the Food and Drug Administration as medical devices. The letters also asked lawmakers to direct the FDA to clarify which AI tools count as general wellness technology and which should be subject to agency review, according to the outlet.

The AMA argued that current frameworks were not designed for generative AI tools that can move from casual conversation to therapeutic guidance within a single interaction, Healthcare Dive reported. On cybersecurity, the group wrote that "a single weakness in a data center can expose chatbot data and erode confidence."

Healthcare Dive noted that the Trump administration has largely taken a deregulatory approach to AI to speed adoption, while some states have passed their own laws. Illinois last year banned the use of AI for therapeutic decision-making, and California enacted a law requiring chatbot developers to monitor conversations for signs of suicidal ideation, the outlet reported.

The AMA said it would keep working with Congress on policies that promote innovation while putting patient safety first.