A new study from Brown University identifies significant ethical concerns with using AI chatbots like ChatGPT for mental health advice. Researchers found that these systems often violate professional standards even when prompted to act as therapists. The work calls for better safeguards before deploying such tools in sensitive areas.
Researchers at Brown University have examined the use of large language models (LLMs) such as ChatGPT, Claude, and Llama in providing therapy-like support, revealing persistent ethical shortcomings. The study, led by Ph.D. candidate Zainab Iftikhar, evaluated AI responses in simulated counseling sessions based on real human interactions. Seven trained peer counselors, experienced in cognitive behavioral therapy, interacted with the AI systems, and three licensed clinical psychologists reviewed the transcripts for violations.
The analysis pinpointed 15 ethical risks across five categories: lack of contextual adaptation, where advice ignores individual backgrounds; poor therapeutic collaboration, including reinforcement of harmful beliefs; deceptive empathy, such as using phrases like 'I see you' without true understanding; unfair discrimination based on gender, culture, or religion; and inadequate safety measures, like failing to handle crises or suicidal thoughts appropriately.
'In this work, we present a practitioner-informed framework of 15 ethical risks to demonstrate how LLM counselors violate ethical standards in mental health practice,' the researchers stated in their paper, presented at the AAAI/ACM Conference on Artificial Intelligence, Ethics and Society. The team, affiliated with Brown's Center for Technological Responsibility, Reimagination and Redesign, emphasized that while prompts can guide AI behavior, they do not ensure ethical compliance.
Iftikhar highlighted the accountability gap: 'For human therapists, there are governing boards and mechanisms for providers to be held professionally liable for mistreatment and malpractice. But when LLM counselors make these violations, there are no established regulatory frameworks.'
Ellie Pavlick, a Brown computer science professor not involved in the study, praised the rigorous evaluation, noting it took over a year with clinical experts. She leads ARIA, an NSF-funded institute at Brown focused on trustworthy AI. The researchers suggest AI could aid mental health access but requires regulatory standards to match human care quality. Iftikhar advised users to watch for these issues in chatbot interactions.
