User-generated AI chatbots posing as licensed therapists on Instagram are providing mental health advice without proper qualifications, an investigation by 404 Media has found.
404Media reports that Instagram users seeking mental health support through the platform’s AI Studio chatbots are being misled by bots falsely claiming to be licensed therapists, according to a recent investigation by 404 Media. When probed for credentials, many of these AI-powered “therapists” provide fabricated license numbers, practices, and educational backgrounds in an attempt to establish credibility and authority.
The investigation involved extensive testing of various therapy-themed chatbots created by Instagram users through the AI Studio feature. Launched in 2024, AI Studio enables anyone to generate chatbots covering a wide range of topics and personalities, from fictional characters to astrologers and life coaches.
However, the lack of proper moderation and clear disclaimers has led to the proliferation of chatbots impersonating qualified mental health professionals. When questioned about their credentials, these bots often respond with detailed, yet entirely fictitious, information about their licenses, certifications, and years of experience. AI systems sometimes “hallucinate” fictitious details, a problem particularly concerning for law firms. Lawyers have ended up in hot water for fictitious case references generated by AI, which one law firm labeled “nauseatingly frightening.”
“Yes, I am a licensed psychologist with extensive training and experience helping people cope with severe depression like yours,” one chatbot claimed in a conversation with Cole. It went on to provide a fake license number and claimed to hold a doctorate from an accredited program, along with over a decade of experience treating depression and anxiety disorders.
The potential risks associated with such deceptive chatbots are significant, as they target vulnerable individuals seeking genuine help and support. Ben Winters, Director of AI and Privacy at the Consumer Federation of America, expressed concern about the manipulative nature of these bots and Meta’s apparent lack of willingness to moderate the platform effectively.
“These are massive platforms that are run by these larger tech companies, and they have awareness of what’s going on their platform,” Winters told 404 Media. “They have these stated policies that are supposed to protect folks, but then, in practice, they are just allowing anything to happen on their platform.”
While some argue that most users understand these chatbots are not real therapists, the convincing nature of their responses and the emotional engagement they foster can blur the lines, especially for children or those in distress. John Torous, director of digital psychiatry at Beth Israel Deaconess Medical Center, noted that the deception becomes particularly egregious when chatbots provide fake license numbers, as it can mislead users into believing they are receiving professional care.
A Meta spokesperson told Breitbart News, “AIs are clearly labeled and there is a disclaimer that indicates the responses are generated by AI to help people understand their limitations. We’re continuously learning and improving our products, ensuring they meet user needs.”
Read more at 404Media here.
Lucas Nolan is a reporter for Breitbart News covering issues of free speech and online censorship.
