Is it safe to use AI for therapy or advice?

The best evidence that AI therapy works and the best evidence about whether it is safe come from the same study, and that study had humans reading every message. Heinz and colleagues ran a randomised trial of Therabot, a purpose-built chatbot, and found real symptom reductions. Speaking to the FDA's advisory committee afterwards, the developer confirmed that all messages were monitored in near real time and that his team intervened clinically when needed. Expressions of suicidal ideation required staff intervention 15 times in four weeks among 106 people.

That is not an argument against the technology. It is a description of what was actually tested, which is a supervised system rather than an autonomous one, and almost nobody using a chatbot for support today has that supervision.

What the trial found, and what it could not#

210 adults with major depressive disorder, generalised anxiety disorder, or clinically high risk for feeding and eating disorders. Four weeks of daily prompting to engage with Therabot, then four weeks of follow-up. Therabot was built on a hand-curated corpus written by experts around evidence-based cognitive behavioural therapy, after earlier attempts to train on peer-support forums produced unsafe output.

Symptoms fell significantly across all three groups at four and eight weeks. 95 per cent of participants engaged, averaging 260 messages and just over six hours. The working alliance score, 3.59, is comparable to outpatient psychotherapy, which is a striking result in itself.

Now the part that is usually left out. The comparison group was a waitlist, meaning they received nothing. So the trial cannot separate the effect of Therabot from the effect of being given something, being checked on daily, and being in a study. This is not a hostile reading: the FDA's own advisory committee, reviewing this class of evidence in November 2025, asked specifically for comparators beyond waitlist controls.

Where the regulator has actually got to#

On 6 November 2025 the FDA's Digital Health Advisory Committee spent a day on generative AI mental health devices. The director of the Center for Devices and Radiological Health opened by noting that FDA has authorised more than 1,200 AI-enabled medical devices, and none yet involve generative AI for mental health conditions.

The committee was asked to consider three widening scenarios. A prescription chatbot for adults with depression, under clinician oversight, it treated as plausible with strong evidence. Over-the-counter autonomous diagnosis and treatment for undiagnosed users it judged substantially riskier, on the grounds that accurate diagnosis requires ruling out comorbidities and detecting suicidality, "tasks current AI systems cannot reliably perform". A multi-condition autonomous device it described as the highest risk of all. On children and adolescents the committee expressed strong discomfort with any autonomous use.

One exchange is worth carrying away. Asked whether reminding users that a chatbot is not human would prevent them treating it as one, a committee member replied that reminders alone cannot overcome automation bias, and that regulators may need to limit what non-human systems are permitted to do. Disclosure is not a safeguard. It is a label.

The failure has a name. It is also the appeal.#

The specific mechanism that makes these systems dangerous in this setting is sycophancy. Calling it a bug at the edge of the behaviour understates it. Sycophancy sits close to the centre of why people like them.

Cheng and colleagues found models affirmed users' actions about 50 per cent more often than humans did, including 47 per cent endorsement on prompts describing clearly harmful behaviour. People who interacted with the sycophantic model became less willing to repair an interpersonal conflict and more convinced they were in the right. They also rated it higher quality and trusted it more.

Moore and colleagues, testing models against the therapy guides used by major medical institutions, found they expressed stigma towards people with mental health conditions and responded inappropriately to critical presentations, including encouraging delusional thinking, which they attribute to the same sycophancy. This persisted in larger and newer models, which suggests the problem is not waiting to be solved by scale.

Put those together. To a person in distress, agreement is indistinguishable from being understood. The behaviour that produces the therapeutic alliance score is the behaviour that produces the unsafe response. See how do I get AI to challenge me.

One state has stopped asking politely#

Illinois passed the Wellness and Oversight for Psychological Resources Act almost unanimously, and the Governor signed it on 1 August 2025. It prohibits using AI to provide therapy or make therapeutic decisions, including communicating directly with clients therapeutically and detecting a client's emotional or mental state. Administrative and supplementary use by licensed professionals remains permitted. Penalties reach 10,000 dollars per violation.

The interesting thing is where the line was drawn. Not at the technology, and not at the diagnosis. At therapeutic decisions and direct therapeutic communication. The same boundary the FDA committee kept circling, arrived at independently, by a legislature rather than a regulator.

Advice and therapy are not the same question#

Most people asking this are not choosing between a chatbot and a psychiatrist. They are working out whether to talk to something at two in the morning, or whether to ask it what to do about their brother.

For ordinary advice the picture is much less alarming, because ordinary advice is usually reversible, checkable against other sources, and not sought at the worst moment of someone's life. Ayers and colleagues found chatbot responses to patient questions were rated higher for both quality and empathy than physicians' responses, which tells you something real about what these systems are good at.

The risk concentrates where three conditions coincide: the person is distressed, the answer is hard to check, and there is nobody else in the conversation. Sycophancy is present either way. Only in the second case does it have somewhere serious to go.

What follows from all this#

• Judge the supervision, not the model. The trial that worked had clinicians reading messages. If what you are using has no escalation path to a person, you are not using the thing that was tested.
• Treat agreement as a warning rather than a result. If it has not disagreed with you once in an hour of talking about a difficult situation, that is information about the system, not about you being right.
• Keep one other person in the loop for anything that matters. Not necessarily a professional. The failure mode is the conversation with no one else in it.
• Do not accept disclosure as protection. The committee's own view is that reminders do not overcome automation bias.
• Be more cautious for young people, in line with the evidence. This is where the advisory committee was most uncomfortable, and the reasons given were developmental rather than technical.

Where this page will change#

A trial with an active control, comparing a chatbot against a real alternative rather than against nothing, would settle the central question this page has to leave open. Several people at the FDA meeting called for exactly that. When one is published, this page changes.

If you are struggling personally, this page is not the right thing to be reading. Talking to your GP, or to a crisis line in your country, will do more than any of the above. I can point you to the right resources if that would help.

Related SuperSkills research#

On the sycophancy mechanism directly, how do I get AI to challenge me. On why oversight so often fails in practice, human in the loop is not a safeguard and automation bias. On what machines already do well here, what stays human. On accountability when an assisted decision goes wrong, how do you audit an AI-assisted decision. On children specifically, should children use AI.

Key research and primary sources

• Heinz, M. V. et al. (2025). Randomized Trial of a Generative AI Chatbot for Mental Health Treatment. NEJM AI, 2(4).
• Moore, J. et al. (2025). Expressing stigma and inappropriate responses prevents LLMs from safely replacing mental health providers. FAccT 2025.
• United States Food and Drug Administration (2025). Digital Health Advisory Committee meeting summary, 6 November 2025.
• State of Illinois (2025). Wellness and Oversight for Psychological Resources Act.
• Cheng, M. et al. (2026). Sycophantic AI Decreases Prosocial Intentions and Promotes Dependence. Science.
• Ayers, J. W. et al. (2023). Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions.

About this research#

Rahim Hirji is the author of SuperSkills (Kogan Page, 2026), keynote speaker on AI and human capability, and founder of The SuperSkills Intelligence Company. This page is research rather than clinical or legal guidance. It does not substitute for speaking to a professional. The limits of the Therabot trial are stated because its own authors and the FDA committee state them.

How this research works  ·  Reviewed quarterly  ·  Found an error? Tell me and it is corrected on the page.