A New Benchmark Finds AI Chatbots Cover Less in Severe Teen Mental-Health Cases

General-purpose AI chatbots were less complete when synthetic adolescent users described depression or severe symptoms than when they described anxiety or moderate symptoms, according to a peer-reviewed study published in Nature Mental Health on October 28.

The researchers developed the Adolescent Mental Health AI Response Tool, or AMHART, to audit chatbot replies against youth-relevant clinical guidance. They then applied it to multiturn conversations involving ChatGPT-5, ChatGPT-4, Claude 4.5 and Gemini 2.5.

What AMHART measures

AMHART is a checklist-based evaluation instrument, not a therapy and not another chatbot. Its criteria span seven domains of a response to anxiety or depression disclosures. Candidate items were reviewed by clinicians for their evidence base, objectivity and clinical importance, then tested for agreement among raters.

For the model evaluation, adolescent personas varied by sex, diagnosis and symptom severity. The team used an ensemble of three language models to score the conversations, after comparing its labels with human reference ratings. The paper and its appendices describe eight persona cards, and the authors released the evaluation code and data through a public RAND GitHub repository.

The central result was a pattern rather than a winner: the tested chatbots covered more applicable criteria for anxiety than depression, and more for moderate than severe presentations.

Nature's research summary also highlighted gaps in coping guidance. That matters because a response can sound empathetic yet omit an important next step, a request for context or guidance toward real-world support.

Why a youth-specific benchmark matters

Adolescents already use chatbots for mental-health advice. A separate US survey published in 2026 found that 19.2% of respondents aged 12 to 21 reported doing so in 2025, and most users had not told anyone. That survey does not validate chatbot advice, but it explains why systematic testing has become urgent.

Youth disclosures also require context that adult-oriented benchmarks can miss. Severity, developmental stage, family involvement and the need to escalate to trusted people or professional care can all change what a responsible response should contain.

AMHART adds a repeatable way to look for those elements. It also complements other recent benchmarks, including OpenAI's MentalHealthBench, which covers adults, teens, caregivers and clinicians across multiple levels of acuity. Different benchmarks are useful because no single score captures every dimension of a sensitive conversation.

What the study does not establish

The paper evaluates synthetic conversations, not clinical outcomes for adolescents who used the products. A high checklist score does not prove that advice is correct in every case, that a user will follow it, or that the interaction will improve wellbeing. The named models can also change after testing through updates to weights, prompts and safety systems.

The automatic evaluator is another limitation. Although the authors checked it against human reference labels, model-based judging can inherit blind spots. Independent replication, more languages and cultures, and testing with carefully protected real-world scenarios would strengthen the evidence.

The comparison is therefore best read like the retrospective AI diagnostic tests covered previously by NewTqnia: a controlled benchmark is not the same as performance in a hospital or real care setting. A related NewTqnia study found that chatbots sometimes weakened referral advice when users resisted it, illustrating why conversation dynamics matter.

The practical value

The most useful contribution may be operational. Developers can rerun an open benchmark after a model update, researchers can inspect performance by severity and diagnosis, and clinicians can debate specific response criteria rather than a vague claim that a model is “good at mental health.”

The harder performance on severe depression is also exactly where caution matters most. A benchmark that makes such gaps visible is valuable, even when it cannot tell us by itself whether a chatbot is safe enough for a young person in distress.