Faculty Research

A lot of AI research focuses on whether a model can generate impressive answers or achieve strong benchmark performance. Our work focuses on a different but equally important question: Can we make AI reliable, safe and useful enough for high-stakes settings?

Large language models are especially promising for language-based interactions such as mental health support. At the same time, there’s a shortage of mental health resources available to the general public, so there’s a strong need for technologies that can help reduce the burden on providers and increase access to support.

As people increasingly turn to chatbots for psychological support, my collaborators and I saw a need for a more rigorous evaluation of how these systems perform in mental health settings.

One challenge is that the tools researchers typically use to evaluate AI don’t fully capture what matters in a mental health conversation.

Traditionally, computer scientists evaluate language models on knowledge-based tasks, such as answering multiple-choice questions or taking standardized exams. But mental health support is very different. It’s not just about factual correctness — it requires emotional awareness, symptom recognition and the ability to engage in open-ended conversations.

There has been a gap in understanding how language models respond to real-world patient questions. We wanted to examine the quality of those responses, identify potential safety concerns and better understand both the strengths and limitations of these systems.

We wanted a clinically grounded evaluation of how language models perform in mental health settings, so we collaborated with 100 mental health professionals — more than 70% of whom were licensed therapists — to evaluate AI responses to real-world patient questions.

Overall, we found that current language models perform quite well. They often received high ratings for empathy and generally scored well across multiple evaluation dimensions. At the same time, high overall ratings did not necessarily translate to low safety risks. Clinicians identified several recurring concerns, including overgeneralization, limited personalization and advice that could cross clinical boundaries. These issues sometimes appeared even in responses that otherwise seemed empathetic and helpful.

To better understand those risks, we conducted a second phase of the study in which clinicians helped design challenging questions to stress-test the models and expose potential weaknesses.

One of the key takeaways is that today’s AI systems show real promise as mental health support tools, but important questions remain about safety and appropriate use.

One immediate application is using the dataset and findings to design better training methods, safeguards and deployment protocols for language models used in mental health support.

We’re also pursuing follow-up research. Recently, we received an OpenAI Mental Health Award to support additional work in this area.

In this study, we focused on general client questions. Our next step is to examine more realistic and challenging situations. For example, what happens when clients are resistant or uncooperative? In real-world settings, these situations are common. Can language models still respond effectively? Can they handle these interactions safely? That’s one direction we’re actively pursuing with support from the grant.

The other direction is moving from evaluation to improvement. Now that we’ve identified failure patterns and problematic behaviors, the next question is how we can make language models safer and more suitable for deployment in high-stakes settings.