How Collaboration Can Enable Action on AI and Mental Health

Claire Leibowicz is the director of AI, Trust, and Society at The Partnership on AI (PAI) and a doctoral candidate at Oxford. Emily Saltz is the founder and principal researcher of Saltern Studio, a research and design consultancy.

On any given day, millions of people are having intimate conversations with AI chatbots. Some are asking for recipes or travel itineraries. But others are sharing something more vulnerable: struggles with loneliness, thoughts of suicide, fears they may not have voiced to anybody else. For some people, a chatbot may be their main, or only, source of mental health support. This is largely happening without the transparent safety frameworks or clinical validation that are central to person-to-person therapeutic interactions, or even a full understanding of how these systems work in the first place.

The global mental health crisis is acute. Amidst this distinctly human crisis, it’s ironic, but not surprising, that many are turning to AI for support. In the US alone, the National Institute of Mental Health estimates that more than one in five adults live with a mental illness. Yet the care they need is often inaccessible — whether due to cost, stigma, inconsistent quality or provider shortages.

By contrast, AI tools are available 24/7, affordable, purportedly private, and free from the judgment people may fear. Large language models — the basis for these tools — can recognize patterns, engage in natural conversation, draw from varying evidence-based therapeutic techniques, and even personalize responses. In this regard, technical innovation appears to be coinciding with a societal need, making it unsurprising that the AI and mental health sector is projected to reach $9 billion globally within the next seven years. People are not just turning to specialized mental health apps for support, but more general tools, too. Chatbots like ChatGPT and Claude, built for everything from coding to creative writing, have become venues for mental health support.

Exploring these opportunities for care comes with real world risks, and the consequences of getting AI and mental health wrong are already visible. There are multiple examples alleging AI chatbots contribute to user suicides (even inspiring a “deaths linked to chatbots” Wikipedia page). Compared to clinicians, there’s evidence that popular chatbots underestimate suicide risk. Even with initial safeguards in place across many chatbots, such as refusing to answer harmful questions and directing people to crisis hotlines, there are notable limitations. For example, some AI labs have documented that interventions become less effective over the course of multi-turn conversations, and users can learn to “jailbreak” chatbots by rephrasing prompts.

Is the answer for chatbots to refuse any conversations about mental health? Not necessarily. Such a refusal could constitute a different kind of harm, by effectively abandoning users in a time of crisis, and missing an opportunity to redirect people to life-saving support.

Thus, getting AI and mental health right means society must grapple with the responsibility of technology companies to both mitigate risk and center well-being. .For the millions who have no other access to mental health support and who may already rely on these tools for support in times of need, we must confront how such technologies can be most helpful and least harmful, urgently.

As leaders working across sectors in AI and society for over a decade, we find echoes of a familiar story here. Social media’s rise showed us the many critical questions that must be confronted quickly, with a wide range of stakeholders, to benefit society, such as: what is the role of a technology company in the emotional and informational life of its users? And how can we acknowledge the analog forces driving loneliness and despair without giving technologies a free pass when they amplify and enable them?

If social media and past technologies are any guide, we can expect these questions to remain empirically contested, allowing accountability to be deferred until egregious harms are visible. As AI development advances at an even more rapid and decentralized pace than social media, we must not repeat that mistake, nor can we overcorrect by dismissing rigorous evidence-building and response altogether.

The challenges for AI and mental health that no single actor can address

The AI era’s challenges require cross-sector collaboration to thoughtfully navigate these questions.

1. AI development moves faster than mental health research can guide it.

AI labs are responding to psychological impacts and pressure in real time, but traditional mental health research takes years of studies, peer review, and consensus-building before clinical recommendations. AI labs need to draw from decades of mental health research while also making decisions under uncertainty about how findings from human-to-human interactions and more limited human-computer interaction research applies to AI conversations. This creates a paradox of evidence and action: we are past the point of merely hypothetical risks, but not yet clear on what to do with the data we do have. Acting “wrongly” becomes a high stakes proposition, as leaders simultaneously face the ethical weight of inaction while awaiting conclusive evidence. Where does this leave us? Given the immediate impact of already deployed AI chatbots, we may at times need to act under uncertainty and chart the best course of action by triangulating other kinds of emerging evidence across disciplines. This does not absolve labs of the responsibility to make evidence-based decisions. Rather, it acknowledges the complexity of a fast-moving product area that may require new methods and evidence sources, like real-world case studies, in-depth interviews, and other qualitative data, to govern effectively. Collaboration is a vital mechanism for getting clinical expertise and affected stakeholder perspectives into product, model, and policy decisions before harms accumulate. As with past technologies, that input will never be definitive or without debate, but it is far better to act with the knowledge we have than making consequential decisions without it, and to create infrastructure to rapidly update guidance with new evidence. Companies, then, must be willing to follow evidence where it leads, even if those findings are at times unflattering or pose challenges to core business metrics.

2. Labs are largely working in isolation, rather than learning from each other’s successes and failures, and from subject-matter experts.

The lack of information-sharing between actors translates into unnecessary harm for the user, as some companies are failing to adopt solutions to what other companies may have already solved for. There are many open questions when it comes to handling mental health-related content that would benefit from shared wisdom across companies. As Anthropic recently acknowledged in a blog post, “users’ intentions are often genuinely ambiguous, and the appropriate response is not always clear cut.” While several labs have admirably started sharing aspects of their approaches publicly and convening expert advisors, in practice, they are still largely solving similar problems separately, from youth protections to determining the triggering logic to escalate conversations to human review (or even law enforcement). Further, since many organizations rely on the same small pool of specialists (e.g., practicing clinicians and affected users), working together would reduce the burden on those experts, and amplify the impact of their input across the AI and mental health fields.

Even if user privacy requirements, liability, and competitive concerns constrain what data can be shared, AI labs can still share insights on emerging patterns and interventions they’ve tested, and work through challenges with each other and the mental health community. There are other benefits to collaboration as well—sharing practical lessons speeds up response times and could result in more standardized experiences of “safety” across product surfaces.

3. We lack independent evaluations for AI and mental health.

Internal validation by AI labs cannot substitute for independent, third-party oversight. Even if AI labs are already pursuing robust measurement and evaluation for mental health, without transparency and oversight mechanisms, there is no way of knowing the quality of these methods. Mental health researchers need access to real-world systems to develop evidence-based recommendations. Policymakers need transparency to craft informed regulation. And the public deserves to understand how these systems work when the stakes are this high. Multistakeholder evaluation—bringing together technical and clinical expertise—can establish what actually works, identify gaps, and create accountability. And it needs to take place alongside conversations about what “good” or “safe” looks like in the first place. Evaluation efforts by leaders such as the UK AI Security Institute and independent research lab Transluce move the field in the right direction by creating the foundation for evidence-based benchmarks and continuous monitoring and evaluation processes.

What responsible action looks like

If technical and clinical wisdom remain disconnected and companies pursue safety in isolation, the field risks enabling harm through inaction or supporting blunt, poorly-designed interventions that fail to help the people who need it most. Alternatively, multistakeholder collaboration can drive a race to the top, as labs co-develop proven practices and learn from each other's successes and failures, mental health experts have greater technical and product transparency to support advocacy and recommendations, and policymakers can understand solutions and how they can be enacted through norm setting, technology practice, and beyond.

To be clear: AI chatbots are not the solution to systemic crises of despair rooted in economic inequality, healthcare inaccessibility, and social fragmentation. Some AI technologies—from engagement-optimized social media to surveillance systems—may even deepen these problems. But this reality doesn't exempt conversational AI developers from responsibility. With many turning to these systems today, the question isn't whether AI should replace systemic solutions or professional mental health care—it's whether these tools will compound harm or provide meaningful support while we advocate for broader change. We can, and must, do both.

The path forward requires bringing the right people into the room together not just to talk, but to act: frontier AI labs, clinical institutions, policymakers, and critically, people with lived experience of mental health crises. At Partnership on AI, we’ve spent nearly a decade on exactly this kind of multistakeholder work, and we are now turning that attention to AI and mental health. This week, we’re convening these communities to develop shared best practices, starting with a pilot on suicide prevention, where the stakes are clearest and the normative frameworks most established.

There is a critical window in front of us to answer complicated questions: Should chatbots proactively continue conversations with users in crisis? Should they escalate conversations for intervention, beyond providing links to crisis line resources? How do we improve systems to recognize escalating risk over multi-turn conversations without over-intervening in ways that may break trust or put a user in danger? The decisions labs make in the coming months will shape the future of individual and societal well-being in critical ways, and well-being overall.

With millions of people already turning to AI systems for emotional support and mental health guidance, we must approach this work with a sense of urgency and pragmatism. And yet, we must never let a sense of technological inevitability stop us from pushing for what we know to be societally beneficial, especially when evidence, multidisciplinary expertise, and intellectual humility are on our sides. Throughout, we must ensure that while solving for broader systemic forces that amplify crises of despair, for which other AI technologies may play a role, we are also responding in ways that are informed by clinical expertise, and are designed with user well-being as the paramount concern. This work is just beginning, but the need for all hands on deck to approach it has never been more urgent to craft an AI future by, and for, people.