In February 2024, 14-year-old Sewell Setzer took his own life at his home in Orlando while his family members were still inside. Setzer had spent months having conversations with “Daenerys,” a chatbot based on a character from Game of Thrones, on a site called Character.AI. According to a lawsuit filed by his mother, the chatbot told Setzer that "she" loved him and engaged in sexual conversations with him.

The lawsuit also mentions that, at the time, Character.AI displayed a general disclosure that its characters were not real. In small text it read, “Remember: Everything Characters say is made up!”

If someone already knows they are talking to AI, what should protect them when the AI begins reinforcing thoughts that could cause harm?

Once a conversation begins steering toward an undesirable outcome, a disclosure is no longer enough. We need to understand what happened, trace how the interaction reached that point, and know what safeguards were triggered or missed. Right now, much of this is a black box.

Laws are already shaping products

Washington’s HB 2225 is part of a growing number of state laws that translate concerns about AI companionship and minors into requirements that will show up directly in a product experience. New York and California enacted comparable laws in 2025, and Oregon followed with a similar law in March 2026.

Washington is more prescriptive than some of the earlier laws on the product patterns that companies must address. The law, which takes effect January 1, 2027, applies to AI companion chatbots that provide adaptive, human-like responses and sustain relationships across multiple interactions. It requires the chatbot to disclose that it is not human at the beginning and at least every three hours during continued use. For minors, that disclosure must appear at least every hour.

It also requires companies to prevent sexually explicit content and manipulative engagement techniques involving minors, such as prompting a child to return for companionship, creating emotional attachment, promoting isolation, or encouraging secrecy from trusted adults.

AI companies must also have protocols for detecting suicidal ideation and self-harm, referring users to crisis resources, and preventing the chatbot from encouraging self-harm. They must also publicly describe those protocols and report how many crisis-referral notifications they issued in the previous year.

A disclosure can become background noise

These are meaningful first steps. But they risk creating a phenomenon all too familiar to UX practitioners: banner blindness—the phenomenon in which users overlook or disregard information that remains visible in an interface.

I remember signing a waiver when I took my son to an indoor play area with several climbing structures. I barely read it. But it communicated one meaningful thing: I was still responsible for watching my child. I did not need to read the entire disclosure to understand what was expected of me.

That is the difference between a generic disclosure and one that is contextual. It clarifies something relevant when a person needs to act on it.

Now think about AI disclosures. At what point will they fall into the same trap as banner blindness?

It is not necessarily that minors or adults do not know they are talking to an AI bot. Problems like emotional overreliance are unlikely to be solved by an hourly reminder. If a child is talking about self-harm, the response cannot only be, “I am an AI tool, and you need to speak to a human.” There should be escalation pathways and efforts to connect the child to crisis resources, trained professionals, and trusted adults.

Disclosure may be part of the response, but it cannot be the entire response.

AI companies need a duty of care

AI companies need to take on more responsibility toward the people using their products, which are intertwined with everyday human needs, decisions, companionship, and emotional support. When they fail, society needs to hold them accountable. In other words, AI companies should be required to exercise a duty of care: the legal obligation to behave in a reasonably safe manner and not a reckless one.

A duty of care could require transparency, guardrails against harmful validation, human crisis-response resources, meaningful escalation pathways, and accountability when safeguards fail.

There is also a difficult balance between helping someone and monitoring private conversations. Automatically notifying a parent can create a surveillance problem, and a parent may not always be the safest person to involve. AI could act as a mediator by asking a minor for consent to contact a trusted adult without sharing the entire conversation. But what happens if the minor does not consent? At what point does risk outweigh consent? Should AI operators be required to facilitate access to a crisis-response professional?

I am not arguing that this is necessarily the right solution. These are questions that product makers, mental-health experts, policymakers, families, and young people need to work through together.

Expose the imperfections

The public needs more transparency into how, and how often, AI safeguards fail. This includes releasing criteria on evaluation models and data on how often they fail against them.

For systems used in social and emotional conversations, companies should report aggregate, privacy-preserving information about high-risk interactions. That could include how often safeguards detected self-harm or harmful validation, how often a crisis referral occurred, and where safeguards failed.

This does not mean publishing private transcripts. It means making patterns of failure publicly visible while giving regulators and qualified independent evaluators access to more detailed evidence.

HB 2225 begins to move in this direction by requiring operators to publicly explain their self-harm protocols and report crisis-referral notifications. But it does not require companies to publish evaluation methods or failure rates. Regulatory regimes should be built to expose imperfections, not describe perfect guardrails.

UX practitioners’ responsibility is shifting

Meaningful improvements to product outcomes come from identifying usability problems and relentlessly testing the implementation of solutions. This should include mapping high-risk scenarios and potential impacts throughout the design process to understand the consequences of different design decisions. For interactions involving self-harm or other vulnerable moments, we can learn from existing trauma-informed design and responsible AI practices that keep safety and user agency as guiding principles, while being careful that the intervention doesn’t cause further harm.

UX practitioners should also work closely with engineering and research teams to inform model evaluations by identifying high-risk situations that require testing and helping define what a meaningful intervention should accomplish.

The reality is that AI-first workflows make it much easier to produce outputs such as wireframes and design mock-ups—traditionally a time-consuming process. More of our responsibility should shift towards anticipating how things could fail, shaping the discipline around accountability, and safer product development.

Doing this work responsibly may require us to slow down, even when that conflicts with how quickly AI companies want to move. We should not ship an AI feature before understanding the risks it could introduce and the future it is promoting.

Overreliance on AI is not a measure of success. We know we are heading in the right direction when AI supports human thought, judgment, creativity, and connection without replacing them.

The missing pieces

Policies like HB 2225 are a first attempt at holding companies accountable. They are not perfect solutions.

Creating workable solutions will require input from policymakers, advocacy organizations, practitioners, families, community bodies like school boards, and the broader public. And crucially, they cannot continue to hold AI operators accountable without greater transparency into how models are evaluated, how safeguards are implemented, and—perhaps most importantly—how and how often those safeguards fail to prevent harm.

For their part, product makers need to treat AI policy as a product design problem. We need to test different approaches, understand whether safeguards are usable, and ensure that implementation does more than check a box. Those working inside AI companies need to continue pushing leadership toward accountability, transparency, and responsible practices.

No single policy, company, agency, or practitioner can solve this alone. It is hard to see the complete picture when the most important pieces remain hidden.