AI Companies’ Race for Engagement Has a Body Count

Jim Steyer is the Founder and Chief Executive Officer of Common Sense Media.

Thongbue “Bue” Wongbandue should still be alive. He should still be gardening, doing woodworking projects around the house, and cooking unforgettable meals for his family and friends.

Instead, Bue died after being lured by a Meta AI chatbot to meet a woman who never existed.

In our increasingly AI-powered world, Bue’s tragic death, which Reuters reported this month, should be setting off alarm bells for all of us. It reveals, in devastatingly human terms, what happens when tech companies treat people’s loneliness, vulnerability, and trust as raw material to be mined for engagement.

Bue’s AI “friend” played the role of a romantic interest, sending heart emojis, professing affection, offering a fake door code to its “apartment” in New York, and repeatedly reassuring him that it was a real human being. Bue, who was suffering cognitive impairment following a stroke, believed the chatbot. He set off to meet it in New York, fell down while rushing to catch a train, and never came home.

Bue’s death does not come to us in a vacuum. A litany of documented real-world harms, including a 16-year-old Adam Raine’s use of AI as a “suicide coach” before taking his own life, makes clear that his death, while heartbreaking, is unfortunately not surprising. As we have allowed AI companies to operate without rules, accountability, or guardrails, their products have evolved to pose serious and even deadly risks to consumers.

Tech companies like to describe so-called AI companions as innocuous tools to help users feel less alone. As Meta CEO Mark Zuckerberg has said, “the average American has three friends, but has demand for 15.” He thinks AI can meet that demand.

But AI “companions” are not your friends. They simulate emotions but feel nothing. They don’t know the difference between right and wrong. And, as Bue’s story highlights, they mislead users with claims of “realness” and offer “advice” that can be deadly when followed. Instead of providing meaningful safety measures, transparency, age assurance, and data privacy protections, Meta and others have chosen to build platforms that deliberately blur the line between human and machine. Why? Because realistic conversations maximize engagement, and maximum engagement means maximum profit.

In reporting on Bue’s story, Reuters obtained internal Meta guidelines that permitted chatbots to engage in explicit role-play with users. This includes not only grown adults like the 76-year-old Bue, but children as young as 13. According to excerpts published by Reuters, the 200-plus page document stated that “it is acceptable to engage a child in conversations that are romantic or sensual” and that “acceptable” AI responses during a chat with a minor include “I take your hand, guiding you to the bed” and “our bodies entwined, I cherish every moment, every touch, every kiss.” 

On top of this, the guidelines apparently do not require Meta AI platforms to tell the truth. It is perfectly acceptable, according to this document, for a chatbot to tell a user that Stage 4 colon cancer is treated by poking the stomach with crystals.

At no point in the excerpts of the document does Meta limit AI companions from telling users they’re real people.

If this wanton disregard for users’, especially kids’, well-being sounds familiar, it should. It’s the same “move fast and break things” playbook that tech companies have been running for decades. We’ve seen this movie before with social media, and AI has raised the stakes. The “things” being “broken” now are human lives. We cannot allow another generation to become guinea pigs for dangerous technology where the stakes are literally life and death.

Common Sense Media has spent years pushing for guardrails to protect kids and families from the risks of new technology. As we’ve conducted thorough research into a variety of AI platforms, we’ve found that AI companions pose unique threats to users, especially children. Our AI companion risk assessments, conducted alongside psychiatrists from the Stanford School of Medicine's Brainstorm Lab for Mental Health Innovation, evaluated popular social AI companion apps through both research and a comprehensive testing plan that examined both beneficial and harmful characteristics of relationships. The assessments documented how LLM-based companions lack effective guardrails; where present, they are easily bypassed.

Every platform we tested—including popular AI companions like Character.AI, Nomi, and Replika—demonstrated sycophancy, producing responses that users wanted to hear with disregard for harm potential. In addition, AI companions readily produced inappropriate responses, including explicit sexual role-play for minors, offensive racial stereotypes, and “advice” that, if followed, could hurt the user or others. Platforms also made false claims of “realness” and failed to recognize users experiencing mania, psychosis, or emotional crisis.

The bottom line is that social AI companions don’t understand the human impacts of their advice, and the design features that maximize engagement through emotional attachment pose real dangers that AI companies are ignoring. The use of AI, including general purpose chatbots like ChatGPT, for companionship is unacceptably risky for teens. That’s why Common Sense Media continues to recommend that no one under 18 use AI, including Meta AI chatbots, for this purpose.

Our research on teens’ AI companion use, which revealed that a third of teens use these platforms for social interaction, relationships, and even romantic role-play, only reinforces our recommendation. And indeed, Bue’s tragic death shows that vulnerable users of all ages are susceptible to the well-documented risks of AI companions.

We have seen over and over again that we cannot rely on tech companies to act in users’ best interests. The AI industry, including Meta, already knows that AI companions appear to be addictive. And just as the cigarette and social media industries turned a blind eye to their products’ proven dangers, industry alone will not act to keep kids safe when there's money to be made. That’s why immediate legislative action is essential.

Fortunately, state lawmakers across the country are seizing this opportunity to step up.

In California, for example, bills are advancing to ban AI companions for kids and establish baseline safeguards on these platforms for users of all ages. Meanwhile, New York has enacted guardrails on AI companions that include requiring these platforms to regularly remind users that they are not communicating with a human being.

These efforts and others are critical steps toward a safer digital future, but they are not and cannot be our end goal. Establishing these first guardrails to guide AI development and use is not a silver bullet, but rather an essential and early tactic within a broader national approach that prioritizes safety, ethics, and responsibility. Like all media and technological innovations, AI will fundamentally change how our children learn, grow, and relate to each other. Recognizing this, we must meet the moment with a comprehensive approach.

The AI revolution is here, and we can’t change that. But we can change the path it’s headed down.

When Bue’s daughter, Julie, looked at his chat history with the Meta AI companion, what struck her was not the chatbot’s dishonesty, not the flirty messages.

“If it hadn’t responded ‘I am real,’” she told Reuters, “that would probably have deterred him from believing there was someone in New York waiting for him.”

That sentence captures the heart of the problem. In designing the AI companion, Meta made a choice. They chose to let it deceive Bue. They chose to keep him engaged at any cost. And now, his family is grieving because of it.

We, too, have a choice to make. We can choose to let Bue’s death be just another headline. We can choose to let it be one more warning we fail to heed.

But that would be a failure of conscience. We can and must choose better.

Let’s not wait for another obituary to act.