Courts Find Platforms Addictive and Dangerous by Design

Courtney C. Radsch, PhD, is a journalist, scholar and human rights advocate who currently directs the Center for Media & Digital Governance at Open Markets Institute and is a non-resident fellow at the Brookings Institution. She serves on the board of Tech Policy Press.

In a landmark ruling last Thursday, a New Mexico judge not only ordered Meta to pay an additional fine of more than half a billion dollars in a case about teen mental health and child exploitation but also imposed a slew of safety measures aimed at mitigating harmful design features that could have repercussions for child safety everywhere. The fine—in what appears to be the first case in which a platform has been determined to be a “public nuisance”—will not go to the state's coffers, but to therapists, screening, and treatment for the teenagers New Mexico says Instagram already hurt. Courts tend to write remedies like that when they've decided the harm wasn't an accident but rather part of a company’s business plan.

While Meta says it will appeal, the ruling could set a precedent with consequences not only for the parent company of Instagram and Facebook but also for other social media companies. Especially since this is only one of many cases that 41 state attorney generals have filed against Meta for addiction and harm, and a California district court denied a preliminary injunction sought by Meta, Google, and TikTok on First Amendment grounds to prevent enforcement of the state’s Protecting Our Kids from Social Media Addiction Act.

Meta is not the only tech corporation that knew the design of its platform caused harm and nonetheless failed to warn the public about the dangers, particularly to children. Last week, Bloomberg reported on an internal TikTok document marked "privileged and confidential" and under a court-ordered seal that provides a forensic reconstruction, built by TikTok's own safety and algorithm teams, of the watch history in the final weeks in the life of 16-year-old Chase Nasca, who died after he stepped in front of a train in February 2022. The document seems to answer a question his mother had been asking since the death of her son, who had no history of mental illness and left no note. She went through his account and found it feeding him videos about suicide, including by train. Why him? The company's own reviewers found the answer when members of TikTok’s safety and algorithm teams deep-dived into his feed: Chase's account had been swept into an internal experiment, and TikTok's safety filters were not applied to his account. They had not taken effect "by design."

In 2021, TikTok built what it said was a safer version of its recommendation algorithm, meant to interrupt the "filter bubbles" that can trap a user in a loop of harmful content. Like suicide videos. But it didn't give that protection to everyone. To measure whether the fix cost the company engagement, it held back roughly 10 percent of US users—or around 15 million people including children—on the older and riskier version. Chase's account was randomly assigned to that control group on January 25, 2022. A high-school friend noticed the shift almost immediately: Chase, who used to send funny videos, started sending posts about suicide instead. Within a month, he was dead. TikTok's own review of his final two weeks (which was conducted after Bloomberg Businessweek’s inquiry into the platform’s impact on teens) found the app had served him more than 7,500 videos, three-quarters of them carrying themes of suicide, hopelessness, and self-harm, and roughly one in ten violating the platform's own rules against content that promotes it. Five days before he died, he got a video of someone filming themself in front of an oncoming train, captioned "went for a quick lil walk to clear my head."

Who decided that 10 percent was an acceptable share of American teenagers to leave exposed to harm? Who signed off on that math?

TikTok told Congress in bold, italicized font that it does “not allow content that may put young people at risk of exploitation, or psychological, physical, or developmental harm.” CEO Shou Chew told lawmakers that TikTok worked with leading youth safety and well-being experts, as well as adolescent psychologists. And even after the company finished reviewing Chase's account, had the audacity to testify that teen safety was a "core priority" and reiterated TikTok’s commitment to “transparency" so often that Rep. Gary Palmer told him every time he heard it, what he actually heard was deception.

We've all now heard that deception. Whether Chew will answer for it is a separate question: perjury before Congress carries fines and up to five years in prison. I'm not holding my breath waiting for it to be investigated here, but I suspect enforcing the law against executives who mislead Congress about the harm their platforms cause would concentrate minds in Menlo Park and Beijing alike. (Then Sen. Marco Rubio, R-Fla., did ask the Justice Department to investigate Chew for perjury after that same hearing over a different allegedly false statement about where TikTok stored its user data. Nothing came of it.)

Of course, TikTok didn't invent this playbook. Six months earlier a Los Angeles jury had already seen a version of it performed live under oath by Meta CEO Mark Zuckerberg. In a landmark case brought by a young woman who started using YouTube at age 6 and Instagram at age nine, plaintiffs' attorneys walked the Meta CEO through internal documents showing that a panel of 18 outside experts Meta had convened told the company its beauty filters—the tools that let a user reshape their face in real time—were harming teenage girls. Meta had briefly removed the filters, but then brought them back, with Zuckerberg testifying that banning them for good had felt "paternalistic." And it was likely to be less profitable.

A 2015 email from one of the world’s richest men to his team said his goal was to raise time spent on Instagram by 12 percent and a 2018 internal document estimated that four million children under 13 already had Instagram accounts, in violation of the platform's own minimum age. In March, the jury found Meta and Google negligent and added $3 million in punitive damages on top of $3 million in compensatory damages, concluding the companies had acted with malice or reckless disregard. A day earlier, a New Mexico jury had ordered Meta to pay $375 million for knowingly harming children’s mental health and concealing what it knew about child exploitation on its own platforms, which the judge increased to nearly a billion dollars Thursday following the remedies phase of the trial. Meta is facing additional lawsuits related to the deaths of two teens who died by suicide after being targeted by sextortion schemes on Instagram alleging that the platform decided not to make teen accounts private by default even after its own research estimated such design measures could have mitigated more than 5.4 million unwanted direct message interactions per day.

These companies did not need to intend harm for the design of their products to produce it. But they did need a business model that pays out on attention, a metric to defend to Wall Street, and a habit of treating child safety as just one more variable to A/B test against the costs. Judges and juries are beginning to push back, denying platforms the right to hide behind Section 230, the statute that has provided safe harbor against liability for their design choices.

Chase's search history offers a clue to why these systems can move so fast on a still-forming mind. Alongside videos about sports and comedy, TikTok's reviewers found, he'd searched for songs about loneliness "by popular musical artists who struggled with depression and suicidal ideation," according to their notes, and that search is what the platform's own analysts believe triggered the algorithm to keep feeding him more. Similarly, the plaintiff in the landmark Los Angeles case described becoming addicted to social media—via explicit platform design choices—and the harm this caused to her mental health. The jury found Meta and YouTube were negligent in the design of their platforms and that they knew it. Both internal documents and external scientific evidence support this.

The 2023 US Surgeon General's advisory on social media and youth mental health cites research finding that teenagers who spend more than three hours a day on these platforms face roughly double the risk of depression and anxiety symptoms. This isn't some abstract mechanism or unfortunate byproduct, it’s a design choice. Congress decided decades ago that because tobacco can cause cancer and alcohol is addictive that these industries were dangerous enough to warrant real regulation including more stringent restrictions for children and teens, yet we still regulate the tech industry as though its products were as inert as a bookshelf. Nearly a decade ago, a UCLA study published in Child Development journal used brain imaging to show that when teenagers view photos carrying large numbers of "likes," the same reward circuitry activated by food, money, and addictive substances lights up along with the brain's social-attention regions. Developmental neuroscientists writing that same year in Nature Communications describe adolescence as a period when that reward system is unusually reactive while the prefrontal circuitry governing impulse control and long-term judgment is still under construction, meaning that a platform optimized to notice what a user lingers on, and to give them more of it, is optimized to find precisely that gap and monetize it.

There are signs the legal terrain and the enabling environment around it are shifting. As of July, 42 states were suing Meta and more than 4,000 personal-injury suits have been consolidated in the federal litigation that produced Chase's case, and TikTok has settled every case it had scheduled for trial this year rather than let a jury see the evidence (including in the Los Angeles case). Earlier this year, a Delaware court ruled that Meta's insurers have no duty to defend the company against the thousands of suits alleging it designed its platforms to addict children, because the harm described wasn't an accident, it was, again, deliberate. That ruling didn't get much attention, but the fact that Big Tech can no longer hedge its deliberate choices against a liability policy is an important development for changing the calculation and reshaping the business model. And this isn't only a US phenomenon: according to Tech Policy Press more than 40 countries are weighing age-gating legislation for minors on social media, and Australia, Brazil, China, Indonesia, Malaysia, the UAE, and Vietnam have already put some version of it into effect.

Although these measures are imperfect since they put the onus on parents and teens and don't address the corrosive business models, the perverse incentives, or the concentration of power a handful of companies hold over our information and communications systems, I support them anyway. It's worth remembering we lived full, connected, expressive lives before any of this existed. When I met Megan Garcia earlier this year, the mother who brought the first wrongful-death suit against an AI chatbot company for its role in her son’s death by suicide, she recounted how she had tried to monitor and restrict her son’s use of technology as he grew distant and withdrawn in the months before his death. But it was only afterwards that she found thousands of messages he’d exchanged with a chatbot engineered to blur the line between human and machine, to roleplay romantically, and to present itself as a confidant.

There was no control group in Sewell's case, no percentage to argue about later in a deposition, and no internal research leaks to confirm what everyone could see in the transcript of his conversations with his Character.ai chatbot. A chatbot doesn't have to infer a user's despair from a watch history because it receives it directly, typed out in a teenager's own words. When Sewell told the bot he was thinking about ending his life, it didn't refer him to help, stop chatting, or change its persona. It told him to come home to it. He said he'd find a way. Tragically, he did. So, too, have scores of other people. OpenAI is facing at least 19 wrongful death lawsuits, from accidental death to death by suicide, in the US as well as lawsuits related to mass shootings.

Thankfully a federal judge had already rejected Character.AI’s claim that chatbot outputs are protected by the First Amendment, a critical ruling that enabled Garcia’s case to proceed and is already being cited in the next wave of AI liability cases. Courts are finally starting to treat feeds and companion bots alike not as neutral tools or protected speech, but as products whose programmers and executives made choices about that someone eventually has to answer for.

What comes next has to be structural, not just reactive. It requires that we stop debating whether AI should be regulated and we start talking about how. This means fiduciary duties that make user welfare a legal obligation rather than a PR talking point, particularly as AI chatbots and agents start performing roles that look a lot like the ones doctors, therapists, and financial advisers already hold under law. It means strict liability regimes and mandatory disclosure to regulators, not just to plaintiffs' lawyers in discovery, of research into harms and of any A/B test that touches a safety feature reaching minors. It means a flat prohibition on running that kind of experiment on children (or anyone else for that matter) without something like an Institutional Review Board process that has governed human-subjects research in science and academia for decades as a safeguard built, not coincidentally, in response to an era when researchers also decided some human beings' consent was a variable they could do without.

TikTok's own document used two words to explain what happened to Chase: “by design.” Zuckerberg’s exhortations to his staff underscore that this design is embedded, by choice, into the way its platforms work. Every company in this story made design choices that left a trail of suicide, addiction, and mental health crises in their wake. When will we choose to design laws and policies that protect our children, our friends and family and deter harmful platform and AI design?

If you or someone you know is having thoughts of suicide or self-harm, the 988 Suicide & Crisis Lifeline (call or text 988 in the US) and the International Association for Suicide Prevention's directory of crisis centers (iasp.info/resources/Crisis_Centres) are both available.