Big Tech May (Finally) Be Taking Explicit Deepfakes Seriously

In the last year, tech giants like Google, Microsoft, and Meta have faced mounting criticism from digital safety experts, womens’ rights advocates, and legal scholars alike for their perceived roles in the “skyrocketing” popularization of non-consensual, sexually-explicit deepfakes online.

Critics have pointed to Google’s reluctance to de-index AI-generated pornography websites in search, Meta’s tendency to carry advertisements for AI-powered “nudify” apps on Facebook and Instagram, and Microsoft’s failure to put adequate guardrails around its own AI tools as evidence of the industry’s complicity in the growing epidemic of deepfake abuse, which studies have found disproportionately impacts women and girls worldwide.

However, several announcements related to these companies in the last week could signal a sea change within Big Tech, and may give those calling for more aggressive moderation of explicit deepfakes reason for optimism.

Meta’s Oversight Board Recommends Changes in Policy

On Thursday, July 25, Meta’s Oversight Board ruled on two cases of “AI images” that depicted women public figures—one from India, and one from the United States—in sexually explicit scenes. In both cases, the Board determined that the images were in violation of Meta’s rules against “derogatory sexualized photoshop,” and required that Meta permanently remove the violating content from their respective platforms. This ruling effectively overturned the company’s original decision to allow the image of the Indian public figure to remain on Instagram.

“​​Given the severity of harms, removing the content is the only effective way to protect the people impacted,” the Board said in a summary of the ruling. “Labeling manipulated content is not appropriate in this instance because the harms stem from the sharing and viewing of these images – and not solely from misleading people about their authenticity.”

The Oversight Board also made a series of non-binding recommendations to the company, aimed at clarifying their content moderation policies. The recommendations included replacing vague, subjective terms such as “derogatory” and “photoshop” with clearer, more intuitive language.

“The Board believes that people using Meta’s platforms should be able to understand the rules,” the ruling reads. “While the term ‘derogatory sexualized photoshop’ should have been clear enough to the two users posting in these cases, it is not sufficiently clear more generally to users.”

Meta has yet to respond to these recommendations publicly, but the company’s own transparency reports suggest it has a history of (at least partially) implementing the Board’s suggestions in previous cases related to sexual exploitation and gender-based harassment on its platforms.

Microsoft Acknowledges the Scale of the Problem

On July 30, just over a week after the Oversight Board handed down its ruling, Microsoft published a report “diagnosing the problem of abusive AI-generated content” and calling on policymakers around the world to update and propose legislation aimed at protecting citizens from “deceptive and abusive AI-generated content.”

“​​The purpose of this white paper is to encourage faster action against abusive AI-generated content by policymakers, civil society leaders, and the technology industry,” Microsoft vice chair and president Brad Smith wrote in the report’s introduction. “Ultimately, the danger is not that we will move too fast, but that we will move too slowly or not at all.”

Smith went on to describe criminal penalties for the creation and distribution of AI-generated non-consensual intimate images (NCII) and child sexual abuse material (CSAM) as “common sense and sorely needed.” Several US states, such as Minnesota and Texas, have already adopted such laws, although none currently exist at the federal level.

And besides the assertion that that tech companies (presumably including Microsoft) should be afforded a seat at the regulatory table, Smith’s remarks largely echoed those of victim advocacy groups such as #MyImageMyChoice, which publicly criticized Microsoft earlier this year after a user was able to use its image generation tools to create the viral Taylor Swift deepfakes that temporarily shut down X searches back in January.

Google Takes Action on Search

And then, on July 31, Google Product Manager Emma Higham announced in a blog post that the company would be cracking down on explicit deepfakes in its search results, in a major shift from Google’s historically laissez-faire approach.

“Today, we're sharing a few significant updates, which were developed based on feedback from experts and victim-survivors, to further protect people,” Higham wrote. The updates include an overhaul of the reporting process by which victims can request to have web pages containing abusive content removed from Google search, and an adjustment to the Google ranking algorithm which will push websites hosting deepfake pornography further down in search results.

Whereas data analysis conducted in 2023 showed that Google was the “single largest driver” of web traffic to deepfake pornography streaming sites, the announcement suggests that non-consensual deepfakes should be much harder to find via Google moving forward.

“For queries that are specifically seeking this content and include people’s names, we'll aim to surface high-quality, non-explicit content — like relevant news articles — when it’s available,” Higham explains. “With these changes, people can read about the impact deepfakes are having on society, rather than see pages with actual non-consensual fake images.”

At the time of publication, Google’s updates seem to be working as advertised, at least for searches containing the names of popular A-listers like Taylor Swift or Scarlett Johansson. Deepfake videos of relatively less well-known social media influencers, however, can still be found in the first three pages of Google search results.

Google’s announcement has been lauded by several advocates on social media, including Dawn Hawkins, CEO of the National Center on Sexual Exploitation. Hawkins called the change a “huge win” on X, writing, “This is a major step forward in combating #IBSA [image-based sexual abuse] and making the internet safer for all.”

Notably, the announcement came roughly three weeks after the publication of a recent Wired investigation which found that Google has ignored or declined to implement far more mitigations for deepfake abuse than it has adopted. For example, several anonymous, former Google employees told Wired that higher-ups have repeatedly refused to employ StopNCII, a software system launched in 2021 which is used by several other major content platforms (including Meta, TikTok, Reddit, and PornHub) to detect re-posts of intimate images previously flagged by the victims as non-consensual.

Whether Google’s new updates will precede more comprehensive deepfake abuse interventions on the search engine is impossible to predict at this stage. In fact, whether any of the developments described above will actually result in better protections for victims of deepfake abuse and a measurable reduction in the volume of such material on major social and search platforms—as opposed to mere corporate gesturing—remains wholly unclear.

However, taken together, these developments do suggest that the unflinching criticisms being lobbed from within the digital safety community are, at the very least, being heard by three of the most powerful tech companies on Earth. And in a moment where the United States Congress is considering a bill that would enshrine the rights of victims of “sexual digital forgeries” to sue their abusers into federal law, they may provide reason enough to hope that a safer internet is actually within reach.