Breaking the Open Web Won’t Stop the Bad Bots

Late last month, more than 300 representatives from major media outlets were on Capitol Hill to advocate for the Stealth Bot Prohibition Act. They came with a legitimate grievance. Newsrooms are being hit by a surge of automated traffic, mostly from AI developers. AI developers extract enormous volumes of online content and then potentially deprive website owners of referral traffic and ad revenue with generative AI features and chatbots trained on that content. It is a real problem that deserves a real solution. But this bill doesn’t directly target the harms caused by abusive crawling. Instead, it would force anyone accessing a website automatically to identify themselves and their purposes for collecting public information. Rather than protect journalists, the bill threatens a basic tool they, and many others, rely on to gather information online.

Cutting-edge investigative journalism relies on automated data collection. In 2022, a Pulitzer-winning Wall Street Journal investigation analyzed over 31,000 financial disclosures from 12,000 senior government employees, revealing that nearly a quarter of federal officials traded stock in industries their agencies regulated. To conduct an investigation at that scale, the Journal relied on web scraping: an automated process where a software bot, known as a crawler, browses the internet and collects information. This fundamental tool of the open web is now under threat from bills that purport to target “stealth crawling,” but in reality would hamper legitimate public-interest activities, including preservation, journalism, research, and security.

Again, aggressive, large-scale scraping is a real problem — particularly as demand for AI training and content has exploded. It can overwhelm websites, increase costs, and threaten institutions that produce and preserve knowledge, including commercial publishers, Wikipedia, libraries, and the Internet Archive, where one of the authors of this piece works.

In an apparent attempt to reduce or eliminate problematic scraping practices, proposed bills, like the federal Stealth Bot Prohibition Act and New York’s Stealth Crawler Prohibition Act, would impose sweeping prohibitions on anonymous scraping without addressing the real harms.

Both bills define a “bot” or “crawler” as software that “retrieves, scans, indexes, scrapes, or otherwise accesses a website or other internet source.” A bot is “stealth” if it fails to disclose its identity and the “specific nature and purpose of the bot.” Under the federal bill, the FTC can pursue penalties of up to $53,000 if a person uses a stealth bot in a way that is “reasonably likely to damage, impair, or burden” a website’s “technical or commercial operation.” The bill provides no definition or standard of what is reasonably likely to impair a website. Under the New York bill, which only applies to covered news sites, a publisher can obtain a subpoena to unmask a crawler’s identity without requiring a showing of any harm.

Knowing the identity behind a crawler tells us relatively little about whether its behavior is harmful. Under the New York bill, a company that discloses its identity and then accesses the website 10,000 times a minute complies with the law even though it is still imposing enormous and likely unjustified costs on the site. But a journalist anonymously using a crawler to investigate potentially anti-competitive practices by tech companies does not. Many of the most socially valuable uses of crawling require anonymity. Revealing a crawler’s identity and purposes can change the very nature of the information that is collected; if the subject of the study knows it is being observed and by whom, its behavior will change. That is the point of studying it anonymously.

Moreover, not all automated data collection is abusive, and not all of it is used to train AI, but these proposed bills risk deterring archivists, researchers, and civic technologists from using automated data collection tools out of fear of liability. Journalists use scraping to analyze public records, as in the WSJ story. Researchers use it to study online discrimination and misinformation.

The Internet Archive’s Wayback Machine has used crawlers to archive more than one trillion webpages, creating a historical record of the internet that would otherwise constantly change or disappear. That archival record has helped provide the factual background for thousands of news stories, evidence in hundreds of court cases across the country, and often simply a backup for when things go wrong.

It is not just archivists who rely on scraping to advance the public good. Disability advocates have used scraping to automatically evaluate over one million websites to detect accessibility problems. Public health researchers use scraping to track the opioid epidemic as well as misinformation surrounding addiction and treatment. The federal government relies on crawlers to scan for threat detection, offering free cybersecurity services to organizations across the country. And for some of these users, anonymity is essential to the integrity and safety of their work. For example, environmental researchers gathering information from public corporate disclosures or satellite image portals may fear retaliation from the companies they are investigating.

The First Amendment has historically protected us from government-imposed identification requirements tied to receiving information. In Lamont v. Postmaster General, the Supreme Court struck down a law requiring recipients to affirmatively identify themselves before receiving certain political publications. The same principle applies to web crawling: requiring people to identify themselves as a condition of accessing publicly available information burdens the freedom to receive information.

That concern is heightened when laws use vague language. If researchers cannot easily understand what it means to “deploy” a crawler in a prohibited way, or what activity is “reasonably likely to burden or impair a website,” they may refrain from scraping rather than risk violating the law.

As we wrote in our letter to New York Governor Kathy Hochul, “deanonymization threatens the core of the open web without actually addressing these technological and potential economic harms of AI-related crawling.” Unmasking a crawler does not reduce the number of requests it sends to a server. It does not compensate a publisher whose content is used to build an AI product. And it does not restore referral traffic that publishers may lose when users turn to an AI chatbot for information.

Indeed, existing laws can be used to address many harmful uses of scraped data. For example, privacy laws can restrict how sensitive data is processed. Copyright law can respond to infringing AI-generated outputs. Consumer protection law can address deceptive data collection or use.

To tackle the specific issues raised by aggressive AI scraping, policymakers could support the development of standardized, interoperable technical solutions to help websites better manage automated traffic. Standards bodies such as the Internet Engineering Task Force are exploring new protocols for managing machine access to online content and developing mechanisms for anonymous attestation of good crawler behavior. Policymakers should allow that work to develop rather than prematurely imposing legal and technical measures that harm “good” bots and that “bad” bots will quickly learn to evade.

And if policymakers conclude that new legislation is necessary, liability should turn on conduct that causes identifiable harms like overwhelming servers with excessive requests, engaging in fraud, or collecting data for prohibited uses.

The spread of AI crawling has created legitimate challenges for publishers and website operators requiring thoughtful policy responses. But policymakers should resist treating anonymity as something wrong in and of itself. The open web depends on the ability to access, analyze, and preserve publicly available information without revealing our identity and seeking permission from every website owner first.

Policymakers who want to curb abusive crawling while preserving a healthy information ecosystem have better options, and we are eager to help them. We invite legislators at the state and federal levels to reach out to us about how alternative proposals could target harmful conduct while protecting the tools that journalists, researchers, archivists, and citizens have long used to navigate the digital world.