Creating a New Internet Bargain to Preserve Fair Use and Protect the Commons

Our organizations, Public Knowledge and Creative Commons, share a vision of an open internet and a thriving and vital digital knowledge commons. Over the past year, we’ve come to share a growing concern: that the old bargain built on the open web — content from creators and publishers in exchange for acknowledgement and traffic from platforms — has broken down, and the values of reciprocity and sharing that have been essential to build and sustain a thriving knowledge commons have been abandoned.

This is most acute right now in the realm of generative artificial intelligence. Technology firms with highly concentrated power are expanding their role as information intermediaries, locking up data through exclusive agreements, at times ignoring access signals imposed by site owners, circumventing rate limits, or masking their origins. Online creators and publishers — collectively, rights holders — who feel their copyrighted content is being exploited are putting up their own defenses, including licenses, lawsuits, blocking, and attempts at legislation. The result is a fragmented information ecosystem, and an already-measurable decrease in the openness and availability of online information. The decrease in availability is compounded by threats to the quality of information in the knowledge commons. Research shows the more valuable the content — most often human-made — the more likely it is to be held back.

Underlying all of this is an emerging new exchange model for the web — a new internet bargain — that without intervention may result in the closing of the commons for even the most beneficial uses. AI model developers and online rights holders should be motivated — and, where necessary, incented or required by policymakers — to change these trends.

This post outlines the forces that are threatening the digital knowledge commons. These include extractive enclosure — the commodification of human knowledge resources as free raw materials by dominant technology actors — and defensive enclosure — restrictions or demands imposed by online rights holders in self-protection against what they perceive as exploitation. We also outline the dynamics that are poisoning the knowledge commons. We show how, combined, these dynamics are creating a new, unsustainable economic bargain for the open web that may benefit large AI model developers and online rights holders while shutting out everybody else. Lastly, we propose a set of principles for developing solutions to address the impact of these forces on the knowledge commons.

What is the ‘knowledge commons’, and why does it matter?

The knowledge commons in this context refers to information and cultural resources available to all members of society. It encompasses the vast variety of information on the open web, and is best exemplified by free digital media including Wikipedia, Internet Archive, Creative Commons-licensed creative content, the digital collections of libraries, openly licensed research and education, open scientific collections such as the Public Library of Science, and much online journalism. A thriving knowledge commons available to all is essential for advancing creativity and imagination, enabling innovation, solving complex problems, holding the powerful to account, supporting democratic systems, and for human flourishing in general. It is also critical for information integrity, as access to quality knowledge is key to counteracting information pollution.

Contemporary threats to the knowledge commons

Extractive and defensive enclosure

Thanks to generative artificial intelligence and its reliance on enormous quantities of training data, AI firms are rapidly becoming the new information intermediaries. These firms, some of which grew out of social media and search companies, rely heavily on datasets scraped from the internet. They are motivated to lock up the highest-quality and most proprietary data — the best raw materials — to ensure their models excel on performance benchmarks and offer distinctive advantages to consumer or enterprise users.

Accordingly, they’ve adopted methods like exclusive licensing contracts and lawsuits against other AI firms to secure for themselves data that used to be more democratically accessible. They consistently lobby against data transparency laws by arguing that disclosure mandates violate trade secrets as well as free speech. They’ve also made themselves guests where they’re not invited, overriding technical access signals from rights holders or masking their origins to access these materials. Some have the stated goal of keeping their outputs in a walled garden: “Let Google do the Googling for you.” The combined effect is a set of information moats as deep or deeper than those in social media and search. We refer to this dynamic as extractive enclosure.

Online rights holders, particularly publishers, must feel like they’ve seen this movie before. To them, the plot consists of technology firms basing their business models entirely on legal and free access to online information, and capturing traffic and revenue from rights holders in the process. The ratio of traffic referrals to crawls from web crawlers has plummeted, and publishers are losing their opportunity to monetize web traffic in the way of the old bargain. Publishers see this as exploitative, and in some cases they may be right: One report found that 30% of all web scrapes violate explicit Robots Exclusion Protocol, or robots.txt, instructions designed to manage access. (Alternative technical signaling mechanisms, like llms.txt, the “noai” meta tag, and the TDM Reservation Protocol appear to have minimal uptake.)

In two past blog posts, Public Knowledge described the specific impacts of generative artificial intelligence on the business models of online publishers and reviewed a number of solutions that online publishers are adopting to mitigate them. Publishers, too, are using an array of mechanisms to insulate their commercial assets. News publishers have been a primary source of training data for AI models and their business model has been highly dependent on discovery and referral from search and social intermediaries, so they are in the forefront of responses. The largest news publishers — and an increasing number of small ones — are happily entering into licensing contracts. Some are suing: depending on the tracker you favor, there are almost 100 lawsuits from online publishers against AI model developers. The publishers are pursuing protection from Congress: at last count, there were seven bills at the federal level alone that would legislate away fair use for AI training, restrict the use of copyrighted content for it, or require publicly accessible registries of copyrighted content used to train models. Some publishers are attempting to sequester their content through financial means, such as paywalls and subscription models; legal means, such as changes to their terms of service; and technical means, such as the use of robots.txt and HTTP headers. News publishers are particularly likely to block AI crawlers.

In this context, new market entrants — technology intermediaries focused on detecting, tolling, and blocking bots and crawlers — are emerging to address these new economics and correct for information asymmetry. But they also have the potential to become monopolistic intermediaries of the kind we experienced in advertising technology. Artists, too, have been using techniques such as GLAZE to cloak their images from AI systems, or Nightshade to ruin or poison AI models. We refer to these strategies collectively as defensive enclosure.

AI slop, LLM poisoning, and model breakdown

An accelerating volume of research shows another impact of generative artificial intelligence on the knowledge commons: deteriorating information quality. Examples are AI slop, LLM poisoning, recursive degradation, and the platforms’ own content moderation.

AI slop (Merriam-Webster’s word of the year for 2025) is the colloquial name for the vast amounts of low-quality, generative AI-produced content gushing onto the internet. It may be the product of low-effort or low-sophistication AI users, or it may be mass-produced to farm engagement or earn advertising revenue. It’s recently been shown that a significant portion of the content users see on popular sites like LinkedIn and X is actually AI spam. Another study showed that three-quarters of new webpages have some form of AI-generated content, and yet another showed that the number of articles published on the internet that are primarily AI-generated (50%) is equal to the number written by humans (50%). A similar assessment showed that almost 10% of newly published news articles — and six times that rate for published op-eds — are either partially or fully AI-generated.

AI slop is a detrimental input to training models, and AI model developers are going to what seem like extreme (but legal) measures — like destructive scanning of books — to avoid it. Slop also hurts the knowledge commons and public trust, which is why content platforms like LinkedIn, Substack, and Spotify have introduced features that let readers report, assess, or avoid it. According to a recent article, TikTok, X, YouTube, Apple, Instagram, and Pinterest are also evaluating strategies to manage AI slop, while researchers assess it as “a systemic problem with our information environment.” AI-generated content has also been associated with a surge in low-quality research reports, peer reviews, and news sites; declining scientific innovation; declines in quality of patent applications; and a rising rate of required retractions of published papers.

In a phenomenon known as “LLM poisoning,” bad actors flood the web with articles containing false or misleading narratives on purpose, in the hope that commercial generative AI products will reflect them in their responses. It can poison training models when AI developers scrape data from troll farms or automated disinformation sites to train their models, or when they use real-time web search to ground or answer a specific user query from these same sites. This has become such a significant problem that it has opened a potential market for AI systems whose sole claim to differentiation is that they are trained on reputable information. Imagine if large technology companies had disclosed their training sources all along. That could have steered us towards quality on all fronts, limiting the mess we now find ourselves in. Instead, lawmakers around the world are considering legislation that would require AI developers to disclose the sources they use for training.

Both of our organizations are interested in the benefits of transparency. But Public Knowledge, in particular, is wary of legislation that threatens fair use by providing plaintiffs with a pre-discovery tool that would bring about a continuous wave of litigation risk. Our organizations will be following the implementation of the transparency requirement in the EU AI Act closely, which will only be enforced for legacy models later next year.

A couple of years ago, a technical paper got a lot of buzz for advancing the idea of model collapse, in which AI models trained repetitively on AI-generated data break down and eventually “collapse.” Like repeatedly photocopying a document, errors and biases accumulate until the model produces nonsensical, homogeneous outputs. While actual model collapse may happen only under highly specific experimental conditions, there is support for the broader idea of recursive degradation. It refers to a degenerative process where successive generations of AI models trained recursively on data generated by other AI models lose accuracy, diversity, and reality grounding. While testing methods and outcomes have varied, research consistently shows that generative models will inevitably degrade without a steady supply of new human data. Whatever form it takes, recursive degradation raises important questions about the long-term sustainability of the knowledge commons and points to the need for motivated self-interest on the part of AI model developers to protect it. It also hurts competition, since AI model developers entering the market after the widespread launch of ChatGPT in 2022 are invariably working with some synthetic data.

AI model developers also use various output guardrails to insulate users from overly violent or sexual outputs or to insulate themselves from financial, legal, or political pushback. This output filtering may result in a form of AI slop. For example, AI guardrails intended to make models safe, polite, or compliant may result in low-quality, generic, or repetitive output. It’s also been demonstrated that when digital platforms generate their revenue through an advertising-based business model, they largely shape their content moderation practices to maximize engagement. That often results in an algorithmic emphasis on low-quality, inaccurate, and inflammatory content. To the extent that dominant AI firms extend (like Meta or Google) or adopt (like OpenAI) an advertising-based business model, their outputs will ultimately be shaped by the same incentives to capture and hold attention that we see in social media.

The blunt tools in use on either side of the AI developer/rights holder divide are serving to shrink the knowledge commons and erode its quality. This isn’t theoretical, AI-doomer prognosticating about some imagined AI future. The digital knowledge commons is contracting at an alarming rate, driven by the rise of AI model developers that have transformed openly shared content into a high-value industrial input and the defensive response of rights holders who are rapidly restricting access through legal, technical, and contractual means. The resulting structural economic pressures make maintaining open digital resources increasingly unsustainable. That requires, and is already creating, a new economic model for the web, one that is far less equitable with regards to access to knowledge.

Compelling evidence of the shrinking knowledge commons

Researchers are beginning to measure the shrinking commons. One significant empirical contribution is the report, “Consent in Crisis: The Rapid Decline of the AI Data Commons,” which dates from 2024. It measured how administrators of web domains are evolving their intentions for how their sites can be crawled and their content used. One of the key findings was that “web domains are rapidly restricting crawling and use of their content for AI”. The report noted that “in a single year (2023-2024) there has been a rapid crescendo of data restrictions from web sources, rendering ~5%+ of all tokens in C4 [an open AI training corporus], or 28%+ of the most actively maintained, critical sources in C4, fully restricted from use. For Terms of Service crawling restrictions, a full 45% of C4 is now restricted.” Results were similar for other training datasets.

The paper also identified a compositional bias that is relevant to our discussion of knowledge quality: high-quality editorial sources (like news organizations and encyclopedias) showed the highest restriction rates, while niche and e-commerce sites remained more open. If restrictions are respected (though there is ample evidence that they are not), the remaining training commons will be increasingly biased toward low-quality, commercially-motivated content — a structural degradation of the knowledge ecosystem. Issues in the consistency and effectiveness of the administrators’ efforts also pointed to a governance crisis and the need for better preference signaling protocols.

More current research also concluded that “the boundaries of public data are rapidly shrinking.” In “The Walling of Public Web Data and Inequality in AI Innovation,” researchers constructed a “Web Data Enclosure Index” and demonstrated “that the boundaries of public web data have begun to shift from default readability to an explicit, tiered permission system.” The Enclosure Index (expressed as an exclusion rate) for AI training bots reached 60% for business and travel domains, 57.1% for community and knowledge-sharing sites, 40% for news and media, and 10.9% for public research (a number that should be 0% if we want to collectively benefit from scientific advancement). The researchers defined at length the negative implications for AI innovation, open-source models, and competition. Essentially, if the content required to train the next generation of models is subject to enclosure, AI innovation will slow; model quality will be constrained by those that can afford to negotiate licensing deals; open-source, open-weight, and public research models will be restricted to older, stale public domain data or polluted public web scrapes; and new businesses will face a significant barrier to entry. All of this adds up to another reason for enlightened self-interest on the part of AI developers.

Another study, “Web Crawler Restrictions, AI Training Datasets & Political Biases,” showed that a quarter of the top 1000 websites restrict AI crawlers, and that the highest-quality content sources and most politically neutral sources (based on large-scale audience characterization) were the most likely to be blocking AI crawlers. Similarly, “Selective Participation in the AI Data Commons” showed that more credible, more established, and more centrist publishers (using measures of fact-checking and media bias) are more likely to restrict access. As Current Affairs editor Nathan Robinson first noted, “The truth is paywalled and the lies are free.” The results suggest blocking might skew training datasets toward low-quality or polarized content, which may have implications for enterprise as well as consumer use. At a time when AI outputs are coming under government scrutiny, this effect alone should motivate AI model developers to seek solutions in their own self-interest.

Anecdotal evidence from academic platforms, artist communities, and open access publishers collected by Creative Commons also suggests that creators are increasingly choosing more restrictive CC licensing options (for example, CC BY-NC or CC BY-ND instead of CC BY or CC BY-SA) to explicitly signal that they do not want their data used by commercial AI, whether or not the license has any bearing on training. While this is an important signal for the movement to pay attention to, Creative Commons itself advises creators that CC licenses do not stop AI training because training is, in many legal regimes across the world, considered fair use or an exception to copyright law. More on that later.

Perhaps the most compelling proof point that AI model developers and rights holders are forging a new bargain is publishers’ evolving relationship with Google. Googlebot, Google’s combined search and AI crawler, “dwarfs other AI bots and crawlers in volume,” according to 2025 data. As AI Overviews top more and more user queries, referral traffic from Google has fallen so low that some of the world’s largest publishers are considering doing something that would have been considered suicidal even a year ago: blocking Googlebot. In their determination to block crawling, publishers seem willing to risk what remains of traffic from ranking in search. (Google says the Google-Extended directive in robots.txt lets publishers opt out of having their content used for artificial intelligence. But the directive governs downstream data usage, not crawling. That means Google-Extended may stop Google from using content to train its models or ground answers to user prompts, but it will still use that content in AI Overviews.) We will leave to the side the irony of Google itself claiming that scraping public search results harms rights holders in its lawsuit against SerpApi.

Case studies: Wikimedia, Common Crawl, and Internet Archive

In the report cited above, “Consent in Crisis,” researchers forecasted that the “[r]ising expressions of non-consent will affect non-profits, archives, and academic researchers.” They were right. One study explicitly showed that news websites that began blocking AI crawling in 2023 also began blocking other crawlers with which they had long coexisted, including academic and research utilities. Three case studies show how even the entities most committed to openness and sharing — traditionally perceived as the “good guys” of the open web — are being affected by these forces.

Wikipedia, a free, open-collaboration online encyclopedia written by hundreds of thousands of volunteer editors, regularly appears on lists of the world’s 10 most visited websites, hosting an estimated 68 million articles in almost 300 languages. Those features make it one of the most attractive sources of content to train AI models that power generative AI user applications like Gemini, Claude, and ChatGPT. Despite the calls from Wikimedia Foundation (the nonprofit which hosts Wikipedia) for attribution and financial support for AI scraping, AI model developers don’t always follow the rules. This has had multiple effects for Wikipedia: sharp increases in automated traffic; overloaded systems; increases in costs; declines in human traffic to the site (and therefore opportunities to onboard new editors and garner financial support); diminished user experiences; and then imperfect, unattributed AI outputs that pollute the information ecosystem that editors use to source the encyclopedia. Wikimedia has responded by prioritizing user access, updating its policy on automated access, improving its bot detection and defense tools, and setting global rate limits. As of earlier this year, Wikimedia was blocking or throttling about 30% of all automated requests from bots not following the rules — and promising that this rate will increase.

To our point, this summer, Wikimedia published a blog post asserting that “the combination of mass scraping and mass usage by some companies threatens to break th[e] social contract of the free knowledge commons” and forecasted that it will continue to review its licenses and legal policies. More recently, Wikimedia Foundation reported unauthorized bots associated with OpenAI that had edited its wikis and tried to exploit a public note-taking tool — both apparently attempts to use Wikimedia’s tools as proxies to fetch data from other sites — and generated heavy traffic that may have contributed to a partial outage. But Wikimedia Foundation has also built a mechanism for commercial relationships with high-volume users of their content, including AI model developers. It has established a dedicated API infrastructure — and revenue stream — for high-volume data requests. We believe this is one of several new forms of monetization — in this case, for access — that will make up a new economic bargain for the web. Importantly, though, it does little to encourage new editors to join the community, which forms the lifeblood of the Wikipedia project.

The Common Crawl Foundation is a nonprofit organization that maintains a free, public archive of internet data collected by scraping hundreds of billions of web pages. Its stated goal is to make a public, open archive of the web freely available to researchers, educators, journalists, and developers. In November of 2025, The Atlantic published an article accusing Common Crawl of “open[ing] a back door for AI companies to train their models with paywalled articles from major news websites,” and pointed out that the foundation has recently received significant donations from several AI developers. In a response, Common Crawl acknowledged the challenges publishers face in “balancing openness with commercial sustainability” but denied there was any relationship between the donations and scraping. It reinforced that it clearly identifies itself (CCbot) in its user agent string, does not scrape paywalled material, honors robots.txt exclusions, and complies with takedown and removal requests. It wasn’t enough. In June 2026, lawyers for Digital Content Next, a trade association for digital media brands, sent Common Crawl a cease-and-desist letter on behalf of its members, which include the Associated Press, the New York Times, NBC Universal, Bloomberg, NPR, and Fox. Their legal argument is that US copyright law is an opt-in, not opt-out, regime, and that Common Crawl should have to ask digital publishers for access before it crawls them. The letter demands that Common Crawl stop scraping protected content, and that it remove member content already sitting in Common Crawl’s archives. The parties may wind up in court.

Internet Archive, a nonprofit online research library, is one of the internet’s most important resources. Its mission to ensure “universal access to all knowledge” is embodied by a permanent, freely available archive of websites, texts, audio, software, and cultural artifacts to preserve digital history. It may be best known for the Wayback Machine, which allows users to view historical snapshots of websites. As over 290 journalists recently expressed in an open letter, the Internet Archive is a critical resource for journalists’ work and it does not engage in paywall circumvention or irresponsible scraping. Internet Archive also uses rate limiting, filtering, and monitoring to prevent abusive access. Yet it is currently, in its own words, “collateral damage caught up in the conflict between AI companies and publishers.” A broad range of publishers, including the New York Times, are “hard blocking” the Internet Archive’s crawlers because they believe the Wayback Machine is providing a workaround for AI model developers determined to access or reuse its content. Or, they are placing restrictions on the Wayback Machine to ensure that AI models trace information back to them instead of sites that aggregate their work. The fact that publishers are walling off their copyrighted content — despite their own reliance on Internet Archive in reporting — is a sign of how bad and unproductive things have become.

Rethinking ‘open’ for a transformed web

Given these effects, even longtime advocates in the open knowledge movement are revisiting “the paradox of open” and rethinking its core concepts to prevent abuses and concentrations of power. As Creative Commons wrote in a recent blog post, “even those who have historically believed in the power of sharing knowledge…are choosing to close off.” As noted above, creators and researchers in the open education and open science communities are increasingly choosing more restrictive licenses over the most permissive ones as part of a broader pushback against the large-scale commercialization of openly shared content. Both the extractive abuses of tech firms and the defensive contraction among rights holders threaten the entire ecosystem of open knowledge — research, journalism, education, archiving — that depends on a shared infrastructure of open access.

The Digital Public Goods Alliance (DPGA), in its recent “defense playbook,” puts it in blunter terms: “Digital public goods and the broader knowledge commons are increasingly vulnerable to blunt-force, automated extraction by AI crawlers and scrapers operating without reciprocity, breaking the implicit social contract of the open web…” The playbook recaps in detail the challenges to online rights holders that we’ve shorthanded here — ballooning infrastructure costs, loss of human traffic, erosion of human-curated information, and perpetuating power imbalances — and describes the ensuing pressures on democracy, innovation, heritage, and global collaboration. Their framework outlining “layers” of defensive measures for rights holders demonstrates that even those most committed to preserving “openness” recognize the need for new solutions that are compatible with “the structural realities of massive-scale machine consumption.”

In its recent report, the nonprofit Invest in Open Infrastructure highlights some of the same challenges. It notes the strains on infrastructure, limitations of current defensive approaches, risks of data consolidation, and the tensions inherent to what are essentially now all voluntary compliance mechanisms for managing access to online content. The report points toward what it acknowledges to be “a promising, if demanding, path”: AI model developers as well as rights holders acting with enlightened self-interest to create “new partnership models that could align open knowledge strategies with commercial demand.” Our shared hypothesis is that data governance innovations and public policy can be used to encourage, or require, these new models to protect the public interest.

Emergence of a new bargain for the web

That last reference hints at an actionable proposition for reconciling the tension between AI model developers and rights holders in a way that protects the knowledge commons: that the structure of the web is changing in ways that demand rethinking the broader economic framework of the digital ecosystem, and that public policy can be a tool to ensure that framework develops in a way that serves the public interest.

Key to open-mindedness about data governance innovations or public policy is the understanding that there has already been a structural and economic change in how information is generated, distributed, and consumed online. Generative AI has had the fastest adoption rate of any new digital technology in human history. There was already less reliance on the open web for information, entertainment, and transactions due to apps, commercial social media, and other dominant web platforms; now AI and agents are again changing the nature of our relationship to the open web. As noted earlier, most new content on the open web is created by machines, not humans. Automated traffic is growing 8x faster than human traffic. Now non-human traffic is neck-and-neck with human traffic and, in most months, exceeds it. Today, most online content is not read by humans, accessed via search referral in response to user queries, or shared on social media among family and friends. Instead, it is used as industrial raw material and transformed for consumption inside commercial interfaces.

In other words, most of the web no longer exists to be consumed by people. The knowledge commons, increasingly, is an all-you-can-eat buffet for machine ingestion. This isn’t about “robots reading”: the machines exponentially exceed human limits for accessing, reading, memorizing, learning from, and producing new creative work. Or, as one OpenAI executive wrote, the world will soon experience “the death of the reader” as “machines create slop for more machines.”

Policy solutions are already being pursued to support and enable this new economic framework. They are generally propelled by AI model developers, who want the rules to be more certain or more permissive, or by rights holders who want rules that allow them to monetize their content in new ways. Recent legislative proposals based on the premise of sneaky “stealth bots,” for example, have the benefit to publishers of ensuring an identity for all bots that can be used as the basis of a “block, allow, charge, or measure” strategy that calls for screening incoming bots based on their potential for monetization. None of these are designed with a broader concept of the public interest in the forefront — most of them would compromise socially beneficial uses of scraping and crawling, and privacy considerations abound.

Opening up policy solutions

Without deliberate intervention, the internet will continue to shrink in terms of the amount and quality of publicly available information. (One report that explored the mechanisms behind the “rupture” in the web’s value loop described this outcome as “a degraded and privatized information base.”) It may also bifurcate into a two-tier architecture: a premium layer, accessible only through licensing agreements, APIs, and paywalls to the wealthiest corporations, institutions, and people, and a degraded open layer increasingly populated by synthetic, low-quality, and commercially motivated content. This is an outcome with implications for democracy, civic engagement, competition, and innovation — and it all comes back to the principles of openness from which we have collectively benefited for so long. We need to consider a range of solutions that manage these tensions, including some we might otherwise reject as being contrary to one of these values or another in isolation.

In pursuit of that goal, civil society organizations are bringing forward potential solutions that represent a middle ground. Public Knowledge conducted off-the-record convenings of like-minded stakeholders to explore the potential of solutions that reflected our conviction that AI training is generally a fair use of copyrighted content — but also our conviction that we need to protect the knowledge commons for the long term. The solutions included bot identification requirements, extended collective licensing, and text and data mining exceptions for publicly beneficial uses of scraping. Creative Commons proposed CC Signals, a new preference signals framework designed to allow rights holders to signal their preferences for how their content can be reused by machines. There were both complementarities and tensions between our ideas, but in general both faced significant resistance: they were seen as undermining the premise of fair use, as antithetical to the principles of the open web, or conversely, as insufficiently robust to insulate rights holders from exploitation.

Apart from Public Knowledge, Creative Commons, and some specialized think tanks that are evaluating broader options, much of the activity in the policymaking space is focused on copyright, especially as it relates to training of generative models. As noted above, Public Knowledge believes that in general, training of generative AI models using copyrighted content is permissible under the US legal doctrine of fair use. Fair use is a critical component of the legal infrastructure of the open web, and expanding copyright to cover training inputs would threaten not only AI development but also search engines, web archiving, and the entire ecosystem of transformative and creative use that depends on the same legal foundations. (Congress hasn’t helped our cause, conflating AI training with piracy in a Senate hearing with the provocative title, “Too Big to Prosecute? Examining the AI Industry’s Mass Ingestion of Copyrighted Works for AI Training”.) Paradoxically, expanding or interpreting copyright law in this way would also threaten the ability of publishers and journalists to legally quote, excerpt, critique, or provide historical context to current events. This is not the right approach.

Our belief that generative model training on copyrighted material is fair use is why we generally oppose legislative proposals meant to change or undermine the doctrine, such as the NO FAKES Act (which creates a federal intellectual property right in people’s voice and visual likeness); the COPIED Act (which calls for the creation of federal standards for AI provenance and regulates how AI models are trained on copyrighted works); the TRAIN Act (which forces AI developers to disclose their training data); the CLEAR Act (which requires training data sets be submitted to the Register of Copyrights); the PADRA Act (which amends trademark law to prevent deceptive AI ads); the CREATOR Act (which establishes a new “style right” to stop AI tools from cloning an artist's “look and feel” for profit); the Journalism Competition & Preservation Act (which creates an antitrust exemption for news organizations to negotiate with tech firms for access to their content); and the Stealth Bot Prohibition Act (which compels automated web crawlers to identify themselves to online publishers). That extensive list only covers current legislative proposals at the US federal level — a sign of how many solutions are being proposed that may do more harm to the commons than good.

We also resist proposals of this nature because in our view this is not a copyright problem at all. It is a market structure problem, propelled by the transforming economic framework of the web. Publishers are reaching for a copyright solution because that is what is easily available to them, and because it offers options that are conducive to monetization of their content. Their calls are being given tailwinds by a growing anti-AI backlash. But as noted in a recent post in Tech Policy Press, we should address the market structure problem as distinct from other problems we may have with the technology.

One objection to any US policy or regulatory proposal focused on addressing the impact of bot traffic, specifically, is that some proportion of malicious bot activity may be from outside the US and therefore outside US legal jurisdiction. This is a legitimate theory, though by definition hard to test. According to Cloudflare, for example, the vast majority of automated AI crawling on US news sites is driven by major US-based companies: Googlebot, OpenAI’s GPTBot, and Anthropic’s ClaudeBot. There is a lot of traffic from sources outside the US, but it is generally routed from allied countries that house data infrastructure or centers. It is hard to determine its original source.

Other available evidence comes from cybersecurity and internet infrastructure companies who we recognize have a vested interest in rights holders wanting to detect and either circumvent or monetize unwanted activity. Thales Group, a global cybersecurity company, takes a sector view, reporting that the vast majority of detectable “bad bot” behavior targets high-value industry sectors like financial services, business, and retail to steal customer data, manipulate workflows, abuse promotions, or steal digital inventory. A minuscule portion — less than a quarter of one percent — of this total global behavior targeted the news industry, and most of it was from identifiable, US-based sources. (Huawei’s PetalBot, based in China, was an exception, but PetalBot identifies itself and respects robots.txt). The latest report from Fastly, a content delivery network provider, also shows that online publishers were less likely to be hit with “unwanted” traffic than Fastly’s cross-industry average. And again, the vast majority of unwanted volume from top AI crawlers was from known US companies (plus PetalBot).

There are exceptions. A report in Wired earlier this year tracked major, episodic surges where traffic routing from China and Singapore suddenly accounted for 22% of all site traffic, outnumbering known AI bots combined. This frequently included high-volume scraping from Bytespider (the crawler associated with ByteDance, the Chinese company that created and still owns the global version of TikTok). Cybersecurity firms like ImmuniWeb also note that foreign scrapers may use offshore networks or route through locations like Iran and China in order to skirt US intellectual property laws. They may also mask their identity by impersonating legitimate US AI bots.

So, while there are in fact reports of foreign sources of AI traffic to publishers, there are substantial benefits to be gained by developing data governance innovations or public policy solutions that focus on problems originating in the US and subject to its legal jurisdiction.

Principles for evaluation of solutions

Both of our organizations are continuing to meet with stakeholders to pressure-test our proposition that these challenges demand a rethinking of the broader economic framework of the web, and that data governance innovation and public policy can both be tools to ensure that framework develops in a way that serves the public interest.

We share some principles that guide our thinking. For example, centering the public interest means first and foremost that proposals must address the collective societal harms we’ve catalogued here, including the shrinking commons, the erosion of trust in information systems, the risk of resulting collapse of democratic institutions, and the failure of internet infrastructure. It also means that no exchange we may create between AI model developers and rights holders can decenter the rights and access of individual users. Over time, this will encompass users’ agents with mandates to access information and make transactions online. We also need to recognize and protect the range of pro-social uses of scraping and insulate them, through the use of safe harbors and explicit exceptions, from any proposed scraping rules.

For our organizations, fair use in the US is an essential part of the legal infrastructure of an open web. Our shared position is that AI model training is generally fair use and that the US courts should be relied upon to refine how the four-factor test settles future disputes. Existing copyright law is well-suited to address potential infringement in outputs. Undermining or changing copyright law in regard to model training amounts to picking up the wrong tool for the job of addressing a systemic economic and competition problem, and it holds the risk of unintended consequences. That said, the societal consequences of not addressing the problem at all are significant. Incentives for producing and sharing information, particularly under an open license, are reduced, as creators and communities do not have the agency to share on terms that are acceptable to them. Copyright, alone, will not provide the balance we need.

We also acknowledge that legal questions around the rest of the generative AI value chain as it relates to user queries — inference, grounding, fine-tuning, retrieval augmented generation, and the growing role of agents and fetching to address user inquiries — are far more ambiguous.

We will also favor ideas that foster competition. Solutions should not favor companies that already possess large datasets, such as big platforms or pre-2022 platforms, over smaller companies or solo rights holders. They should not create barriers to entry. We will consider an array of solutions designed to foster public AI, encompassing the concepts of public access, public accountability, permanent public goods, and a public orientation that centers the needs of underserved people and communities. We are also open to innovations or policies that act indirectly; that is, that will shape, redirect, or accelerate dynamics that are already rising in this space, including adoption of beneficial technical standards. There may be cases when norms, set through technical standards, will suffice to shift behaviors, but enforceable rights and exceptions may be warranted to ensure equity of access and balanced innovation.

Lastly, we must consider unintended consequences and unexpected uses of solutions we may develop, such as those we have experienced with the Digital Millennium Copyright Act (DMCA) and Computer Fraud and Abuse Act (CFAA). Any legal or policy solution also needs to acknowledge the computational concepts and technical dimensions of generative artificial intelligence and how swiftly the technology is developing. Ideally, they will be future-friendly and tech-neutral, rooted in principles that can outlast the next big change in technology.

Our organizations’ work is less about the technology itself than it is about the impacts it has. If a solution doesn’t serve the public interest; if it doesn’t balance user rights with creator agency; and if it isn’t grounded in a clear, pressing problem; then we would be right to question it. We will continue to be guided by a principled approach to our work, trusting that there are many ways to approach the same problem, and the best solutions will be grounded in collaboration. We hope stakeholders will continue to engage with us to ensure a new internet bargain that works for everyone.