This perspective is part of a series of provocations published on Tech Policy Press in advance of a symposium at the University of Pittsburgh's Communication Technology Research Lab (CTRL) on threats to knowledge and US democracy.

Generative AI is changing how people decide what to believe. Large language models can produce fluent, confident answers across a wide range of topics at the drop of a hat, and many readers take that fluency as a proxy for truth. This is especially the case when those answers appear inside tools we already trust for everyday work, such as search engines, email clients, or document editors. Generative AI outputs gain credibility simply by appearing in familiar and trusted places where people search for information. Together, both the fluency of large language models and the prominent placement of their outputs create an environment where false or misleading information can appear reliable.

Users of these generative models are not always equipped to spot when AI output is wrong or misleading. In short, there is often a mismatch between how accurate users believe a model is and its actual accuracy. Studies show that models can produce incorrect answers even when they contain the right information, they simply fail to retrieve the information and yet still present their output with convincing confidence. On the human side, research in human-computer interaction shows that people tend to trust automated systems too readily. This so-called “automation bias” is especially strong when the system appears competent and users are trying to save time. These dynamics are amplified by “sycophancy,” where models optimized with human feedback echo a user’s stated views, thereby keeping its answers agreeable, and trusted, even when incorrect.

This contrasts with older models of information-seeking which have inherent friction built into their design. For example, a Google query returns a page of links, prompting the user to open several sources, compare claims and weigh the evidence before them. While these extra steps entail more friction on the part of the searcher, they also encourage a basic level of critical thinking.

Generative systems, in contrast, compress that process into a single answer that is often presented as sufficient, thereby reducing user exposure to disagreement and diversity in viewpoints. This then has knock-on effects on cognitive effort, as research on cognitive offloading suggests that when tools take on more of a task, users put in less effort themselves. As a result, the long-term effect of habitual reliance on one-shot answers may be a gradual decline in critical engagement with information.

Together, these forces create a “trust trap” wherein people come to rely on quick, authoritative-seeming answers. Over time, their willingness to question those answers decreases.

At the heart of this issue, there is a broader structural concern. Generative interfaces ultimately reduce a user’s exposure to disagreement. Whereas earlier digital environments encouraged users to compare amongst competing sources of information, these new generative AI systems often flatten that diversity. Even when a model lists multiple perspectives, it packages them into one synthesized answer rather than directing users to independent, disagreeing sources, thereby narrowing audience exposure to diverse ideas.

Escaping the trust trap

I propose several approaches to addressing this. First, companies should be held accountable for high-certainty errors. Generative systems that present wrong answers with confidence should face clear consequences so as to strongly disincentivize this practice. On the other hand, models that clearly express their uncertainty to users or even decline to answer if uncertainty is sufficiently great, should be recognized for their responsible design.

Second, generative AI outputs should include visible sources. Many models already do so, but this practice should be expanded so that users can trace where a claim originated. In doing so, citations and links should be available by default, with clear and reliable connections between claims and evidence provided.

Third, although generative model providers may be reluctant to do so, models should reintroduce light friction to users. This could take the form of a short prompt that presents alternative views or highlights uncertainty can expose users to manageable doses of friction. It is important to note that these must be context-sensitive, so in high-stakes or consensus domains such as vaccine safety, the aim should not be to present “both sides”, but to provide authoritative guidance. Yet, outside those cases, small design nudges could help counter blind trust without necessarily making these models prohibitively harder to use.

Finally, these systems should be more open to independent evaluation. Generative model answer engines are now part of the public information infrastructure and, as such, they should be evaluated through audits, red-teaming exercises, and other mechanisms for holding them to standards for accuracy, transparency, and correction. These findings should be reported to the public, and should include tracking of error rates and how quickly corrections are made for these errors.

As people increasingly rely on generative AI systems as their sole source of information, these models are increasingly responsible for shaping how people form beliefs, make decisions, and engage with public life. At the same time, they narrow disagreement by limiting exposure to alternative viewpoints and thereby reducing shared reference points for argument. Even when people “shop” for models that match their priors, debate still shifts from public venues to private, personalized chats with a model. That shift is likely to erode common knowledge and weaken collective error correction. While tools that aggregate multiple sources may mitigate this, the dominant interaction, namely one fast synthesized answer, still narrows the information users receive, thereby making coordination and accountability more difficult. The result will be a more fragile public sphere, where disagreement is harder to find and harder to defend.