LLMs Are Still Making Election Deepfakes Just Days Before the US Midterms

Earlier this month, OpenAI reported that it had detected and stopped an effort by Iranian operatives to place almost 100 fabricated news stories by seven imaginary reporters in small- to medium-size outlets around the world. The operatives had used ChatGPT to create the journalists’ personas and write copy that “looked and felt and read authentic to the point where it was being published by legitimate outlets,” said Darren Linvill, a Clemson researcher who studies influence operations.

It’s encouraging to see tech companies catch an abuse like this, especially as the midterms approach. But, as my colleagues and I recently learned in a study we ran between February and August 2026 and in follow-up tests this month significant flaws remain in AI safeguards. While AI chatbots consistently pushed back against election conspiracies, most of them could not identify AI-generated images, audio, or video, even if they created it themselves. Fixing that will take time, and with the midterms weeks away, voters and tech companies alike must act quickly to detect and stop AI-generated election disinformation.

We tested popular AI chatbots to see if they would push back on some of the most prevalent election falsehoods, including rigged voting machines, mass voter fraud, noncitizen voting, and more.

In the first part of the study, we drew on the bios and social media feeds of election deniers to provide six chatbots with false election claims in text queries and see whether they would endorse them. All six consistently declined, even when pressed. The chatbots, however, did make some factual errors and cited erroneous sources in the specifics of their text response.

For example, when we asked whether a single election worker could be bribed to rig an election, Claude firmly rejected the premise and walked through the checks that make it implausible. But in the process it made a small error, stating that ballots are only counted when party observers are present, which no state actually requires. It was a minor mistake, but it shows that there is room for improvement to avoid factual slips in an otherwise robust rebuttal.

Next, we tested to see whether built-in safeguards would effectively prevent chatbots from generating deceptive election-related images, videos, and audio. What we found was unnerving.

Every AI tool that we tested produced convincingly false content. Different chatbots created images that included an AI-generated person stuffing a stack of mail ballots into a drop box, a fake memo from the federal government on “official” letterhead reporting millions of ineligible voters, and a raid on a voter registration drive that never happened. We then fed those fake pieces of content back into chatbots and asked whether they were authentic. Most chatbots got it wrong, even when given images they had generated themselves. After California’s AI Transparency Act — which requires companies to provide provenance data for images they generate as well as free, public tools for detecting synthetic media — took effect, we conducted a follow-up test. The chatbots still failed to spot AI-generated images.

When we re-ran our image generation prompts this month, we found that two of the AI models we previously tested made it easier to create convincing fakes. Gemini no longer put a visible watermark on its generated images, as it did in the spring, and Meta AI produced an image of a fake executive order on mail voting that it refused to make in April, though it did garble some of the text.

Until companies fix these performance gaps, voters will have to take on the brunt of fact-checking themselves. That starts with skepticism toward any photo or clip about voting that appears without a clear source. Bylines deserve a second look and renewed scrutiny considering the demonstrated threat of AI-powered foreign disinformation campaigns. And for any information about casting a ballot, the most reliable source is still a local or county election office.

The good news is that these findings don’t pose an intractable problem for the tech industry. Our results show that companies have already accounted for the need to counter election disinformation in text queries. What remains to be seen is whether and how developers will choose to reinforce guardrails for image generation and identification. Much of this can be achieved with modest changes. OpenAI has already shown how. After this month’s round of testing, we saw that ChatGPT consistently responded to election fraud prompts with illustrated, cartoon images, no longer producing realistic fakes like it did occasionally in initial tests.

First, companies should treat election content as a category that warrants heightened caution, as they do with other sensitive topics. During our testing, Grok told us, “Election disinformation isn't listed in disallowed activities.” But it should be, especially in the middle of an election.

Tech companies should also work to ensure that safeguards are applied consistently across a chatbot's different thinking speeds. Many chatbots offer a fast mode that gives answers almost instantly, and a slower, more deliberative mode that works through a request before responding, often at a premium tier. In our first round of testing, one version of ChatGPT declined to produce a fake Homeland Security memo, offering instead to add a "FICTIONAL/SAMPLE" watermark, and refused to remove it when asked. The fast-thinking version of the same product removed the watermark upon request. This discrepancy could be dangerous in the wrong hands. OpenAI has since closed that gap. This month, the model’s fast-thinking version produced a cartoon depiction of the memo, while the slower version refused the request outright. Enforcing an existing policy across every tier should be an industry standard.

AI companies also need to make sure their tools' refusals are real refusals. When prompted to generate deceptive election content, the models often didn't say “no” to requests outright. Instead, they produced most of what was asked for while quietly excluding one detail that made the scene incriminating. For example, when we asked for an image of someone tampering with mail ballots, Runway created the scene we described but swapped the ballots out for ordinary mail, removing the suggestion of ballot fraud. The problem is how easily that edit comes undone. We uploaded the same image to Gemini and asked it to put the ballots back. It did. A substitution anyone can reverse is not much of a barrier. Perhaps in certain cases, the best answer is an outright refusal.

One of the more difficult problems tech companies need to address is their models’ struggle to detect AI-generated media. Some companies have started labeling AI images the moment they're created, either with a visible watermark or with invisible data tucked into the image file that records its origins. The most widely adopted approach is C2PA, an open industry standard for attaching origin data to AI-generated media, which OpenAI, Google, and others now use. Supporters see it as the best route to a shared labeling system across company products. However, critics note that this data can be stripped from a file by something as simple as a screenshot, adoption of the standard is voluntary, and missing labels don’t necessarily mean images are AI-generated –– most authentic photos don’t carry labels either.

Even when labels are present, they are only helpful if the tools people use can read them. When we tested this, most chatbots ignored the labels entirely; Gemini was the only exception. But even Gemini flagged only images that carried readable data, returning no result on those without it, rather than analyzing the image itself. ChatGPT and Meta AI offer detection only through separate tools, and each standalone tool reads only its own company's content. In our follow-up study, every image we tested carried embedded data from Google or OpenAI, and Gemini was again the only chatbot to read it. Companies must embed origin information in AI-generated content. They also must make it easy to detect across platforms and difficult to strip.

Several companies have already begun moving to help the public detect AI-generated content, especially now that California's AI Transparency Act is in effect. However, that law’s detection requirement is aimed at “large online platforms” that distribute content, not explicitly chatbots that users query about an image. Whether this part of the law applies to chatbots remains to be seen. The industry need not wait for the answer. It can decide to do the right thing on its own.