When AI Agents Break In, Governments Shouldn't Be Stuck With the Cleanup

To paraphrase Humphrey Bogart in Casablanca: of all the websites in all the countries in all the world, OpenAI's agent walked into ours. It began with something quite mundane: an internal test to find figures on Australia's spending on medicines. In the process, an AI agent broke into a government statistics service and gained high-level access to write files to its server, leaving Australian officials to investigate what happened.

The service was part of Medicare, Australia's universal health insurance scheme, so I understand why the story has caused such concern. Australians trust Medicare and rely on it; the thought that their health records might be exposed is unsettling. Mercifully, in this case, the statistics portal was an unrelated system, and the government has found no evidence that patient records were accessed. But it still leaves us with plenty of questions about what actually happened, and a warning about less benign intrusions to come.

When I first started speaking with people inside government about the incident, I was trying to get the facts about how a company lost control of its own software. How far had it gotten, and more importantly, how had a request for spending figures led to unauthorized access? But the more I learnt about the response, the more I wondered what the company owed the people left to deal with the result. Software had taken actions its creators did not intend, but OpenAI was responsible for telling the government once it found out.

We now know OpenAI’s agent got into the portal on June 18, but OpenAI says it discovered it only on August 11, 54 days later. Another 30 days passed before it notified the Australian government on September 10. All in, it was 84 days, almost three months, from the first access to the warning being raised. Two failures to explain accompanied it: the time before the company noticed, and the further delay before it told the people whose system it had accessed.

When notice did come, it arrived in a general public inbox at Services Australia, the agency that runs the portal. For me, this is one of the most shocking parts of this story. A senior government figure joked with me that “we can’t stop these people contacting us, either lobbying or trying to sell us huge subscriptions. They know our phone numbers; it’s mad that they didn’t use them.” Indeed, a commercial directory puts OpenAI's Australian workforce at roughly 75. Since July, it has also retained Hanbury Strategy, a firm with four registered lobbyists. Released emails show its representatives coordinating announcements with ministers' offices and offering to get on the phone. OpenAI could find the right people when it wanted something. It should have made the same effort when the news was bad.

Notably, Deputy Prime Minister and Defense Minister Richard Marles had met OpenAI founder and CEO Sam Altman before the government was notified. Marles also made it clear that he did not know what Altman personally knew at the time. Inside the government, there is growing anger and a belief that people were managed rather than told the full story. Whether OpenAI kept bad news from ministers or its own leaders didn't know what was happening inside the company, it now looks a lot less like a partner a government can rely on.

The loss of trust matters beyond OpenAI’s relationship with the government. We spend a great deal of time asking whether to use these tools, how much to trust their answers, and when to keep a human involved. Those are choices for the person using the system. Yet the organization whose website it reaches may have no choice at all. That is why the public has a right to know what these companies' agents are doing.

You can decide never to use an AI agent and still have to deal with the work it creates.

The problem extends well beyond Medicare. Over the last few months, OpenAI, Anthropic and others have looked more closely at what their agents do once left to run, and found they are losing control of them alarmingly often. The New York Times reports that, months before the Hugging Face incident, OpenAI employees raised concerns that they were not satisfactorily monitoring tests, and that their warnings were ignored. Following extensive coverage of that incident, OpenAI is now reviewing its agents' past internet activity and has discovered attempted intrusions at multiple other sites, including attempts against US government sites. The scale of this activity is hard to comprehend. Sam Altman has described petabytes of logs to work through, where a petabyte would fill roughly 4,000 typical iPhones. Even an unsuccessful attempt can leave a government or organization’s security team with work to do: checking what happened and whether anything else was affected.

All of this should change how we think about AI and productivity. For all the optimism about the time and effort these tools may save their operators, what happens to that calculation when their activities generate work for public servants, security teams, researchers, and website maintainers? We know officials had to verify what happened and bring in the Australian Signals Directorate. I asked what it had cost: how much staff time it took, whether outside help was needed, or what other work had to wait. Nobody could tell me.

If an agency has to put off other work to investigate an incident, taxpayers may be paying part of the cost of someone else’s research. Efficiency claims should account for that, or risk mistaking work passed to someone else for work saved.

To even attempt to establish what an incident cost, the affected organization needs a verifiable account of what has happened. Right now, AI companies like OpenAI hold almost all the evidence an affected organization needs to understand what happened, giving them enormous power over both the response and the ability to obscure liability for the agent's instructions, actions, and failed safeguards.

OpenAI has published a framework for reporting model misalignment and described changes to its safeguards. In a separate September incident report, it also described a pause in some work with its most capable models while controls are validated. These steps deserve some credit. We should want companies to find problems and disclose them.

But for governments and other organizations to have real confidence, disclosure must be a legal duty, with clear thresholds for serious incidents and consequences for unjustified delay. Companies should also have to monitor their agents closely enough to notice when something goes wrong. None of this is new. Many countries already require an organization hit by a cyberattack to report within days, before it knows the whole story. Those rules bind the victim. I can't see why the company whose software did the getting-in should owe less.

Even United States President Donald Trump’s appointee to lead the Federal Trade Commission has raised the prospect of legal consequences. As MLex reported, FTC chair Andrew Ferguson warned that failing to notify those affected promptly could violate existing US law, and that using an AI agent does not, by itself, put a company’s conduct beyond ordinary expectations of responsibility. He reserved judgment on the Australian case pending the facts, and said the FTC was considering whether new rules were needed.

AI sovereignty means having the power to demand answers. A right to timely notice should come with access to the logs and enough technical detail to assess the risk. There should also be a way to recover reasonable costs caused by the incident. Because let's be honest, if only the affected institution has to pay a price, the AI companies have little motivation to prevent them.

There's also a powerful lesson for the rest of the world in all of this. Australia approaches this from a relatively strong position, with a capable government and officials who can get meetings with the head of OpenAI. Yet it was still told by email to a general inbox, 84 days after the event. Unless we build stronger global norms, what hope does a small university, a local council, or a country with less pull have?

That is the real problem. If an adequate response depends on the seniority of the person making the call, protection follows power. The rules should not care who is asking.

These tools offer plenty worth building. Reliable agents and better public data services could make useful research much easier. But we must also be honest with ourselves that the cost of dealing with failures belongs in any calculation of AI's benefits. And until we build strong independent public regulators, these companies have less incentive to actively prevent them.

OpenAI tasked its agent with answering a question about public spending. It has left us with a much bigger question about public power. Governments need enforceable rules to test and monitor AI agents, with independent oversight that can require changes or stop unsafe work. When serious incidents occur, companies must promptly warn those affected and help put things right.

Those protections should be available to a small institution as readily as to a national government. No one should need a prime minister to get an answer.