UK Wants to Lead the World on AI Safety. It Can’t Decide Where To Start

Evie Breese is a UK Reporting Fellow at Tech Policy Press and openDemocracy.

Andy Burnham, the nascent United Kingdom prime minister, has in recent weeks sought to position the UK as a global leader in AI safety. At his first party conference as Labour leader, Burnham said that he will use the UK’s G20 Presidency next year to put AI “center stage,” followed by the development of a “new global code to capture its benefits whilst being clear-eyed about its risks.”

Also attending the Labour Party conference last month — his first as a member of the Cabinet — AI Minister Kanishka Narayan made the somewhat surprising pronouncement that the UK has already effectively banned superintelligence. This would be a world first, given that no country currently has superintelligence-specific laws (though proposals have been put forward in the UK, EU and the United States).

Narayan explained that the UK does not have the physical infrastructure, such as energy or data center capabilities, “to build superintelligence in this country,” and besides, copyright laws make it illegal. “...It is currently illegal to develop a frontier large language model based on the transformer architecture, given the copyright position that we have,” he told the Politico Pub. Therefore, he argued, he is not going to “do a bill for the sake of doing a bill.”

Pressure on Narayan and Burnham’s government to legislate for frontier AI has built in recent weeks. Some have pushed even further, focusing their calls on a hypothetical form of AI they believe could surpass or outmaneuver human control, called “superintelligence,” brought into widespread public consciousness by the resignation of Anthropic researcher Jacob Coxon, citing the failure of AI companies to “act responsibly.” Adding fuel to the fire were increasing reports of AI agents created by US-based frontier AI labs acting in unpredictable ways to conduct cyberattacks on government bodies in Australia, the US and Canada.

UK legislators are caught between two competing priorities: preventing future catastrophe and addressing present harm. Calls to mitigate existential risks — conceptualized by Coxon as the belief that “AI could kill us all by the end of the decade” — appear urgent and acute. But AI has already caused tangible harms, from a false intelligence report that could have escalated into warfare to widespread negative impacts on mental health.

Where does UK legislation currently stand on frontier AI?

Speaking to Tech Policy Press, AI partner at law firm CMS John Buyers said that while Narayan was “overstating the position under English law,” the UK does regulate the input data needed to train frontier AI models.

“Narayan is referring in practical terms to the commercial difficulty and expense of obtaining rights over the very large datasets needed to train transformers,” he said. This is because training is only lawful if the developer uses licensed materials, its own materials, or obtains a license to use materials.

But while training on unlicensed material can be copyright infringement, it is unlikely to be a criminal offense. Infringement of copyright law “is generally enforced by content rights holders through civil litigation,” he said. In other words, a developer who trains on unlicensed material risks being sued by the owners of the material it copied, but this is different from being forbidden to train.

Dr. Gregor Schmid, partner at law firm Taylor Wessing, points to the Getty Images v Stability AI case as the closest the UK courts have gotten to testing copyright legislation in relation to AI development. Getty originally alleged several infringements, including that Stability AI copied its images to train Stable Diffusion, a deep learning text-to-image AI model. By trial, Getty accepted there was no evidence that training took place in the UK, so the Training and Development Claim was abandoned. Alleged infringements at the output stage were also dropped during the proceedings due to a lack of evidence.

A further claim of “secondary infringement” of copyright was rejected by the High Court, as the AI model did not store copyrighted works. Getty's claim therefore failed. “To my knowledge, there's not a leading case on the training of AI with copyright-protected material,” he told Tech Policy Press.

Narayan’s comments on the UK’s lack of compute stand in stark contrast to the government’s data center ambitions. In the same way that “superintelligence is not a defined legal concept,” as stated by Dr. Schmid, there is also no consensus on how much compute would be needed to develop what is theorized as superintelligence.

Yet, Narayan told the Politico: “We're on something like 1.4 (gigawatts). It'd be really nice if we could get to a similar single-digit number by the end of this decade so we can build a minimal level of sovereign capability that the British public demands.” Narayan sees developing frontier AI models on British soil to be a national priority, but it’s unclear at what point in development he would consider this posing enough risk to pull the plug. Contrary to the minister’s figures, DSIT estimated the UK already had 1.6 GW of data center capacity in autumn 2024, and is expected to rise to between 3.3 GW and 6.3 GW by 2030.

Aside from “sector-specific laws which may have applicability, including in relation to medical devices and financial services regulation, and very general sweeping laws dealing with emergencies (such as the Civil Contingencies Act 2004)”, there is “nothing framed specifically at frontier models,” said Buyers.

To do this, the UK would need to establish in law who is responsible when increasingly autonomous AI systems cause harm, whether existing or theoretical. “In the UK, which shares many common law similarities with Australia, AI agents appear incapable of being held liable for fraud, for example,” said James Arnott, partner at Pinsent Masons, in response to the Australian cyberattack.

This summer’s cyber hacks are only the latest in a series that have occurred “as a result of humans developing and deploying software, regardless of intent,” Michael Birtwistle, Associate Director at the Ada Lovelace Institute, told Tech Policy Press. Birtwistle described how the developers and deployers of those AI agents – in this case, OpenAI – “are the only actors capable of holding responsibility, because they are the only actors that could have changed the outcome.”

AI’s risks don’t start with superintelligence

In a “world-first,” Labour MP Alex Sobel introduced a private members' bill to Parliament in early September — drafted with lobbying group ControlAI – seeking to prevent the development of artificial superintelligence (ASI).

His bill defines this as “an AI system which can cause serious damage to the security of the United Kingdom because of its capability to neutralize, displace, circumvent, subvert, or render ineffective relevant human authorities in the exercise of their functions.” The bill has been popular among legislators, receiving the backing of more than 70 MPs and peers. It parallels a comparable push in the US, where Senator Bernie Sanders has introduced the Ban Artificial Superintelligence Act.

Sobel’s concerns center on how the development of AI is “moving from a human coding situation to an AI growth situation where the AI effectively generates its own code without human intervention.” Sobel believes that if this trajectory continues, AI systems will become independently capable of reviewing and improving their own development at a rate and speed beyond human control.

The legislation would work by monitoring the state of ASI precursors, both domestically and abroad, against a set of “designated ASI development indicators” which include self-preservation capabilities and the ability to improve independently. Under the new law, it would become a criminal offense to develop ASI, with the Secretary of State granted the power to seize and destroy the compute behind it.

Sobel acknowledges that his proposed legislation has no power to prevent the development of such technology in the US or China, which is why, he said, pushing for a global treaty on international governance is the bill’s overarching goal.

As a Labour Co-op MP first elected in 2017, Sobel also chairs the Joint Committee on Human Rights (JCHR) and has a strong rebuttal to criticisms that focus on nonexistent superintelligence as a distraction from real, tangible AI-created harms today. “I'm absolutely not dismissing those,” he told Tech Policy Press, “but secondly, a lot of those issues, hacking particularly, are precursive things on the road to ASI.”

“We understand why policymakers are concerned about extreme risks stemming from AI, but we think that you do not need superintelligence for many of those harms to realize,” said Birtwistle. Indeed, the cyberattacks of the summer, the security risks this creates, and unpredicted actions taken by AI agents — these were not undertaken by superintelligent systems.

What levers do middle powers, including the UK, have at their disposal?

In recently published research, the Ada Lovelace Institute explored four scenarios for UK AI regulation. Option A is a continuation of business as usual: allowing existing regulators to govern AI in their sectors. The weaknesses in this approach are clearly detailed in a recent inquiry from the Joint Committee on Human Rights (JCHR), chaired by Sobel, which concluded that existing regulators “lack the power to test and evaluate AI systems before they are publicly released, or to prevent their release if they are considered to pose unacceptable risks, except in very limited circumstances.”

The regulatory challenges created by AI technologies were explored in the AI regulation white paper and the following consultation run by the Sunak Conservative government in 2023/2024. To address the gaps this work identified, Conservative peer Lord Holmes introduced the Artificial Intelligence (Regulation) Bill to create an AI Authority tasked with strengthening existing regulators. Birtwistle agrees that such legislation would go some of the way to filling in the gaps, yet even with expanded and strengthened powers, regulators may not have the resources to address the full range and extent of the threats posed by frontier AI.

Burnham has sought to convince global leaders that the UK’s AI Security Institute (AISI) can play a key role in testing the latest frontier models developed by labs across the world. Yet, AISI has already faced major stumbling blocks in doing so. Anthropic delayed Claude Mythos 5.1 from AISI, choosing instead to hand its latest AI model to US organizations for pre-release testing.

Days after US President Donald Trump announced a “morally-binding” AI agreement between leading tech executives, including Google’s Sundar Pichai, Google announced that it had given its latest pre-release frontier model to the US government for testing. The company said it AISI would only receive it afterward, despite much of the development work taking place at London-based Google DeepMind.

In response to questions on the scope of AISI’s work, the Department for Science, Innovation and Technology (DSIT) told Tech Policy Press that it does not provide a running commentary on the AI models the institute has tested. As a research institute, AISI does not have regulatory powers to act on the harms it detects.

A ban on superintelligence would have limits

In scenario B: a ban on superintelligence, the Ada Lovelace report finds that Sobel’s bill would be largely symbolic, given it can only cover what is built in the UK. Implemented in isolation, it would leave the UK vulnerable to AI systems already in use.

Focusing debate around an AI Bill too narrowly focused on the most severe risks, ranging from cybersecurity risks to superintelligence, would mean that “a large range of visible risks, like human influence, mental health effects of chatbots, how liability laws work in the UK won’t be addressed,” said Birtwistle.

Ada Lovelace concludes that a fourth scenario, D, a comprehensive AI bill, is the only scenario that would protect the UK from the full range of AI harms. This would involve an independent body with the remit to define what counts as a ‘safe’ AI system, and give mandatory pre-deployment testing powers to AISI. It would also hold developers and the platforms accountable for the technology they develop and deploy.

At present, however, the UK government appears most persuaded by “Scenario C:” the introduction of narrowly scoped, binding regulation with a focus on preventing risks to national security. “I’m personally much more focused on cybersecurity and biosecurity risks because they are risks to the British public here and now, and risks that are also very material,” Narayan told Politico.

This scenario is favored by many in the AI industry, including OpenAI’s head of European policy, Tom Duff Gordon, who said legislation should be binding, but narrowly focused on the most serious risks related to national security and cybersecurity.

This shift away from industry's long-standing preference for voluntary commitments could be read as a tacit admission that self-regulation has run its course. But a national security-focused AI bill built on capability-based rules would cover only the risks industry is already willing to see regulated, leaving most of the harms that matter to people's daily lives out of its scope.

There are others in Burnham’s government that appear more wary of industry-backed regulation, with Foreign Secretary Ed Miliband telling the Labour Party conference that the "lesson of history is that we cannot leave it to corporations to put in place the guardrails to serve the public interest.”

There is widespread agreement among politicians that the time to take action to regulate AI is now — if not years ago — but it remains to be seen which direction the government will take, and whether it’ll be enough to earn the trust of global leaders for Britain’s big break at the G20 next year.