AI Is Killing Autonomously. Time for New Rules.

Next week, leaders will gather at the United Nations in Geneva to decide whether to negotiate a treaty on autonomous weapons. They must act decisively: machines of this kind have already killed soldiers in eastern Ukraine, and three civilians in Zaporizhzhia last July. As defense companies rush to bolt AI onto their products, many more could die without new rules governing this technology.

The first incident happened two years ago, when autonomous drones killed humans — probably for the first time. A swarm of ten machines slammed into a field near Bakhmut, ending the lives of at least two combatants. Ukrainian drone maker Alexander Kokhanovskyy said the targets were identified by AI: “We just launch it and we know everything will be dead” in the selected area. This lethal targeting without human intervention announces a new regime of violence that we are utterly unprepared for: although international laws govern human decision-making, they fail to capture the new realities of digital warfare.

Many AI-enabled weapons are black boxes. Given their ability to adapt in real time, machine learning systems are responsive to countless environmental stimuli. To achieve their goals, they can identify and exploit opportunities that their designers did not foresee, exhibiting what scientists describe as “emergent” behavior. Put simply, their choices cannot be fully predicted or explained ahead of time as required by law. Kill decisions are also deeply networked, arising from sensor and platform interaction rather than from a single, isolated judgment. In contrast, international humanitarian law assumes unity of purpose by treating each attack as flowing from a discrete decision; yet new technologies like drone swarms are decentralized by definition. Not surprisingly, the International Committee of the Red Cross has called for these unpredictable systems to be prohibited outright.

Some argue that drone commanders are liable under existing laws for damage caused by their weapons. According to this position, operators deploying the system remain responsible for any fallout, including its precautionary assessments. But using a legal framework that relies on individual intent is difficult when human agency is severed from the drone’s decision matrix. For example, assigning blame to humans becomes highly questionable when equipment is operating behind enemy lines, where drones fly in dark mode and pilot input is completely absent: in these electronically jammed missions, it becomes impossible to verify what the drones saw or targeted — even after strikes happen — and human control is reduced to an illusion.

Even if commanders could be blamed for crimes committed with these weapons, current rules need updating for a second, more fundamental reason: humans have become so entangled with technology in waging war that they have surrendered many cognitive functions to machines. Think of the infrastructure enabling the remote war on terror, where drone pilot contributions to critical command decisions became diluted in cybernetic assemblages. This symbiosis is not slowing down: a 2019 US Army report identified neural, ocular, and auditory enhancement as technically feasible by mid-century, further blurring the lines between operators and the platforms they deploy. In a world where cybernetic systems are generating violence exceeding human intent, the concept of a distinct “human soldier” making a deliberate choice is quickly becoming an obsolete legal fiction.

If policymakers keep clinging to this fiction, they risk ignoring a widening enforcement gap because international law barely registers the new paradigm of degraded human control. Consider the incident in July, when a Russian drone guided by experimental AI killed three Ukrainians. The unit, resembling a small model airplane, was programmed by human operators to destroy propane fuel tanks at a gas station. Forensic analysis showed that the drone was flying with no links to a human operator and used a commercially available Nvidia chip to steer itself. After failing to clear a building, it exploded near sheltering civilians. Who intended those deaths? The drone operators selected a target category, not a victim.

To strengthen accountability, officials meeting in Switzerland next week should start negotiating a binding treaty. The latest report on the table drafted by governmental experts offers a good start, but fails to close the enforcement gap for weapons systems operating in unobserved environments.

The report’s language covers human control before and during deployment. But once a drone is jammed, only inputs made prior to launch remain relevant for liability. This could leave operators off the hook for violations because targets are not “seen” before a drone goes dark: AI models are trained to learn what a target category looks like, but are never shown the actual asset. Such an attack could still violate humanitarian law, which requires targeting verification to minimize collateral damage. But given the absence of human input in final target selection, accountability would be elusive. Similarly, international criminal law requires human intent tied to a specific attack: a commander who launches a system that kills unforeseeably in an unobserved window is unlikely to be prosecuted. And worryingly, the text assigns no responsibility to the tech companies building these systems, who fiercely protect their commercial secrets from public scrutiny.

This new algorithmic reality urgently needs new rules that reach beyond individual human intent: enforceable obligations for states that field unpredictable killing machines, and liability for the companies that build them. If diplomats fail to negotiate a new treaty, ever more lives will hang by a line of code.