Log in
The UN met this week to discuss, among other things, the threat of autonomous weapons.
“The technology is developing at a pace that is very concerning,” Laurent Gisel, the head of the International Committee of the Red Cross’s (ICRC) legal warfare team, told 404 Media. Gisel was in NYC this week to address the UN about all the things that are driving conflict across the planet. His prepared remarks include a statement on the illegal arms trade, nuclear weapons, mines, and – of course – AI.
Autonomous weapons have been a hot topic at the UN this year. In September, the UN attempted to pass a treaty that would govern the use of killer AI weapons. Diplomats from Russia and the US stripped it of its power.
But the UN has pressed forward. On Tuesday, the international governing body announced a new initiative that would grapple with the issue. But, as Gisel said: “multilateral processes take quite some time” and “we already see the effects on the battlefield.”
AI is already picking targets and killing people. Gisel said the technology is “amplifying the destruction, amplifying the violation of the law, amplifying the humanitarian harm caused by conflicts.”
Israel used an AI powered by Microsoft to pick targets in Gaza. In Ukraine, AI-powered Russian drones are deciding which humans to kill. The United States military is pitching itself as an “AI-first” fighting force. Secretary of Defense Pete Hegseth recently announced Project Meridian, an initiative to guide the AI-powered autonomous weapons of the future. Three of Meridian's leads are Elon Musk, defense contractor Palmer Luckey, and Newt Gingrich.
For Gisel, one of the biggest dangers of AI is how fast it is. A core selling point for the tech is that it can help commanders make decisions faster. But that also accelerates the pace of mistakes. “Targets are created at higher speed and higher frequency, and are actually then prosecuted quicker, and with that give us great concerns in terms of the level of human control that is retained,” he said. “The human is supposed to remain in the loop.”
The human does not always remain in the loop. The Israel Defense Forces told journalists it did not thoroughly vet the kill lists generated by Lavender, the AI system it used to pick targets in Gaza. “It's known in the [civilian world] [...] that humans tend to trust the systems,” Gisel said. “That it's more the case when the humans are tired, when the humans are under stress, when the humans have to deliver quickly, and that's of course exactly the situation in which soldiers are in. So all of that gives us a lot of concerns in terms of retaining control on the use of force in armed conflict.”
AI boosters often pitch AI as an “easy button,” a way for people to brute force a complicated task and get it over with. But the consequences are much different when someone in Silicon Valley is parsing spreadsheets or reverse engineering code than when they’re prosecuting a war in the Middle East.
“That's something that the developers of those systems don't necessarily realize either, because some of those systems are developed by technological companies very far away from the battlefield, and they are not naturally all developed for battlefield use,” Gisel said.
International humanitarian law is an ever-evolving legal negotiation between world powers meant to limit the devastating effects of war. It’s imperfect and countries with vast armies like the US and Russia often do not abide by it. The ICRC, which Gisel represents, exists explicitly to shape these rules and laws. Some of the laws are old, far older than AI, but according to Gisel humanitarian law applies to “all weapons, means, and methods of warfare. Those of yesterday, those of today, and those of tomorrow.”
Beyond the opportunity to code humanitarian law into AI, Gisel said there should be new rules and regulations for the new technology. “Not to prohibit them all, but to prohibit specific categories of autonomous weapon systems and put specific restrictions on others,” he said.
"...Once the genie is out of the bottle, it's very, very much difficult to put it back in the bottle"
But he does have redlines. The first is any AI support system that removes a human being from the kill chain. “That requires specific limits and guardrails and restrictions from our perspective,” he said.
“The other type of autonomous weapon system that we recommend prohibiting is those that would target human beings directly, where the target profile is the human being, as opposed to a tank, weapon systems, or another typical military objective, and the reason why is based on ethical concerns, humanitarian concerns, and a moral line,” he added.
Autonomous weapons have existed for a long time. Some US warships have gatling guns that track and destroy incoming targets automatically. The difference is that those weapons track ships and missiles, not human beings.
According to Gisel, one problem with autonomous weapons that target humans or AI support systems that pick targets is that they’re unpredictable. He said banning unpredictable weapons should be a non-brainer. “Military discipline is about your forces [...] delivering the commander's intent,” he said. “If you have a weapon that is unpredictable in its effect, how can you deliver the commander's intent?”
But it’s been a sticking point in negotiations. “We would have thought, military considerations, would align. But actually, that has also faced quite some pushback. Strangely enough,” he said. He spoke about the negotiations in August and September from earlier in the year. “There were references in the last proposal by the chair of the group on precisely all those questions: that the autonomous weapon systems need to be reliable, explainable, understandable, and predictable. And all of that was taken out by some states.”
Gisel also opposes AI in nuclear command and control. In 1983, the Soviet Union’s early warning systems reported that the US had launched a nuclear weapon at the country. Soviet lieutenant colonel Stanislav Petrov decided the computer had made a mistake and decided not to report the attacks up the chain. He was right.
“States will continue to use satellite systems and therefore algorithms that go with those satellites,” he said. “But it's so important to ensure that it remains entirely under human control and a really meaningful human control.”
While the UN deliberates and the ICRC wrangles with how to regulate all this, the wars continue and military forces continue to embrace AI. “That's why it's urgent to have decisions being made by the international community on what to prohibit and what to restrict [...] once the genie is out of the bottle, it's very, very much difficult to put it back in the bottle,” he said.
Gisel said the world needs to act now. “Otherwise, it's the technology that will decide what is acceptable, and that's not the world we want to live in, right?”
