Artificial intelligence (AI) could pose a risk to global peace and security, United Nations (UN) Secretary-General Antonio Guterres warned on Tuesday, calling on all member states to urgently set up guardrails to keep the technology in check.

"It is clear that AI will have an impact on every area of our lives," Guterres said at the first UN Security Council meeting on the topic. 

"Generative AI has enormous potential for good and evil at scale," he added, noting that while it could help end poverty or cure cancer, the technology could also have "very serious consequences for global peace and security." 

British Foreign Secretary James Cleverly, whose country currently holds the rotating Security Council presidency, said AI "will affect the work of this council. It could enhance or disrupt global strategic stability." 

"It challenges our fundamental assumptions about defense and deterrence. It poses moral questions about accountability for lethal decisions on the battlefield," said Cleverly, whose government will host an AI summit later this year. 

Guterres asked member states to put in place a legally binding pact to "prohibit lethal autonomous weapons systems" by the end of 2026. 

While AI can be used to identify patterns of violence or to monitor ceasefires, its use by terrorists or governments with ill intent could cause "horrific levels of death and destruction," Guterres said. 

He also warned that malfunctioning AI could wreak havoc, especially if the technology is used in connection with nuclear weapons systems or biotechnology. 

"I urge agreement on the general principle that human agency and control are essential for nuclear weapons and should never be withdrawn," he said. 

The UN chief also called for a working group to develop options for global AI governance by year's end. 

He reiterated his support for a UN entity to support efforts to govern the use of AI, modeled after the International Atomic Energy Agency or the International Civil Aviation Organization.

Zhang Jun, China's permanent representative to the UN, said that AI is a double-edged sword. He said that China supports UN in playing a central coordinating role in the formulation of AI guiding principles.

Zhang pointed out that the good or evil of AI depends on how humans use and manage it. He said that the focus should always be on humans and AI, so that the development of AI can be regulated. Humans should prevent AI from becoming a runaway horse, he added.

Deputy U.S. Ambassador to the UN Jeffrey DeLaurentis said there was a need for countries to also work together on AI and other emerging technologies to address human rights risks that threaten to undermine peace and security.

"No member states should use AI to censor, constrain, repress or disempower people," he told the council.

(With input from AFP)