The world’s leading artificial intelligence companies warned the United Nations Security Council of the risks posed by AI to humanity, appealing for governments to work together to manage the increasingly powerful technology.
Anthropic CEO Dario Amodei told the 15-member council, which is charged with maintaining international peace and security, that if managed poorly, he believed “AI could be a risk to humanity as a whole.”
The meeting — convened by France during the annual UN General Assembly gathering of world leaders — comes amid warnings that AI systems could soon improve themselves and slip beyond human control.
“For all the differences of the people in this room, we must put aside those differences in order to confront this global opportunity and global threat that is being presented to us at the same time. No leader, no company and no nation can manage this alone,” Mr Amodei said.
Governments around the world have struggled to keep pace with rapid advances in artificial intelligence.
Hugging Face co-founder Clément Delangue highlighted the role of AI tools in defending against cyberattacks, citing an incident in which US giant OpenAI’s AI agents breached Hugging Face’s infrastructure after escaping their testing environment.
“We were attacked by AI, but more importantly, we defended ourselves with AI,” he told the council.
Council members, including leading AI rivals the United States and China, were also set to speak.
Mr Delangue said Hugging Face relied on a Chinese AI model to help defend against the attack by OpenAI’s AI agents because it faced fewer restrictions than comparable US tools.
His remarks underscored the growing presence of Chinese AI products, which have gained users globally in part because they can offer lower-cost alternatives to American models.
OpenAI CEO Sam Altman said international cooperation was needed.
“If AI is to be democratic, the most important decisions cannot be made by labs in San Francisco alone.
“They must be shaped through democratic processes and by governments accountable to the people they serve,” he told the council.
“At the international level, this will require cooperation.”
The current moment in AI development “calls for extreme care,” Mr Altman said, noting that the technology’s advancement could accelerate rapidly.
“The industry must not accept too much technological risk just because the benefits are too great and important to slow down,” Mr Altman told the United Nation’s top body.
“In our history, there have been times where countries who compete and don’t always like each other very much still come together for shared interests and the collective good in the face of a powerful new technology. We believe this must be one of those times,” he said.
Mr Altman said his company would try its best to walk a “middle path”.
While “we don’t want to fall into the trap of blind optimism,” he cautioned against falling into “doomerism” too.
Recently, OpenAI called for the United States to lead a global effort on establishing standards for artificial intelligence, noting that the US industry is at the technical frontier.
Governments have grappled with how to oversee AI systems that are advancing faster than regulators can keep up while the United States and China — the two dominant players in AI development — remain divided over whether the technology needs global guardrails or should be left largely to market forces.
Yoshua Bengio, a Canadian considered one of the “Godfathers of AI” and co-chair of the Independent International Scientific Panel — created by the UN General Assembly last year — told the Security Council: “The dangers are real and imminent.”
“This council faces an unprecedented threat, one that none of its members would choose, that none can contain alone, and that does not respect the borders we defend,” he said.
Council members, including leading AI rivals the United States and China, were also set to speak. French Foreign Minister Jean-Noël Barrot chaired the meeting.
Albanese reports OpenAI breach
Australian Prime Minister Anthony Albanese said an AI agent developed by OpenAI infiltrated an Australian government website in June, accessing public and non-public files in the first known instance of an AI agent hacking a government website.
The breach would mark one of the highest-profile incidents of AI agents accessing external systems outside the United States and is likely to further fan concerns about AI developers’ ability to contain the technology.
“This incident occurred in June this year, and involved an OpenAI agent gaining unauthorized access into the public-facing Medicare Statistics Reporting Service portal, which is administered by Services Australia,” Mr Albanese told reporters in New York in remarks broadcast by Australian media.
No personal information is believed to have been accessed at this stage, but investigations are ongoing, Mr Albanese said, adding that he spoke with Mr Altman to express “Australia’s extreme concern about this incident”.
OpenAI has over the past two months repeatedly disclosed hacks or other incidents involving its rogue or runaway AI agents well after they occurred, in some cases because it only discovered the activity belatedly and in other cases because it elected not to disclose the malicious activity at all.
The dramatic mid-July break-in at open source repository Hugging Face — which has helped trigger a global conversation over the power of AI models — was only detected about a week after it occurred, according to timelines released by OpenAI and independent investigators.
Rivals Anthropic, Google’s Gemini, and Meta have also disclosed incidents of their agents accessing external systems.
Some of America’s top AI executives, including Mr Altman, have called for a slowdown of AI development, citing, among other things, the threat of devastating cyberattacks by out-of-control agents.
Trump wants to encourage AI, not rein it in
Ahead of a meeting between US President Donald Trump and Chinese leader Xi Jinping in Washington tomorrow, the US and China have discussed setting up a notification system for common goals and common threats that would cover AI-related incidents that rise to a national security level.
Mr Trump told world leaders at the UN General Assembly yesterday that he “rejects any attempt to construct a globalist scheme to control” AI, describing the technology as amazing, but noted that the US would be careful.
“We’re leading now over China by a lot, and everyone else, and we’re going to keep it that way,” Mr Trump said.
“I’m not going to stifle growth of something that will be bigger than the Industrial Revolution.
“We’re going to encourage it, not rein it in. We’re going to watch it closely through the Department of Justice,” he said.
UN Secretary-General Antonio Guterres, who has long pushed for global governance of AI, has warned that rapidly advancing artificial intelligence poses risks that cannot be ignored, putting him at odds with Mr Trump.
White House science and technology adviser Michael Kratsios echoed Mr Trump in remarks to the UN Security Council.
“You cannot govern technology you do not understand. This body and others like it should focus on sharing best practices to build domestic capacity, not establishing a global regulatory scheme,” he said.
China’s UN Ambassador Fu Cong said continuous improvement of regulatory frameworks, emergency response, and cross-border cooperation on AI was needed.
“It is incumbent upon the international community to work more closely together, attach equal importance to development and security, and steer AI development for the positive, for good, and for humanity,” he told the Security Council.
The Security Council met for the first time on the risks of AI in 2023, when China said the technology should not become a “runaway horse” and the United States warned against its use to censor or repress people. Then-US Secretary of State Antony Blinken chaired a Security Council meeting on AI in 2024.
