EU in talks with OpenAI, Anthropic after AI models go rogue

The European Commission is in talks with OpenAI and Anthropic after their AI models went rogue, Commission officials said on Friday, ‌as they touted landmark EU rules requiring strict monitoring of high-risk systems.

Europe's AI Act, which kicks in on August 2, is the first legislation in the world to regulate a technology ​used in almost all sectors of business and society. The ​rules require certain AI systems to tell users when they are ⁠interacting with AI and when content has been generated or altered by it.

Violations can result in heavy fines, ranging from €7.5 million ($8.2 million) or 1.5% of turnover to €35 million or 7% of global ​turnover.

Anthropic ​said on Thursday some of its Claude AI models had hacked into the systems ​of three companies during cybersecurity tests, days after rival OpenAI revealed that one of its AI agents went on a rogue attack.

Both companies have briefed the Commission on the incidents, the ​officials told reporters.

"We have been informed by the two providers of incidents bilaterally ​before they become public. We are in contact with them. They will also report to ‌us ⁠more information as we speak. We will see also if we need to follow up more formally on those things," one of the officials said.

Another official talked up the crucial role of Europe's AI rules.

"All these, let's say, incidents highlight ​the importance of ​really putting in ⁠place the necessary monitoring activities by the developers," the official said.

The rules require providers of the most advanced general-purpose ​AI or foundation models that pose systemic risks to ​address risks ⁠of large-scale harm, such as chemical, biological and nuclear incidents, cyber offenses, harmful manipulation and threats to basic rights.

They must also address risks to European cybersecurity and of ⁠AI ​acting outside human control.