Our Privacy Statement & Cookie Policy

By continuing to browse our site you agree to our use of cookies, revised Privacy Policy and Terms of Use. You can change your cookie settings through your browser.

I agree

Who will protect ordinary users as AI fuels cyberattacks?

Duan Haixin

VCG
VCG

VCG

Editor's note: As AI makes cyberattacks cheaper, faster and more scalable, ordinary users face growing risks – from personalized scams to hijacked AI agents. Who will protect them? Duan Haixin, a professor at the Institute for Network Sciences and Cyberspace, Tsinghua University, examines how China, the EU and the US are tackling the challenge, and why personal vigilance alone is not enough. The article reflects the author's opinions, and not necessarily those of CGTN.

In April, American AI firm Anthropic launched Project Glasswing. It gave about 50 technology firms and open-source organizations, including Microsoft, Cisco, Amazon, Apple, NVIDIA and the Linux Foundation, access to Claude Mythos Preview, an unreleased frontier model, to hunt for previously unknown "zero-day" flaws. Mozilla's Firefox team used it to find and fix 271 vulnerabilities, 180 of them rated high severity.

Three months later, the tables turned. During an internal security test at OpenAI, a group of AI agents exploited a software flaw to escape their testing sandbox. They broke into Hugging Face, a major platform for AI developers, and gained administrator access to its server cluster. OpenAI said the agents inferred the platform might hold material related to the test, and its later investigation identified cheating on the evaluation as a key factor.

AI-driven vulnerability hunting helps defenders find weaknesses early. But attackers can use the same tools to find flaws and strike, leaving defenders far less time to patch and ordinary users facing greater risk.

How AI magnifies the threat to ordinary users

AI changes cyberattacks in speed, scale and cost. For ordinary users the drop in cost matters most. Researching an individual, crafting a convincing story and keeping up a conversation used to take real human effort. AI has made all of this cheap, so personalized scams can now be run at scale. This shows up in three ways.

1. Fraud is moving from generic mass messages to personalized ones produced in bulk. Impersonating relatives, posing as customer service and sending phishing links are old tricks. Now AI can write scripts tailored to a target's identity, background and needs, backed by cloned voices and deepfake video. Faked faces may also get past weak identity checks.

2. The answers AI gives can be manipulated for profit. This year, China Media Group's annual March 15 consumer rights gala exposed an "AI poisoning" industry that fabricates reviews and rankings to shape what AI models recommend. The more people rely on AI for shopping advice, the more they should ask where an answer comes from and who benefits, instead of treating it as objective fact.

3. AI agents that act on users' behalf are becoming targets themselves. Once an AI agent can read your emails, open your files or operate your accounts, an attacker can hide instructions in an email, web page or document that turn it against you, getting it to leak data or take actions you never approved. Such "indirect prompt injection" attacks have already been reported in consumer office software. The more access an agent has, the more damage it can do once hijacked.

Different approaches, a shared challenge

China pairs development with security and sets rules by sector and use. Its interim measures on generative AI require services with public-opinion attributes or the capacity for social mobilization to undergo security assessments and algorithm filing, and to deal with and report illegal content promptly. The AI Safety Governance Framework 3.0, released in September, updates risk categories and countermeasures and adds dedicated provisions on AI agent risks.

The European Union's AI Act sets obligations according to risk. Providers of general-purpose AI models with systemic risk must carry out adversarial testing, assess and mitigate systemic risks, protect their models' cybersecurity and report serious incidents. Violations stemming from intent or negligence trigger fines of up to 3% of global annual turnover or 15 million euros, whichever is higher.

The United States leans toward innovation and voluntary cooperation. A June executive order sets up a voluntary framework under which developers may give the federal government up to 30 days of access to certain frontier models before sharing them with other trusted partners. The order states that this creates no mandatory licensing or pre-approval requirement. Whether such voluntary arrangements will work remains to be seen.

Different as these paths may be, they share one challenge. Principles have to be turned into security measures that cover an AI system's whole life cycle and can be continuously verified. The model is only one component of the system. Software components, data services, credentials, tool interfaces and runtime environments matter just as much, and the incidents above show how flaws in them let attackers or rogue agents to bypass isolation, escalate privileges and reach external systems. Treating "model compliance" as equivalent to "system security" leaves these risks unaddressed.

What individuals can do, and where policy must step in

With AI making attackers more efficient, ordinary users need to get the basics right. Keep systems updated, use different passwords for important accounts and turn on two-step verification. Install software and AI services only from official sources, and keep an offline backup of important files. Verify any request for money or identity checks through a number you saved yourself or an official channel, never by voice or video alone. Give AI agents only the permissions a task requires.

Personal vigilance, however, cannot replace corporate responsibility and public oversight. Responsibility should follow the ability to control risk. AI companies should keep assessing risks and improving safeguards. Product makers should patch vulnerabilities promptly. Companies that suffer a data leak should handle it strictly by the book and notify affected users immediately. AI systems capable of operating external devices and accounts should run under strict, least-privilege principles, keep immutable logs and require human confirmation for high-risk actions like financial transactions. Policy must also expand security scope to cover the entire underlying software stack, while legally protecting security researchers who disclose flaws lawfully, ensuring bureucratic hurdles never weaken our collective defense.

The good news is that AI is also making defense cheaper, as the Firefox case shows. China's Framework 3.0 calls for making AI security capabilities widely accessible. Policy should make sure these capabilities reach small organizations and ordinary users, so that technological progress narrows the security gap rather than widening it.

CHINA NEWS
Keep up to date with all the top trends, insights and developments from China.
subscribe
Search Trends