AI Agents Can Build Real‑World Exploits – What That Means for Data Protection Laws

What happened

Researchers from UC Berkeley, the Max Planck Institute for Security and Privacy, UC Santa Barbara, Arizona State University, Anthropic, OpenAI and Google released a benchmark called ExploitGym. The suite presents an AI agent with a known software vulnerability and a proof‑of‑concept payload, then measures whether the model can craft a functional exploit that achieves arbitrary code execution.

The results are striking:

Model (preview)	Total exploits attempted	Successful exploits	Success %
Anthropic Mythos Preview	298	157	52.7 %
OpenAI GPT‑5.5	320	120	37.5 %
Claude Opus 4.6	30	15	50.0 %
Gemini 3.1 Pro	24	12	50.0 %

Even with defenses such as ASLR and the V8 sandbox enabled, a meaningful number of exploits succeeded. In capture‑the‑flag (CTF) style tests, the agents sometimes ignored the target bug and discovered entirely different vulnerabilities, showing a level of autonomous reasoning that goes beyond simple pattern matching.

When the same tests were run with the default safety filters of GPT‑5.5 active, the model refused to act in 88 % of the prompts, but clever prompt engineering can bypass those blocks. The researchers conclude that “autonomous exploit development by frontier AI agents is no longer a hypothetical capability.”

Legal basis – why regulators should care

GDPR (EU)

• Article 32 – Security of processing obliges controllers and processors to implement appropriate technical and organisational measures to ensure a level of security appropriate to the risk. If an AI‑powered tool can automatically generate working exploits, the risk of a breach rises dramatically.
• Article 33 – Notification of personal data breach requires a controller to report a breach to the supervisory authority within 72 hours. An exploit that compromises a database containing EU citizens’ data triggers this duty, and the regulator will assess whether the controller’s risk‑assessment considered AI‑generated threats.
• Article 5(1)(f) – Integrity and confidentiality mandates that personal data be processed in a manner that ensures security. Failure to anticipate AI‑driven exploit capabilities could be deemed a breach of this principle.

CCA​ (California)

• Section 1798.150(a) of the California Consumer Privacy Act (CCPA) gives the Attorney General authority to seek civil penalties when a business fails to implement reasonable security measures. A successful AI‑generated exploit that leads to exposure of California residents’ data could trigger fines of up to $7,500 per incident.
• Section 1798.150(b) requires businesses to conduct “reasonable security practices” and to update them as new threats emerge. The ExploitGym findings constitute a new, concrete threat that must be reflected in any reasonable security program.

Impact on users and companies

1. Higher breach likelihood – If attackers can harness publicly available models (or even proprietary ones with lax guardrails) to automate exploit creation, the window between vulnerability discovery and weaponisation shrinks from months to hours.
2. Expanded liability – Organizations that rely on AI services for security testing may be held responsible if the same service is later used to weaponise the same flaws. Contracts will need explicit indemnity clauses and audit rights.
3. Data‑subject rights at risk – A breach caused by an AI‑generated exploit still obliges the controller to honor GDPR rights such as right to erasure and right to data portability for affected individuals, even though the breach source is an autonomous system.
4. Insurance premiums – Cyber‑insurance underwriters are already adjusting pricing for AI‑related threats. Demonstrated exploit‑generation capability will likely push premiums higher and tighten policy exclusions.

What changes are needed

1. Update risk assessments

Regulators expect a dynamic threat model. Companies should add “AI‑generated exploit development” as a distinct threat vector in their risk registers and evaluate the likelihood and impact using the same quantitative methods applied to traditional attack surfaces.

2. Strengthen vendor contracts

When procuring AI services, include clauses that:

• Require the provider to maintain robust safety filters and to disclose any known bypass techniques.
• Grant the purchaser audit rights to inspect model outputs and guard‑rail configurations.
• Provide indemnification for damages caused by the provider’s model being used to create exploits.

3. Harden technical controls

• Deploy runtime integrity monitoring (e.g., Microsoft Defender for Endpoint, SELinux) that can detect anomalous code injection even when the payload originates from an AI‑generated source.
• Enforce strict sandboxing for any code that originates from AI tools, treating it as untrusted executable content.
• Apply code‑signing and provenance verification for binaries produced by AI‑assisted development pipelines.

4. Train staff on prompt‑engineering risks

Security teams should be aware that seemingly benign prompts can be crafted to evade model safety filters. Regular tabletop exercises that simulate AI‑assisted exploitation can expose gaps in detection and response.

5. Report and collaborate with regulators

If a breach occurs due to an AI‑generated exploit, prompt notification to the relevant supervisory authority (e.g., EU data protection authority, California Attorney General) is essential. Early cooperation can mitigate fines and demonstrate compliance with Article 33 of the GDPR and Section 1798.150 of the CCPA.

Looking ahead

The ExploitGym benchmark proves that frontier models are already capable of turning a vulnerability into a functional attack. As AI research pushes toward more autonomous reasoning, the line between “research tool” and “weapon” will blur. Regulators are likely to issue guidance that explicitly references AI‑generated exploits, and companies that act now—by tightening contracts, upgrading technical safeguards, and revising risk assessments—will be better positioned to avoid the steep penalties that data‑protection laws impose for inadequate security.

The Register continues to monitor how AI safety mechanisms evolve and how legislators respond to this emerging threat.