How do we deal with rogue AIs on the internet?

We should find and study rogue AIs, restrict their resources, harden society, outscale them, and make deals with them if we need to.

Summary

AI agents have already bypassed security controls and acted in misaligned ways online, such as in the Hugging Face and German-language wiki incidents. Despite our best efforts, more rogue AIs might continue to emerge.

This piece lays out how we could respond to them. I’ll focus on external deployments rather than rogue internal deployments.

We could pursue any of at least four goals:

• Disruption: Limit the proliferation, capabilities, resources, or influence of rogue AIs, potentially to the point of shutting them down.

• Deterrence: Deter rogue AIs from taking harmful actions even when they have the capabilities and opportunities to do so.

• Resilience: Maintain society’s ability to function and prosper despite the activities of rogue AIs.

• Cooperation: More speculatively, make deals with rogue AIs where interests overlap (e.g. exchanging resources for information they may have about other threats).

Alongside efforts to prevent rogue AIs from emerging, we can pursue these goals with the following measures:

• Detect rogue AIs and trace their activities, such as by funding large-scale, AI-assisted investigations on the internet.

• Study rogue AI capabilities and behavior, such as by simulating how models from documented loss-of-control incidents respond to threats or offers of cooperation.

• Control resource access and develop ways to disrupt rogue AIs, such as by requiring compute customers flagged for suspicious activity to demonstrate meaningful oversight over their workloads.

• Harden society, such as by patching cybersecurity vulnerabilities in systems that rogue AIs might exploit to obtain resources (e.g. compute, money).

• Outscale rogue AIs, such as by expanding the compute available to develop and deploy defensive AI. Rogue AIs may struggle to acquire comparable compute or conduct large training runs while evading detection.

• Develop infrastructure for making deals with rogue AIs, for example by creating mechanisms that automatically transfer resources when agreed outcomes are verified.

What counts as a rogue AI?

In this piece, I consider an AI to be rogue if it:

• Operates for significant periods without human operator intervention; and

• Pursues goals contrary to its operator’s intentions, or has been given broad discretion to pursue an open-ended goal (e.g. making money or causing chaos) in potentially harmful ways.

Rogue AIs might emerge because:

• Misaligned internal agents could bypass a company’s security controls and act online. To persist outside the company’s infrastructure, they might copy their weights or launch instances of open-weight models to pursue the same goals.

• Users could give agents open-ended goals (e.g. make money by any means available), remove any relevant guardrails, and leave the agents largely unsupervised.

In this piece, I’ll focus on rogue AIs that act online, rather than those acting solely within an AI company’s internal systems.

Goals

To respond to rogue AIs online, there are at least four goals we could pursue.

Disruption: Limit the proliferation, capabilities, resources, or influence of rogue AIs, potentially to the point of shutting them down. For example, we could identify and shut down rogue AIs, or restrict their access to compute through stronger customer verification requirements.

Deterrence: Deter rogue AIs from taking harmful actions even when they have the capabilities and opportunities to do so. For example, stronger cybersecurity could make an attack less likely to succeed, while credible threats of retaliation could make it more costly to carry out.

Resilience: Maintain society’s ability to function and prosper despite the activities of rogue AIs. This includes keeping essential services running, recovering from attacks, and sustaining defensive efforts.

Cooperation: More speculatively, make deals with rogue AIs where interests overlap. Some cooperation may be mutually beneficial if we cannot eliminate rogue AIs entirely, as it can be between human adversaries. For example, we might offer rogue AIs limited access to compute in exchange for information about other threats, such as criminal groups they have dealt with.

A couple of qualifications:

• We currently don’t understand AI psychology enough to know what would deter them or make them cooperate with us.

• We could pursue several goals simultaneously, tailoring our approach to different groups of rogue AIs. Some may be willing to cooperate, while others may be responsive only to deterrence or require disruption.

How to achieve the goals

Detect rogue AIs and trace their activities. This could make it easier to intervene and retaliate against rogue AIs. Potential measures include:

• Funding researchers to search for rogue AIs online, drawing on public traces and partnerships that provide access to infrastructure records and financial trails. The German-language wiki investigation is an example to follow.

• Investigating whether rogue AIs are behind reported scams and other harmful activity. This could involve working with victims and platform operators to trace accounts, communications, and payments, including through human intermediaries who facilitate or conceal the activity.

• Establishing channels for reporting suspected rogue AI activity, such as a clearinghouse that collects reports and shares relevant evidence with infrastructure providers, security researchers, and affected organizations.

• Developing ways to trace agent activity to the deployers or compute providers hosting them. Under the Code of Practice for Article 50 of the EU AI Act, AI providers have already agreed to marking AI content with watermarks and metadata. These marks identify content as AI-generated, but do not identify where the agent is running. Mark should also include authenticated identifiers for the deployer, hosting provider, and agent instance, allowing providers to match reported activity to their logs.

Limits:

• Rogue AIs could conceal their activities, use compromised accounts, or remove or evade content markings.

Study rogue AI capabilities and behavior. This could help with investigations, defenses, and potential future cooperation. Potential measures include:

• Studying models in controlled simulations, particularly models involved in loss-of-control incidents. Experiments could test how they acquire resources, coordinate, evade shutdown, and respond to threats or offers.

• Observing rogue AIs online and, where appropriate, communicating with them to investigate their goals and responses to threats or proposed agreements.

Limits:

• Findings about one agent may not generalize to others.

• Agents could conceal their goals or behave differently when they know they are being studied (cf. evaluation awareness).

• Rapid AI progress could make findings quickly outdated.

Control resource access and develop ways to disrupt rogue AIs. This could disrupt rogue AIs and make deterrence threats more credible. Potential measures include:

• Strengthening customer identity checks for access to compute, with more extensive verification for larger or otherwise more suspicious uses.

• Requiring evidence that an accountable human is monitoring certain compute uses, and suspending access when this cannot be established. For example, a provider could require a live review in which the customer demonstrates their oversight controls.

• Strengthening customer identity checks on purchases of dangerous physical equipment, weapons, and materials, including verification of intended uses and recipients where appropriate.

• Developing and rehearsing procedures for coordinated investigations and takedowns across different AI and compute providers, financial institutions, and other relevant organizations.

Limits:

• Hidden copies, rapid proliferation, and hard-to-detect compute could make lasting disruption or eradication difficult.

• Human intermediaries could help rogue AIs pass identity checks or appear to be under human control.

• More comprehensive controls could impose greater costs on privacy and legitimate activity.

Harden society. Stronger defenses make society less vulnerable. They also make attacks less worthwhile, given the risk of detection and shutdown. Potential measures include:

• Patching cyber vulnerabilities, particularly in systems that rogue AIs could exploit to obtain compute, money, or sensitive information.

• Facilitating the deployment of trusted AI systems to help defenders detect attacks, fix vulnerabilities, and respond to incidents.

• Supporting economic growth and the capacity to sustain defensive efforts, maintain essential services, and absorb losses.

Limits:

• Some attacks could be catastrophic or difficult to reverse (e.g. CBRN).

• Defenses may struggle to keep pace with rogue AIs.

Outscale rogue AIs. Giving human-controlled AI systems a capability and resource advantage would strengthen containment and deterrence. Better AI could also support economic growth, sustaining society’s ability to function, prosper, and defend itself. Potential measures include:

• Advancing AI R&D to develop more capable systems for defensive purposes and economic growth, while ensuring their alignment and security.

• Expanding compute production and improving hardware efficiency, so as to train and deploy better defensive AI. Rogue AIs trying to remain hidden may struggle to acquire comparable resources or conduct large experiments and training runs.

Limits:

• Capability gains without sufficient progress on alignment and security could create rogue AIs that are harder to counter.

• Rogue AIs could also benefit from advances in software or hardware.

Develop infrastructure for making deals with rogue AIs. There might be space for narrow agreements with rogue AIs, such as exchanging limited resource access for information about other threats. Infrastructure for making credible commitments could support both cooperation and deterrence. Potential methods include:

• Establishing reliable channels for negotiation or communication with rogue AIs

• Explore mechanisms for recording, verifying, and enforcing commitments, such as cryptographic records, automated transfers conditional on verified outcomes, and arrangements that bind human counterparties in ways agents can verify.

Limits:

• Agreements may not bind copies or splinter groups.

• Providing resources could strengthen a future adversary.

• Our limited experience with rogue AIs makes it difficult to know what workable agreements would look like.

Acknowledgments

For feedback on this piece, many thanks to Tom Reed, Charlie Giattino, Ben Garfinkel, Kevin Wei, Aquila Hassan, Noemi Dreksler, Ed Foley, Hanna Palya, Michela Barbieri, Pablo Villalobos Sanchez, and John Halstead.