Summary
AI verification research has produced preliminary technical tools, but much of this work is either not well-motivated by plausible international agreements on AI safety, or gets only part of the way towards what such agreements would require.
In this piece, I’ll argue that verification researchers should work more on verifying commitments in frontier safety frameworks (e.g. Anthropic’s RSP, OpenAI’s Preparedness Framework). This is because:
Future international agreements will likely draw from frontier safety frameworks, even if such agreements are not literally of the form, follow your framework
Verification methods built for such frameworks have a clearer path to adoption in the short term, and adoption enables iteration and testing that improves the methods
Such frameworks contain many potentially tractable targets worth trying to verify
Drawing from such frameworks, promising targets for further research include:
Verifying whether monitoring covers all R&D and internal use workloads
Verifying whether security mitigations are properly implemented
Verifying whether risk thresholds have been crossed
Verifying whether development has been paused (e.g. because a risk threshold has been crossed)
Verifying whether mitigations are sufficient to resume development after a pause
Verification researchers should build more technical tools for these targets or third-party organizations to provide human-led assurance of them.
That said, verifying some of these targets with high confidence may be difficult in the near term, and the net benefit of including them in an (interim) international agreement remains uncertain. Still, it is valuable to start trying to verify such targets now, especially since an interim agreement (e.g., a 6-month training pause) may not buy enough time for this work.
What has (technical) verification focused on?
Most existing technical work has focused on:
Verifying whether large-scale training runs are occurring.1
Verifying whether a model was trained as claimed,2 e.g. that required safety mitigations were applied, prohibited techniques or data were not used, and the training process was not poisoned.
Verifying where chips are located,3 to confirm that they have not been diverted.
Verifying whether inference happened as claimed,4 e.g. that a workload involves inference rather than training, or that claimed model outputs are real.
Verifying that a given cluster is not running undeclared workloads,5 e.g. to make sure that an actor is not secretly training a model under the guise of running inference.
Although this work is largely still experimental, it will likely be useful for international agreements. (1), (2), and (5) could help verify agreements of the form, No training run above X FLOP, unless it satisfies condition Y. (3) and (5) would help to ensure that actors capable of such training runs fall within the scope of an agreement, and that their major compute uses are subject to verification. (4) and (5) would help to verify agreements of the form, No use of AI for application X or Models must undergo evaluation Y.
At the same time, this work leaves important sources of risk unaddressed:
We might not be able to stop large training runs for long enough for us to figure out how to verify other important things, like whether appropriate safeguards are implemented.
Risk comes not only from large-scale training runs: it can also come from smaller individual training runs (e.g. small RL post-training runs) or even evaluations (e.g. the Hugging Face incident).
Verifying that training or inference happened as claimed is useful, but many important risk mitigations are not solely about the model (e.g. security infrastructure, internal-use monitoring, proper design of training environments).
Verification researchers can and should work to fill these gaps. As I’ll argue in the next section, frontier safety frameworks are a useful north star for identifying such gaps and providing useful claims to verify.
Why frontier safety frameworks?
A frontier safety framework (or just “safety framework”) sets out how a company intends to manage risks from frontier AI. Typically, such a framework describes risk assessment procedures, risk thresholds, risk mitigations, and governance practices. Examples of safety frameworks include Anthropic’s RSP, OpenAI’s Preparedness Framework, Meta’s Advanced AI Scaling Framework, Google DeepMind’s Frontier Safety Framework.
Verification researchers should focus more on verifying the practices and commitments in safety frameworks for three reasons.
First, future international agreements will likely draw from such frameworks (but will hopefully be stronger than them).6 To understand why, let’s consider what practices a workable agreement would likely focus on. These practices should:
Reduce risks, yet be costly enough that verification is needed7
Be plausible for companies to adopt
Be verifiable
Meeting the last condition is the job of verification research, and I will say more about this in the next section. Safety frameworks are a rich source of practices that fulfill the first two:
Frameworks contain many costly practices that would reduce risks if adopted industry-wide, such as SL3-level security, comprehensive internal-use monitoring, and pausing development until sufficient risk mitigations are ready.
These practices could also plausibly be adopted:
US frontier companies already have safety frameworks and make at least some effort to follow them.
There are efforts to get Chinese companies to write safety frameworks.
Regulation also create incentives for Chinese and US companies to write safety frameworks (cf. General-Purpose AI Code of Practice, SB53).
As a result, techniques for verifying what is in frameworks are likely to carry over to future international agreements. There are few other documents that currently meet these criteria as well as safety frameworks do.8
Second, safety frameworks provide a concrete testbed for verification tools. Even when not fully followed, frameworks still signal demand for the practices they contain. Laws that require audits of framework compliance, such as Illinois’s SB315, increase this demand. Verification methods for safety frameworks therefore have a plausible path to adoption, and adoption enables iteration and testing.
Third, safety frameworks contain many commitments worth verifying. I’ll talk more about these in the next section.
Safety frameworks can be vague or too weak, but neither problem is insurmountable. Researchers can operationalize vague commitments into more precise versions (e.g. defining SL5-level mitigations), which could itself shape what goes into international agreements. And some weak commitments can still be useful to work on, since the tools to verify them could apply to stronger commitments. For example, methods for verifying the amount of compute allocated to monitoring work the same whether the required amount is low or high. Finally, domestic law could make future safety frameworks more precise and demanding.
What should verification researchers prioritize?
Verification researchers should focus on practices or commitments that are:
Important for reducing risk
Costly enough that defection is tempting
Feasible to implement
(Potentially) feasible to verify
Here are some examples from safety frameworks that I think meet these criteria. Whenever a commitment is vague, I suggest some ways to operationalize it more precisely.
Verify that monitoring covers all R&D and internal use workloads.
Risk reduction: Models can take unauthorized or harmful actions during training, evaluation, or internal use. Monitoring is important for catching these issues.
Incentive to defect: Monitoring is costly because it uses compute that could otherwise be spent on R&D or customers.
Feasibility of implementing: Monitoring internal use is rapidly becoming common practice amongst frontier companies (e.g. Anthropic’s monitoring commitment, OpenAI’s monitoring practices). OpenAI has expanded monitoring to cover tool-using RL training and evaluation of models.
Feasibility of verifying: Verifiers would need to make sure that (1) there are no undeclared workloads; (2) R&D and internal-use workloads are correctly classified as such; and (3) those workloads are monitored (effectively).
(1) is the subject of much current verification work.
For (2), some existing technical work can distinguish training from inference, but finer-grained classification needs more work. Because categories can be fuzzy, review of project documentation or other records may be required.
For (3), verifiers would need to establish that approved monitors actually process the required inputs, i.e. agent transcripts from relevant workloads. TEEs would help attest that the intended monitor code is running. One difficulty is ensuring that monitors act on the right inputs without letting verifiers see those potentially sensitive inputs. Evaluations and red-teaming could separately assess how effectively monitors detect the relevant dangerous behavior.
Verify that security mitigations are properly implemented, making it difficult to exfiltrate model weights or for agents to undermine internal infrastructure. Examples of security mitigations include securing training and evaluation environments, encrypting stored model weights, and putting in place shutdown procedures (and other AI control measures).
Risk reduction: Safeguards on stolen weights can be fine-tuned away and agents that undermine security infrastructure can lead to loss of control incidents (cf. Hugging face incident). Security mitigations can prevent these issues.
Incentive to defect: Strong security plausibly slows down development and can be annoying for employees to deal with.
Feasibility of implementing: Frontier companies have already committed to (sometimes vague) security levels in their frameworks, and other reports contain more detailed practices that an auditor could check against: GDM’s Frontier Safety Framework and AI Control Roadmap, Guidelight’s Control Standards, and RAND’s Securing AI Model Weights.
Feasibility of verifying:
Security levels need to be further operationalized into concrete practices. Existing work includes: SL5 Standard for AI Security, Highly Secure Inference Data Centers.
Existing security assessments typically combine document review, staff interviews, and tests of security measures. Such assessments need to be adapted for AI-specific threats and practices.
For a given level of access, security measures vary widely in how confidently their implementation can be verified. For example, checking the implementation of multi-factor authentication is easier than checking that security vulnerabilities are always patched within a given timeframe.
Because the practices are newer, verifying implementation of AI control measures seems particularly neglected. Examples include verifying that: AIs have distinct user identities with limited permissions, AI transcripts are tamper-evident, development and testing environments are appropriately isolated from core infrastructure, and mechanisms for blocking agent actions or shutting agents down are operational.
Verify whether a risk threshold has been crossed.
Risk reduction: If thresholds are not measured properly, they can be crossed without any of the promised mitigations activating.
Incentive to defect: Measuring a threshold can be costly if it involves in-depth evaluations. Crossing one also triggers costly mitigations or deployment delays.
Feasibility of implementing: The safety frameworks of Meta, Anthropic, GDM, and OpenAI have risk thresholds, commonly measured through evaluations. Because these thresholds tend to be somewhat vague, further operationalization is required.
Feasibility of verifying: The approach depends on the evidence used to assess the threshold, which could include model evaluation results, uplift experiments, and qualitative observations.
For thresholds defined by evaluations, we need to verify that the evaluations were actually run and that the reported results are accurate. Ongoing work on inference verification is useful here. Secure enclaves can prove that a specific model ran a specific evaluation without exposing weights or evaluation prompts.
More complex thresholds likely require human experts or sufficiently capable AI systems to make judgment calls. Uplift experiments require checking that the experimental protocol was followed. Thresholds incorporating qualitative observations need judgment calls about whether the evidence supports the threshold determination.
Verify whether development has been paused (e.g. because a risk threshold has been crossed without appropriate mitigations), including smaller RL post-training runs, evaluations, or experimentation.
Risk reduction: Pausing lets developers time fix problems without making it worse, such as by investigating the causes of misalignment rather than naively training it away.
Incentive to defect: Competitive pressures and launch timelines push strongly against pausing.
Feasibility of implementing: The safety frameworks of Meta, Anthropic, GDM, and OpenAI specify conditions under which development would be paused. In response to recent incidents, OpenAI and Anthropic also paused some development activities to improve their safety and security mitigations.
Feasibility of verifying:
Much verification work focuses on large-scale uses of compute. It remains unclear how to verify pauses for small-scale training runs, evaluations, or general experimentation in a privacy-preserving and cost-effective way.
We will likely also need to train human auditors or sufficiently capable AI to make judgment calls. Diagnosing a problem and testing fixes may involve activities that look like development, such as re-running training. Telling the difference requires looking at the actual activity, so it’s difficult to define precise criteria beforehand.
Verify whether mitigations are sufficient to resume development. Although the required mitigations are difficult to anticipate, common ones include monitoring internal use and R&D workloads, securing training and evaluation environments, maintaining chain-of-thought monitorability, and ensuring that training data or environments meet some quality bar (e.g. minimise reward hacking).
Risk reduction: A pause does not provide lasting risk reduction if development resumes with the problem unresolved.
Incentive to defect: As with pausing, competitive pressures incentivize companies to claim that mitigations are more effective than they are.
Feasibility of implementing: The safety frameworks of Meta, Anthropic, GDM, and OpenAI specify high-level conditions under which development would be resumed if paused. Further work is needed to turn these conditions into precise targets that can be verified.
Feasibility of verifying:
Verification would involve checking that specified mitigations are implemented and/or that risk criteria for resuming development are met. Researchers could try to develop verification tools in advance for common classes of mitigations (see above). The threshold-assessment methods discussed further above could help determine whether risk criteria are met.
In either case, trusted third parties would likely be needed to investigate unforeseen problems and judge whether proposed fixes are sufficient. For example, they could assess whether a company has addressed the root cause of misalignment or merely trained away its visible signs to pass an evaluation.
Limitations
Anchoring to safety frameworks has limitations. As discussed above, safety frameworks can be vague and may not provide a precise guide for verification researchers. Work anchored only to frameworks will also miss any risk-reducing practice that no company has committed to. And the space of politically feasible commitments could widen dramatically, especially after a major incident. Some verification work should therefore look beyond frameworks.
It may not be feasible in the near term to robustly verify some of the commitments in safety frameworks, whether because the technology will not yet exist or trusted third-party assurance providers will be lacking. If so, it is unclear whether even interim international agreements should include such commitments. On one hand, doing so could establish shared goals and get buy-in to improve relevant verification methods. On the other hand, it could erode trust by enabling gaming, and also divert effort from more verifiable claims. Even if these latter claims are less useful for safety, an interim agreement centered on them could help to build trust and verification capabilities.
If verification research receives a lot more money and attention in the future, it might make sense to focus now on blunt (e.g. no training runs above X FLOP), more tractable commitments that could form part of an interim agreement. We could then use the time bought by the interim agreement to make progress on verifying the more ambitious commitments in safety frameworks. But an interim agreement (e.g. a training pause) may be difficult to sustain long enough for this work. This risk means we should devote at least some significant effort now to verifying framework commitments.
Next steps
More verification researchers should pick a safety framework, select a commitment within it, and figure out how to verify it. This work could involve developing new tools or processes, or organizations that can provide assurance.
Another important line of work is figuring out what specific commitments should actually go into an international agreement, whether interim or longer-term.
Acknowledgements
Thanks to Scott Singer, Halfdan Holm, Mauricio Baker, and GovAI staff for useful feedback on this piece.
A potential international agreement might even have the form: Everybody needs to follow their safety framework, and everybody’s safety framework needs to include at least XYZ.
Even if safety is in some sense rational in the long run (e.g. nobody wants to die in a loss-of-control event, releasing unsafe products costs companies market share), people don’t necessarily behave rationally. They also hold different beliefs about how severe or likely any problem or bad outcome would be. Decisions could also be shaped by competing factions within companies.
Some potential exceptions include: Guidelight’s standards, the GDM AI control roadmap, and RAND’s report on securing model weights. These documents would also be reasonable sources of targets for verification research.