Our Work

We analyze what AI evaluations can and cannot do for preventing catastrophic risks. While they can help determine lower bounds on AI capabilities, evaluations face fundamental limitations and cannot be solely relied upon to ensure safety.

In this research report we provide an in-depth overview of the mechanisms that could be used to verify adherence to international agreements about AI development.

We respond to the BIS Request for Comment on the Proposed Rule for Establishment of Reporting Requirements for the Development of Advanced Artificial Intelligence Models and Computing Clusters.

In this short paper we identify key assumptions in using evaluations to prevent harm from AI systems, and propose that regulation should require developers to adequately justify these assumptions.

We provide comments on the document Managing Misuse Risk for Dual-Use Foundation Models from US AISI. This guidance should emphasize the uncertainty in AI capability evaluations, and use less optimistic language.

We provide comments on the NIST AI Risk Management Framework profile for generative AI. We suggest including risks from misaligned AI systems, and offer several concrete actions which could be added to the profile.

We offer comments on the Framework for Mitigating Extreme AI Risks, from Senators Romney, Reed, Moran, and King. The framework is an excellent first step, and could be strengthened by including risks from autonomous AI systems.

We respond to the OMB Request for Information on Responsible Procurement of Artificial Intelligence in Government. We urge the OMB to be forward-looking, so today’s policies can address the challenges of future AI systems.

We respond to the NTIA Request for Comment on Dual Use Foundation Artificial Intelligence Models With Widely Available Model Weights. We highlight the need to set a good precedent for future AI systems.