AI Safety Policy Fellowship

Many leading AI scientists and entrepreneurs have called for mitigating “the risk of extinction from AI” to be “a global priority alongside other societal-scale risks such as pandemics and nuclear war.” This week, we delve into what exactly is the alignment problem.

This week, we examine what is used to predict the timeline of AI development and how reliable they are. We discuss the use of biological anchors, compute trends and ML algorithm trends as constraints to AI development.

We discuss the potential consequences of an intelligence explosion, including the compute-centric framework of AI takeoff speeds, what happens to the economy when intelligence becomes too cheap to meter, and how offense-defense balance scales as investments into a conflict increase.

We discuss the distinct regulatory challenges posed by “frontier AI models”: dangerous capabilities can arise unexpectedly; it is difficult to robustly prevent a deployed model from being misused; and, it is difficult to stop a model's capabilities from proliferating broadly. We discuss evaluation methods for frontier models’ capabilities and risks, and the problem of the trustworthiness of model evaluations.

Even if some governments enact excellent AI regulations, AI developers in countries that do not set adequate guardrails could still cause global damage. This week and the next, we’ll consider: how can policymakers guard against that? Two (complementary) approaches are: (i) slowing the proliferation of advanced AI capabilities beyond countries with adequate guardrails (after some countries establish such guardrails); and (ii) reaching agreements in which multiple governments put in place adequate guardrails. These measures could each give regulated AI labs more (and hopefully enough) time to develop critical safety methods and defensive technologies. That could reduce risks from AI deployments within any country. Aside from these approaches, domestic regulations in any country could have international influence by creating economic incentives and setting examples.

This week, we’ll study the first of the above approaches: non-proliferation. In particular, we’ll focus on some policy levers that may help achieve this. Note there are also potential limitations and downsides to the non-proliferation approach: it may be difficult to predict which states will set adequate guardrails on AI; accelerating some AI developers could leave less time for developing safety methods; and adversarial approaches to AI policy could motivate counter-investments and (if a lead is maintained through aggressive government actions) potentially lead to international conflict. For these reasons, some see non-proliferation as a backup option, for cases where cooperation fails.

This week, we will study a more cooperation-oriented approach to guardrailing AI: states could establish international agreements on AI safety regulations. This can take on many forms in terms of the number of states involved, sources of incentivization, formality, and method of eliciting compliance. This week’s readings emphasize broader context on international diplomacy, as well as how enforcement/verification of compliance with AI safety agreements could be reliable, narrowly targeted, and privacy-preserving.