AI Safety Technical Fellowship

Get up to speed with the foundational concepts in ML. This week is a prerequisite to be done before the course starts and can be skipped if you have relevant experience.

This week focuses on the concept of reward misspecification, which occurs when RL agents are rewarded for misbehaving. We start by looking at some toy examples, both when rewards are hard-coded, and when they are based on human feedback. We then move to focus on how today’s most capable models (known as foundation models) are currently affected by reward misspecification, and how similar techniques might cause serious failures as AIs get more powerful. We finish by exploring a quantitative model for when AGI might be built.

This week discusses the notion of goal-directed behavior in AI systems. Such behaviors are common; for instance, agents trained via RL to solve mazes exhibit the goal-directed behavior of “move towards the maze exit.”

We first look at some examples of goal misgeneralization, where AI systems in training develop goals different from the ones their developers intended. Sophisticated AI systems with goals, regardless of what those goals are, could pursue certain instrumental subgoals like power-seeking. Moreover, under certain assumptions, sophisticated AI systems with unaligned goals may instrumentally behave well during training in order to prevent their goals from being modified. This failure mode is called deceptive alignment.

This week, we discuss how the development of AI capability might bootstrap itself to acceleration. We then look at scalable oversight as a method that enables humans to oversee AI systems that are solving tasks too complicated for a single human to evaluate. We discuss two approaches to scalable oversight: iterated amplification and adversarial techniques.

When we train a ML model, it might learn internal representations of the task that are related to the truth, but not the actual truth. For example, if we train it to generate true texts that humans rate highly, it may output errors that seem true to human evaluators. This is one important cause for misaligned behaviours. We discuss the approach to address this called eliciting latent knowledge. Instead of trying to explicitly, externally specify truth, we try to search for implicit, internal “beliefs” or “knowledge” learned by a model.

Our current methods of training capable neural networks give us little insight into how or why they function. This week we cover the field of mechanistic interpretability, which aims to change this by developing methods for understanding how neural networks think on the level of individual neurons and circuits.

What if it’s very difficult to make progress on alignment through modern ML techniques, and we instead need to take a step back and ask more foundational questions: how does an agent model a world bigger than itself? How does abstraction work? We dip into a few of these more conceptual approaches and zoom out again in our discussion of AI safety.

An important crux for working on AI safety is timelines. When do we expect to see transformative AI? What about AGI? What kinds of signs should we expect to see along the way? We’ll be discussing bioanchors, one of the leading frameworks for thinking about AI timelines forecasting, look at the current predictions of forecasting platforms like Metaculus.