Research areas and programs

Whether and when we can profitably describe AI systems using the vocabulary and theoretical machinery of agency informs many approaches to AI safety.

Key questions

1. Under what assumptions about a system’s goals and its environment does power-seeking follow from pursuing almost any goal, and which of those assumptions, dropped, make it fail?
2. If we cut the channels two copies of a model use to communicate, can they still coordinate? Causal and evidential decision theories predict different answers.
3. What would show that a system treats an outcome as within its control, and can we read that off its beliefs and desires?

Understanding the internal structure of AI systems, and how that structure produces their outputs, can help us both make deployment decisions and guide monitoring and intervention efforts. We aim to provide practice-informed and conceptually rigorous foundations of interpretability.

Key questions

1. A model fails an eval for a dangerous capability. What licenses the conclusion that it lacks the capability, especially if it can tell it is being tested?
2. What would make a probe for deception a measurement of deception, and not of something that merely goes with deception in the probe’s training data?
3. Probes are trained on our sentences. How do we find the propositions a model’s own states are about, when they need not line up with ours?

Decisions about how to train and deploy AI systems both explicitly and implicitly encode certain values into the systems. We want to make this process more explicit and provide guidance on how to make difficult value choices under conditions of uncertainty.

Key questions

1. RLHF and model constitutions already settle whose feedback counts and how conflicts are resolved. Which aggregation rules do they assume, and how should those rules be compared?
2. How should AIs make decisions on our behalf, if they are uncertain of our preferences?
3. How should an AI system aggregate multiple dimensions of value, when such values are difficult to compare?

Perfect alignment is unlikely, so the practical question is when imperfect alignment is good enough. We aim to develop a calculus of misalignment and capability trade-offs that can guide deployment decisions.

Key questions

1. A proxy is a measurement, valid under the conditions in which it was calibrated. How far outside those conditions can a system optimize it before the principal is worse off?
2. Can a system share our values while reasoning by a different decision theory?
3. How reliable must the measurement of a trait be for a given deployment decision?