AI safety research
We are searching for a path to safe AI, if one exists.
We use decision and game theory, mechanistic interpretability, and philosophical analysis to understand advanced AI systems as agents, and to search for principled routes to interpretability, alignment, and safety.
Aydin Mohseni · Carnegie Mellon University
Daniel Herrmann · University of North Carolina at Chapel Hill
Research
Agency
How should we model artificial systems as agents with beliefs, preferences, and decision procedures?
Interpretability
What evidence warrants attributing beliefs, goals, or representations to advanced AI systems?
Value
How should a system choose when it is unsure what it values, or what the people it acts for value?
Alignment
How can humans safely delegate to increasingly capable systems whose objectives may diverge from their own?
Programs
- Evolutionary Theory of Advanced AI AdaptationsThe pressures that shape the behavioral dispositions of advanced AI systems, starting with power-seeking.
- Naturalizing AgencyRecovering causation and control from an agent’s own beliefs and desires.
- Measuring Beliefs and Desires in AI SystemsCriteria for attributing beliefs and desires to AI systems, tested on trained models from small transformers to language models.
- Decision under Value UncertaintyLearning what one values, and acting for someone whose preferences one does not know.
- AI Alignment and DelegationA calculus of delegation: how much misalignment we can tolerate, given our beliefs about the system’s goals and capabilities. Everything bottoms out in the behavior of the system and the consequences for us.
Selected outputs
- Radical AI InterpretabilityBook, forthcoming with Cambridge University Press. A framework for attributing beliefs, desires, and meanings to AI systems.
- A Decision-Theoretic Approach for Managing MisalignmentIASEAI 2026. When it is rational to delegate decisions to an AI system despite misalignment.
- Learning the Value of Value LearningWorking paper. A value-of-information theorem for learning what one values.