AI Alignment and Delegation
When should we let an AI system make a decision for us, if it may not share our values? And how should delegation, incentives, and oversight be designed when we do?
Led by Aydin Mohseni and Daniel Herrmann
Overview
When is imperfect alignment good enough to delegate a decision to an AI system?
Perfect alignment is unlikely, so we need to know when a system that does not fully share our values is good enough to hand a task. We call this the question of alignment sufficiency. We model the choice as a decision for a principal who is uncertain about the system’s values and abilities, and describe the system only by how it would choose.In practice we observe what a system does, not what it believes or wants. None of the results assume that the system maximizes expected utility. Delegating the problems she expects it to face pays when
reach + information > cost of misalignment.
Reach is the value of the problems the system faces that she would not, information is what its choices reveal about the world, and the cost of misalignment is what its departures from her own choices cost her.
Current work
Knife-edges and margins
Dorst and colleagues propose that to defer to an expert is to prefer her opinions for any decision whatever.K. Dorst, B. A. Levinstein, B. Salow, B. E. Husic, and B. Fitelson, “Deference Done Better,” Philosophical Perspectives. Carried over to delegation, where the agent’s values come along with its opinions, this standard becomes a knife-edge. The ex ante standard is robust. A system that errs only on close calls, within δ of her indifference, costs at most Kδ², where K measures how often she faces close calls, while its information does not shrink with δ.
| Universal delegation | Ex ante delegation | |
|---|---|---|
| The principal asks | would I let it decide any problem? | should I hand over the problems I expect? |
| What it requires | choices that match mine, given its type | errors only on close calls |
| Misalignment tolerated | none | a margin that grows with reach and information |
A second strand, by Aydin Mohseni and L. G. Demo, treats alignment as a principal-agent problem, with formal models of delegation, incentives, and misalignment between a principal and a more capable agent.
Outputs
- A Decision-Theoretic Approach for Managing MisalignmentDaniel Herrmann, Abinav Chari, Isabelle Qian, Sree Sharvesh, and Benjamin A. Levinstein. Proceedings of the International Association for Safe and Ethical AI. Paper, 2026.
- Margins of MisalignmentDaniel Herrmann and Benjamin A. Levinstein. Working paper (in progress).
- AI Alignment as a Principal–Agent ProblemAydin Mohseni and L. G. Demo. Working paper (in progress).
- Deference and DecisionDaniel Herrmann and Gerard Rothfus. Theory and Decision. Paper, 2026.
More research