Naturalizing Agency
Claims about AI risk use the vocabulary of agency: a system seeks power, plans around its overseers, controls its environment. We show how notions like causation and control can be recovered from an agent’s own beliefs and desires. This renders claims made in that vocabulary empirically testable and free of metaphysical baggage.
Led by Aydin Mohseni and Daniel Herrmann
Overview
Can the notions we use to describe agents be recovered from the agent’s own beliefs and desires?
De Finetti showed that an agent who treats a sequence of trials as exchangeable behaves as if she were learning an unknown chance, so chance can be recovered from credence. Skyrms extended the program, treating chance as resilient credence.B. de Finetti, “La prévision: ses lois logiques, ses sources subjectives,” Annales de l’Institut Henri Poincaré 7 (1937); B. Skyrms, Causal Necessity, Yale University Press, 1980. We take the same approach to causation, counterfactuals, and an agent’s sense of what is under her control. Each looks like a fact about the world that an agent must track. The aim is to show that each can be recovered from her attitudes once she represents herself as part of the world she is trying to predict and change.
| Notion | Recovered from | Work |
|---|---|---|
| Chance | exchangeable credence | de Finetti; Skyrms |
| Causation | conditioning on naturalized intervention variables | A Bayesian Reduction of Causation |
| Counterfactuals | in progress | current work |
| Control | the propositions an agent’s attitudes treat as hers to decide | Naturalizing Decision Theory |
Current work
Causation and control
Results on Pearl’s causal hierarchy seem to show that causal claims cannot be reduced to probabilistic ones.J. Pearl, Causality, Cambridge University Press, 2000; E. Bareinboim, J. D. Correa, D. Ibeling, and T. Icard, “On Pearl’s Hierarchy and the Foundations of Causal Inference,” 2022. The irreducibility comes from an impoverished set of variables. Add variables for interventions, treated as events in the world, and interventional claims become conditional probabilities:
P(Y = y | do(X = x)) = P(Y = y | X = x, IX = on).
For control, desirability tracking reads off, from an agent’s beliefs and desires, the propositions she takes herself to control and the decision problems she takes herself to face.
Why it matters for safety
Power-seeking is the pursuit of options an agent takes to be within its control. Planning around an overseer needs a world model that supports reasoning about interventions. Both claims presuppose an account of what a system treats as a cause and as within its power. If those can be read off its beliefs and desires, then the worry that a system seeks power or plans around its overseers becomes a claim about its attitudes that our interpretability program can test. The adaptations program can deploy this account of control.
Outputs
- A Bayesian Reduction of CausationDaniel Herrmann, Aydin Mohseni, Benjamin A. Levinstein, and Bruce Rushing. Working paper (in progress).
- Subjective Naturalism & Act ProbabilitiesDaniel Herrmann. Working paper (in progress).
- Deference and DecisionDaniel Herrmann and Gerard Rothfus. Theory and Decision. Paper, 2026.
- Naturalizing Decision TheoryDaniel Herrmann. PhD dissertation, University of California, Irvine. Dissertation, 2023.
More research