Measuring Beliefs and Desires in AI Systems
How can we tell what an AI system believes and wants? We develop criteria for attributing beliefs, desires, and meanings to AI systems, and test them on trained models, from small transformers to language models.
Led by Aydin Mohseni and Daniel Herrmann
Overview
How can we tell what an AI system believes and wants from facts about its computation?
Interpretability methods read beliefs and goals off a model’s internal states. We provide clear criteria for when such methods are successful. We take up the problem of radical interpretation,Radical interpretation: the problem, posed by Davidson and Lewis, of working out what an agent believes, wants, and means from facts that do not presuppose any of these. with the facts about the agent given by its computation:
Find the beliefs B, desires D, and meanings M that together best explain the computation C of system S.
The three are solved for together. Belief, desire, and the structure of the propositions they concern constrain one another, so a method that holds one fixed while measuring another inherits its distortions.
Current work
Criteria. Probes for belief in language models fail to generalize in basic ways,B. A. Levinstein and D. A. Herrmann, “Still No Lie Detector for Language Models: Probing Empirical and Conceptual Roadblocks,” Philosophical Studies. and a representation must meet four conditions to count as belief-like.D. A. Herrmann and B. A. Levinstein, “Standards for Belief Representations in LLMs,” Minds and Machines. The book Radical AI Interpretability ties criteria of success, on representationalist and interpretationist views alike, to tests that current methods can run. The next step is to give deception and situational awareness the same treatment.
| Condition | The representation must | A typical test |
|---|---|---|
| Accuracy | track the truth of what it represents | probe accuracy on held-out statements |
| Coherence | respect logical relations among its contents | consistency under negation and conjunction |
| Uniformity | be the same across domains and contexts | one probe transferring across datasets |
| Use | guide the system’s behavior | intervening on it shifts what the system does |
Model organisms. In a synthetic decision task, the true posterior over hidden states and the value of each option can be computed exactly, so attributions can be checked against ground truth. A small transformer trained on choices alone develops separable linear representations of both, and steering the posterior changes choice through the subspace that encodes the options’ values, as a Jeffrey–Bolker reading predicts. A pretrained language model shows the same organization. Jeremy Saal leads this work, in collaboration with Simplex. A paper is under review.
Language models. When a language model’s choices are read as expected-utility maximization, how coherent are the credences and utilities this attributes, and how robust is the attribution? We test whether it predicts choices in and out of sample, whether steering the putative credence and utility representations shifts choices as expected-utility theory predicts, whether it agrees with probes, and whether it is stable under meaning-preserving rephrasings.
Outputs
- Radical AI InterpretabilityDaniel Herrmann and Benjamin A. Levinstein. Cambridge Elements in Philosophy of AI, Cambridge University Press (forthcoming). Book, 2026.
- Standards for Belief Representations in LLMsDaniel Herrmann and Benjamin A. Levinstein. Minds and Machines. Paper, 2025.
- Still No Lie Detector for Language Models: Probing Empirical and Conceptual RoadblocksBenjamin A. Levinstein and Daniel Herrmann. Philosophical Studies. Paper, 2025.
- Measuring Agency: Belief and Preference Elicitation for Language ModelsAydin Mohseni and Dorian Clay. Working paper (in progress).
More research