Decision under Value Uncertainty
How should an agent choose when values are uncertain, whether her own values or those of someone she acts for? Both questions arise for AI systems meant to learn and serve what people value.
Led by Aydin Mohseni and Daniel Herrmann
Overview
How should values that are not yet settled enter a decision?
Decision theory handles uncertainty about facts well and uncertainty about values badly. Two cases matter for AI: an agent refining her own values, and an agent acting for someone whose preferences she does not know.
Current work
Learning what one values
Standard frameworks let an agent learn about the world while her options and values stay fixed. We extend the Jeffrey–Bolker framework so that values themselves can be refined, and prove a value-of-information theorem for this axiological refinement. Between agents, mutual refinement tends to turn zero-sum games into positive-sum ones, and can yield Pareto improvements in Nash bargaining.
Acting for someone whose preferences are uncertain
A benefactor chooses on behalf of one person and is uncertain about that person’s preferences. The natural rule is to maximize expected utility across the candidate preferences. But each candidate utility function is fixed only up to a choice of scale, and rescaling one of them can change which option comes out best.Multiplying one candidate utility function by ten changes nothing about the preferences it represents, but gives that candidate ten times the weight in the expectation. This is the problem of interpersonal comparisons, arising within a single person. What the benefactor needs is a rule
f: probability functions over the person’s possible utility functions → rankings of the options.
We ask how far the standard solutions to interpersonal comparisons carry over to the intrapersonal case, among them normalizing every utility function to the same range and egalitarian equivalence. We also ask what changes when the benefactor acts for a group whose preferences are uncertain. There the order of aggregation matters: aggregating across people first and aggregating across possible preferences first can give different rankings.
Outputs
- Learning the Value of Value LearningAlex J. London and Aydin Mohseni. Working paper (in progress).
- Intrapersonal Comparisons under Preference UncertaintyPietro Cibinel and Daniel Herrmann. Working paper (in progress).
More research
- Naturalizing AgencyAgency
- Measuring Beliefs and Desires in AI SystemsInterpretability
- AI Alignment and DelegationAlignment