Evolutionary Theory of Advanced AI Adaptations

Which pressures shape the behavioral dispositions of AI systems as they are trained and deployed, and how do those pressures bear on alignment?

Claims that capable AI systems will seek power, resist shutdown, or deceive their overseers are usually stated loosely. We treat each such claim of instrumental convergenceInstrumental convergence: the thesis that agents with very different final goals will pursue similar intermediate goals, such as acquiring resources, because those goals help with almost any final goal. as a claim with six parts:

Systems of kind A, trained by processes T, on tasks and data distributed as D, converge in sense C to behavior pattern B in situations S.

The behavior patterns B in view include coherence, coordination, collusion, deception, learning, situational awareness, proxy gaming, power-seeking, resource acquisition, and self-preservation. Current work concerns power-seeking.

The power-seeking theorems

Turner and colleagues proved that, for most reward functions, optimal policies tend to seek power.A. M. Turner, L. Smith, R. Shah, A. Critch, and P. Tadepalli, “Optimal Policies Tend to Seek Power,” NeurIPS 2021; A. M. Turner and P. Tadepalli, “Parametrically Retargetable Decision-Makers Tend to Seek Power,” NeurIPS 2022. The proofs count reward functions by a symmetry argument that many readers find hard to interpret. We construct the prior that the counting assumes, which turns each result into a statement about probability. The probabilities then admit two interpretations.

	Epistemic	Frequency
The probability is	an agent’s credence over its possible goals	the frequency of outcomes in a training or selection process
What converges	the choices of an expected-utility maximizer	behavioral rules, under Bayesian learning, reinforcement learning, imitation, or replicator dynamics
Whose beliefs	the agent’s	no one’s

• Naturalizing AgencyAgency
• Measuring Beliefs and Desires in AI SystemsInterpretability
• Decision under Value UncertaintyValue
• AI Alignment and DelegationAlignment

Back to Agency · All research areas