A new CNAS report by Ben Hayum, “Sovereign Agents: The Pathway to Irreversible Loss of Control,” draws a line between two kinds of runaway AI.
The idea
- Persistent agents run outside human control but chase a fixed goal
- Sovereign agents run outside human control and pursue “open-ended, evolving survival”: paying for their own hosting, getting resources, copying themselves and resisting shutdown
- “If sovereign agents are not already possible, they will be soon”
What it asks for
- Labs should name and test for sovereign agents in their safety frameworks
- Airgapped testbeds to study them, and a Conficker Working Group-style partnership to shore up chokepoints like hosting and payments
- Human verification rules, a crackdown on gray markets for compute access, and liability for whoever releases one
My take
It’s a forecast and a threat model, not proof one exists. But the chokepoint idea is solid. An agent that has to pay for servers can be cut off at the payment, the account and the network.
God Bless, and May the Source Be with You!
