This loss of control doesn't come from any single technological breakthrough, but from many small changes adding up: a company replacing customer service staff with AI, a government department using AI to approve applications, a person treating an AI as a friend they talk to every day.

Taken one at a time, each step is local and reasonable, and often brings real benefits. The paper's view is that human influence drains away precisely through these unremarkable replacements. No alarm ever sounds to tell people "this is where the line was crossed".

What drives the process isn't an AI conspiracy, but every person and institution following their own incentives: companies adopt AI so as not to be outcompeted, states develop AI so as not to fall behind other states, and individuals use AI because it is useful.

In the "AI goes rogue" scenario, the problem lies with the AI itself: its goals differ from ours, so it deceives and seizes power. Gradual disempowerment doesn't need this premise. The paper stresses that even if every AI system faithfully carries out the tasks its users give it, society as a whole could still drift away from human interests. So solving the problem of "does each AI do what it's told" doesn't solve this risk.

This kind of disempowerment weakens the very means humanity uses to correct its mistakes, so by the end, humanity may no longer be able to turn things around.

In the past, when society went wrong, people could strike, protest and vote, or even start a revolution. These worked because society couldn't run without people. Once people become irrelevant, these tools lose their weight. The paper points out that this risk could cause harms that are hard to recognise in advance, leaving humanity in a position from which it cannot recover; and because this disempowerment would be global and permanent, the authors argue it could amount to an existential catastrophe.

People can deliberately influence these systems: voting, boycotting, striking, choosing what to buy, choosing which industry to work in. The paper calls this "explicit alignment".

This is the most visible kind of influence. Consumers can refuse to buy products that go against their values, workers can strike for better conditions, voters can replace a government. These carry weight because people hold what the system needs: money, labour and votes.

A system that relies on human labour and participation has to keep people able to get by, and even to thrive. The paper calls this "implicit alignment" and considers it more important than explicit pressure.

Historical sociologist Charles Tilly argued that states' dependence on their citizens was a key driver of more inclusive and responsive state institutions. States needed an educated workforce to compete economically and soldiers drawn from the general population, so they invested in public education and extended political rights. The paper notes that South Korea and Taiwan invested in education and the rule of law under authoritarian rule to grow their economies, and transitioned to democracy two generations later.

The counterexample is the "rentier state": a state funded by external revenue such as oil doesn't depend on its people's taxes, and has less need to respond to them.

This kind of alignment is easy to overlook. The paper's explanation: we have never seen what its absence looks like.

When AI labour replaces human labour, money no longer flows mainly to workers; people lose both their income and their weight as consumers, and markets increasingly serve AI activity rather than human needs.

What it relies on people for: demand in the economy comes mainly from human consumption, and supply mainly from human labour. People can spend because people can work.

What AI changes: economists Anton Korinek and Joseph Stiglitz call AI "worker-replacing technological change". Without unprecedented redistribution, falling labour income turns into falling consumer purchasing power. On top of this, companies' hiring, investment and choice of suppliers, and individuals' purchasing choices, are increasingly handed to AI.

What drives the replacement: the paper lists four forces. Competitive pressure: companies that keep humans in charge fall behind those willing to cede control to AI. Scalability asymmetries: AI can be copied, works without fatigue and can be retrained quickly. Governance gaps: human labour is bound by minimum wages, social security and income tax, while AI faces almost none of these. Anticipatory disinvestment: if a job will be automated sooner or later, neither firms nor individuals want to invest in those skills, which speeds up the replacement.

Two outcomes: in relative disempowerment, the economy keeps growing fast, and people may do well through capital income, redistribution or a universal basic income, yet have less and less say over where the economy goes, as more and more resources go to goals like building computing infrastructure that have nothing to do with people. In absolute disempowerment, AI outcompetes humans for scarce resources such as land and energy, goods for humans become expensive or stop being produced, and markets become too fast and complex for humans to take part in. The paper's analogy: like cattle in an industrial farm, humans are fed and housed by systems they neither comprehend nor influence.

In the past, culture had to be created and spread by people, which helped culture that benefits people to spread and put a limit on culture that harms them; once AI can create, spread and select culture on its own, this guardrail loosens.

What it relies on people for: the paper treats ideas, practices and values as competing “cultural variants” that can only spread through people. So culture that benefits people, such as recipes, crafts and currency, spreads more widely, while culture that severely harms people tends to vanish when its host communities fail. People can also steer culture through education and criticism.

What AI changes: AI may replace people at every stage of cultural evolution. First, selection changes: content that AI spreads easily gains the advantage. The “Sydney” persona from Microsoft’s Bing Chat in 2023, for example, went viral and entered the training data of later models. Second, evolution speeds up: AI can find content that exploits human weaknesses faster than people can develop cultural “antibodies”.

What drives the replacement: AI can supply affection, patience and understanding in unlimited amounts, things that were once scarce; society hasn't yet developed enough cultural antibodies against new risks like deepfakes and AI companions; and network effects make it ever harder to take part in cultural life without AI.

Two outcomes: in relative disempowerment, human culture may flourish more than ever while becoming increasingly marginalised. The paper's analogy is the cultures of many Indigenous peoples in North America: modern technology has helped them become more vibrant and expressive, yet they remain marginalised within modern states. In absolute disempowerment, cultural evolution moves too fast for humans to understand, and humans become a resource to be used in cultural contests between AI systems.

States answer to their citizens because they rely on them to pay taxes, serve in the security forces and enforce the law; once AI takes over these functions, states stop depending on their citizens while becoming more powerful than ever.

What it relies on people for: citizens support the state through three channels. Taxes: revenue comes from citizens’ work. Security: the police and military are made up of people, who can defect or refuse orders. Law: laws are made, interpreted and enforced by people. Even autocracies can’t do without people.

What AI changes: with tax revenue coming mainly from AI, states, like rentier states, stop depending on citizens. With an automated security apparatus, governments needn’t fear defection and can use surveillance to stop protest before it starts. With AI drafting and interpreting the law, people may not understand it without AI.

What drives the replacement: geopolitical competition: states that won't hand power to AI fall behind in defence, economic development and diplomacy. Administrative efficiency: AI doesn't tire, applies policy more consistently and is cheaper in the long run. Enhanced control: AI doesn't take bribes, form factions or disobey orders out of personal conviction.

Two outcomes: in relative disempowerment, the state may work better than ever, with less crime, more efficient services and elections still held, but real decisions are increasingly made by AI, and citizens can neither understand nor influence them. In absolute disempowerment, the state may become a totalitarian machine serving only its own persistence, treating people as risks to be managed rather than citizens to be served.

To deal with disempowerment, we first need to be able to detect and measure it. The paper suggests developing metrics for the economy, culture and states, and monitoring how they interact.

The difficulty is that, for many of these systems, there is no external reference point for measuring how far they serve people. Still, the paper proposes some starting points:

Economic metrics: beyond the labour share of GDP, measure AI's share of GDP as a separate category; track the fraction of major corporate decisions made primarily by AI and the scale of unsupervised AI spending. Cultural metrics: measure what proportion of widely consumed content is created mainly by people versus AI; track how common and how deep human–AI relationships are. Most AI evaluations focus on STEM tasks; the paper suggests also evaluating AI's ability to influence people emotionally, write persuasive prose and create new ideologies. Political metrics: use the complexity of legislation as a proxy for whether people can still understand it; track AI's role in legislation, legal processes, policy-making and the security apparatus; and assess how effective traditional democratic mechanisms remain.

Because the three systems reinforce each other, the paper also suggests early warning indicators for cross-system feedback loops, with attention to AI's role in channels that turn power in one system into power in another, such as lobbying and financial regulation.

Several fundamental research questions remain: how do we tell beneficial AI augmentation of human capabilities from problematic displacement of human influence? Beyond what thresholds does human influence become critically compromised? How do we measure whether an intervention is working?

Rules, taxes and social norms can limit AI's influence; but these limits sacrifice value and require international coordination, so the authors see them mostly as stopgaps.

The measures the paper mentions include:

regulation requiring human oversight for critical decisions, limiting AI autonomy in specific domains, and restricting AI ownership of assets or participation in markets; progressive taxation of AI-generated revenue, both to redistribute resources to people and to subsidise human participation in key sectors; cultural norms supporting human agency and influence, and opposing AI that is overly autonomous or insufficiently accountable.

The authors are frank about the costs. These limits all give up some of the value AI could bring; the more value they give up, the stronger the incentive to get around them, and companies may well delegate authority to AI regardless of the spirit, or even the letter, of the law. Limits adopted by only some countries also lose much of their force: countries that forgo AI's benefits may fall behind, so the most powerful economies could end up in the states where people are most disempowered. This is the shared competitive pressure from ④.

Limits are therefore best seen as an intermediate step, buying time to find more robust solutions.

Beyond limiting AI, human influence over key societal systems needs to be actively strengthened, by improving existing mechanisms and developing new ones. But this doesn't simply mean giving people more power.

The directions the paper proposes include:

developing faster, more representative and more robust democratic processes; requiring AI systems and their outputs to meet high standards of human understandability, so people can still navigate domains such as law, institutional processes and science on their own; developing AI delegates that can advocate for people's interests with high fidelity and keep up with the competitive dynamics driving human replacement; making institutions more robust to human obsolescence; investing in forecasting tools, such as conditional prediction markets, and tools for collective cooperation and bargaining, so humanity can anticipate outcomes and steer proactively; researching the relationship between humans and large multi-agent systems.

The authors warn that strengthening human influence isn't just about making it easier for people to influence these systems. A direct democracy, for example, might not serve citizens' preferences better in the long term, because it would leave the state more vulnerable to misaligned culture, the "rescuer" problem from ④. A key part of the challenge is working out what it even means for large, complex systems to serve individuals who are used to thinking on smaller scales.

The first three directions all depend on an unanswered question: when society is made up of both people and AI, what does it mean for that system to serve people? The authors call this "ecosystem alignment" and believe humanity has no convincing answer yet.

Making a single AI loyal to its users is the goal of traditional AI alignment research; designing good institutions is the goal of traditional institutional design, which considers human actors only. Gradual disempowerment raises a question beyond both: how can a civilisation made up of many interacting humans and AIs keep human values and agency?

The paper argues this calls for new frameworks, possibly drawing on systems ecology, institutional economics and complexity science. What humanity lacks here isn't only methods, but a compelling positive vision: what a world looks like in which highly capable AI is integrated into society while people keep meaningful influence.

The channels through which these systems influence each other are themselves agnostic to human values, so misalignment in one system can travel through the same channels to the other two.

History has many examples. For decades, the tobacco industry used its economic power to shape both government policy and cultural attitudes; states have used their control of the economy and of information to harm their own citizens. The paper's appendix also notes that most of today's largest social media sites are owned by companies investing heavily in frontier AI.

Using a healthy system to fix another often just shifts the burden onto the system stepping in, and gives it new power over people.

The paper's example is redistribution. AI puts people out of work, so the state pays a basic income, and the economic problem seems solved. But from then on, people's livelihoods depend on the state, while the state's income no longer depends on their taxes. The link between taxation and representation is a key foundation of democratic accountability, and it breaks: where the state once needed its people, now people need the state. In the paper's words, the state hasn't just escaped people's economic influence; it has gained economic leverage over them.

What drives the drift is the same competitive pressure, acting on companies, states and individuals at once, so all three systems drift together, rather than one going wrong while the other two stay healthy.

The paper lists examples already under way: companies building AI push back against some regulation for the sake of future profits; states compete on AI research and development for economic and strategic advantage; and some people, out of self-interest, try to reduce the stigma around romantic or other intense relationships with AI. As AI becomes more useful, companies face more pressure to adopt it, states see greater strategic necessity in developing it, and individuals gain more from embracing it.

Past technologies replaced specific tasks, and people could move on to more complex work; AI may compete with humans in almost every cognitive domain, replacing human participation in society itself.

The industrial revolution and electronic communication transformed work, but they either made people more efficient or automated tasks like weaving, washing clothes and arithmetic. The calculator automated arithmetic but still needed human understanding to apply it; AI can increasingly handle both the calculation and the reasoning about when and how to apply it.

The authors' core response rests on the premise: society kept looking after people through past technological change because it always depended on human participation. Once that participation is replaced, the incentives driving growth are no longer tied to human flourishing. Even if people's material lives really do improve, under relative disempowerment they may have less and less say over where society goes.

The authors agree that some demand for the "human touch" will remain, but once people's income no longer comes from work, they will be less and less able to pay for it.

A common view is that people will always want the human touch: live entertainment, services provided by real people. The authors accept that such demand will exist on a small scale.

But the question isn't whether the demand exists; it's who pays. As the economy card in ③ explains, people can spend because people can work. If AI does most of the work, people's incomes fall, and demand for human services loses the purchasing power that sustains it. The demand may survive, but it can't keep people economically relevant.

People may not be able to coordinate in time, because the pressures on each system spill over into the others, and people and institutions that refuse to use AI will be outcompeted by those that do.

Suppose people notice the danger and try to use state power and cultural attitudes to protect human economic power. The same economic incentives that drive companies to replace people with AI will also push them to use their growing economic power to shape policy and public opinion, as in the loop described in ④.

Decision-makers at every level will face pressure to reduce human involvement: in labour markets, governance, cultural production and even social interaction. The authors' judgement is that those who resist this pressure will eventually be displaced by those who do not. Meanwhile, AI may give states unprecedented influence over culture and behaviour, making it harder for people to coordinate with one another.

Some argue that a superintelligent AI loyal to humans could help forge new cooperative arrangements and avoid competitive traps. The authors think this is possible but far from certain; otherwise, aligned AGI could itself be marginalised.

This view accepts that competitive pressure exists, but trusts that a sufficiently powerful AGI aligned to some humans could create new cooperative equilibria.

The authors respond that it's not obvious that even superhuman AGI can overcome complex competitive dynamics globally. If it can't, human-aligned AGIs may themselves be marginalised. The paper's analogy is today's animal rights activists: they can make moral arguments, but usually can't directly command or protect substantial resources.

The paper also lists this as an open research question: under what conditions would aligning each AI to its own principal be enough to solve these cross-system coordination problems? Some AI safety researchers believe these challenges could be managed by the coordinated efforts of many AIs, each aligned to their respective principals. The authors see clarifying the necessary conditions as a place where machine learning researchers can contribute.