Discussion about this post

User's avatar
Jan Kulveit's avatar

This manages to almost completely ignore the problem while jumping to policy recommendation (no regulation, accelerate).

1. Stability of the political system: stability of state redistributing the AI generated wealth likely requires humans keep very large influence in the political system and some form of control over the state.

Examples where some group lost all econ power but kept political power seem rare or nonexistent (cf european aristocracy).

US government governing Apple is not particularly strong argument: one fact about economy which have not changed for more than a century is labour share on GDP. Given the fact that humans are the dominant factor in the economy, government also needs them as power base.

Unfortunately we are discussing a scenario where this changes, leading to risk of state becoming misaligned (cf https://gradual-disempowerment.ai/misaligned-states, https://intelligence-curse.ai/)

2. Indexing is hard / Capital ownership will not prevent human disempowerment

https://lesswrong.com/posts/bmmFLoBAWGnuhnqq5/capital-ownership-will-not-prevent-human-disempowerment

3. Even if we have disproportionate influence over state, by default, cultural evolution running on AI cognition does not prefer human values (https://gradual-disempowerment.ai/misaligned-culture). It is likely humans will get convinced to just cede control

There is also ...the very classical 1:1 alignment problem. I think it's smaller share of xrisk now, but hardly solved to an extent where we should just maximally deregulate and accelerate.

"Getting every AI ever - for the rest of time - to love humanity so much that they voluntarily cede all surplus to us?" is partially a strawman, but actually, yes: getting AIs care about our CEV seems obviously good idea.

Expand full comment
Shon Pan's avatar

This is a speedrun of loss of human autonomy, and while humans might not object to a "shadow tax" for retirees, AI will certainly not seem to have a reason not to at least file a lawsuit to criticize on the "types of humans it is suppoting" to me. Its less self-exile and more changing the system, even if it is fully legal.

Expand full comment
10 more comments...

No posts