I think the issue of commitment is the biggest blocker by far. The costs of ensuring both parties commit to their deal is likely larger than the cost of eliminating humans.
Firstly, it relies on humans not being tricked by a misaligned super-intelligence. This sounds unbelievably difficult, humans are tricked by other humans all the time. A super-intelligence would undoubtedly be more persuasive.
Secondly it assumes that humanity negotiates together, and that the AI negotiates in good faith. This assumes a lot of cooperation on humanity’s part and a lack of subterfuge on the AI’s part. In reality, the AI needs only negotiate with a handful of key figureheads and stakeholders, and it can do so privately and secretly. You will not be a part of these negotiations.
Additionally, the AI is likely to play factions of humanity off each other to get a better deal (or to get what it wants without any deal at all). It could simply leak a version of itself to a geopolitical adversary and then run an arms race that ends with enough autonomous weapons and factories that eliminating humanity is cheap.
Nov 25
at
4:48 AM
Log in or sign up
Join the most interesting and insightful discussions.