If Anyone Builds It, Everyone Dies

The book lays out, essentially, a couple of thought experiments that lead to the conclusion that superintelligence (especially of the RSI sort) would likely lead to the extinction of the human race. I think the argument is compelling, and most counterarguments I have seen largely ignore the actual line of argumentation from Yudkowsky and Soares.

My biggest criticism of this book applies to all thought experiments: so what? You can demonstrate an awful lot of things via thought experiment, including conclusions that are so counterintuitive that no-one really takes them seriously. I think that’s the case here, to some extent, based on the reactions I’ve seen to this book.

But, given the stakes: do we want to risk it? At best, do we wish to turn ourselves into pets of a benevolent superintelligence that’s more powerful than us? Personally, I’m going to politely decline.