Maciej Ceglowski: Superintelligence: The Idea That Eats Smart People: "I was delighted some years later to come across an essay by Scott Alexander about what he calls epistemic learned helplessness. Epistemology is one of those big words, but all it means is 'how do you know what you know is true?'. Alexander noticed that when he was a young man, he would be taken in by 'alternative' histories he read by various crackpots. He would read the history and be utterly convinced, then read the rebuttal and be convinced by that, and so on. At some point he noticed was these alternative histories were mutually contradictory, so they could not possibly all be true. And from that he reasoned that he was simply somebody who could not trust his judgement. He was too easily persuaded. People who believe in superintelligence present an interesting case, because many of them are freakishly smart. They can argue you into the ground. But are their arguments right, or is there just something about very smart minds that leaves them vulnerable to religious conversion about AI risk, and makes them particularly persuasive? Is the idea of 'superintelligence' just a memetic hazard?...
#noted
