Just parking this here for the next time I need to make fun of Less Wrong. Wikipedia summary of the Roko’s basilisk controversy,
In July 2010, LessWrong contributor Roko posted a thought experiment to the site in which an otherwise benevolent future AI system retroactively tortures anyone who did not work to bring the system into existence. This idea came to be known as “Roko’s basilisk,” based on Roko’s idea that merely hearing about the idea would give the hypothetical AI system stronger incentives to employ blackmail. Yudkowsky deleted Roko’s posts on the topic, calling it “stupid” and “dangerous” and claiming that it had caused nightmares for some site users. Discussion of Roko’s basilisk was banned on LessWrong for several years before the ban was lifted in October 2015.
RationalWiki has a much more extensive entry on the topic,
The LessWrong reaction
Silly over-extrapolations of local memes, jargon and concepts have been posted to LessWrong quite a lot; almost all are just downvoted and ignored. But forthis one, Eliezer Yudkowsky, the site’s founder and patriarch, reacted to it hugely. The basilisk was officially banned from discussion on LessWrong for over five years, with occasional allusions to it (and some discussion of media coverage), until the outside knowledge of it became overwhelming.
Thanks to the Streisand effect, discussion of the basilisk and the details of the affair soon spread outside of LessWrong. Indeed, it’s now discussed outside LessWrong frequently, almost anywhere that LessWrong is discussed at all. The entire affair constitutes a worked example of spectacular failure at community management and at controlling purportedly dangerous information.
Some people familiar with the LessWrong memeplex have suffered serious psychological distress after contemplating basilisk-like ideas — even when they’re fairly sure intellectually that it’s a silly problem. The notion is taken sufficiently seriously by some LessWrong posters that they try to work out how to erase evidence of themselves so a future AI can’t reconstruct a copy of them to torture.
Yudkowsky does not consider open discussion of the notion of acausal trade with possible superintelligences to be provably safe, but doesn’t think the basilisk would work:
… a Friendly AI torturing people who didn’t help it exist has probability ~0, nor did I ever say otherwise. If that were a thing I expected to happen given some particular design, which it never was, then I would just build a different AI instead—what kind of monster or idiot do people take me for? Furthermore, the Newcomblike decision theories that are one of my major innovations say that rational agents ignore blackmail threats (and meta-blackmail threats and so on). He also called removing Roko’s post “a huge mistake”.