Whatever else you say about the AI doomers, they have more than two decades’ history of explicitly spelling out exactly what they believe and why, even when it was complete science fiction to talk about AI at all. They’ve earned the right to be treated as sincere.

it is not even obvious that AIs aren’t conscious! People often dismiss this point by diagnosing it (to my mind, the absolute worst way to argue against anything), or by pointing at some philosophical theory that suggests it might be impossible in principle to construct artificial sentient minds.

I think we are asking the wrong question. Maybe the question should be, are humans obviously conscious?