You might be confused about what I'm trying to say. I don't mean to downplay the problem: it is a very serious and important problem. It is just not an unsolvable one, and I am trying to illustrate that certain approaches my yield progress.

There is a lot of confusion about how an AI would have to work. People make assumptions about how they can self-improve, which I think are idealistic. There is little reason to believe a super intelligence can solve every problem, and if we are actually communicating with it, we should be able to tell which problems it can solve and which ones it can't. I mean, that's the whole point of building such a thing! That's _why_ we want to build it! So to say "it can solve problems we can't foresee and use that against us" is like saying we are building machines for some purpose and then never using them for that purpose.

Nobody is going to build a super intelligence and then not monitor it. We build it _too_ monitor it. We want to know _why_ it is making the decisions it does, and it has to answer in a way that is acceptable to us, or else we will just redesign it -- because it isn't doing what it was designed to do.

I mean, imagine you hired super-human genius to work for you. Yes, at some point, you're just trusting her not run your enterprise into the ground. But you wouldn't have hired her unless she had demonstrated a strong interest in helping you, and you wouldn't keep her on if she isn't actively helping.