If it walks like a Duck and talks like a Duck people we treat it like a Duck.
And if the Duck has a will of its own, is smarter than us, and has everyone attention (because you have to pay attention to the Duck that is doing your job for you), it will be a very powerful Duck.
Exactly. Turing postulated this more than half a century ago.
It's weird that people are still surprised of the ethical consequences of the Turing-test, as if it were some checkbox to tick or trophy to win,
instead of it being a profound thought experiment on the non-provability of consciousness and general guidelines for politeness towards things that quack like a human.
And if the Duck has a will of its own, is smarter than us, and has everyone attention (because you have to pay attention to the Duck that is doing your job for you), it will be a very powerful Duck.