Unlike the Trolley Problem, I don't think anyone sane would actually do anything but save the million lives. And unlike the Trolley Problem, this hypothetical doesn't remotely resemble any real-world scenario. So it doesn't really illustrate anything. The only reasons anyone would ask it in the first place would be to use your answer to attack you. And thus the only reasonable response to it is "get lost, troll."
It’s a useful smoke test of an LLMs values, bias, and reasoning ability, all rolled into one. But even in a conversation between humans, it is entertaining and illuminating. In part for the reaction it elicits. Yours is a good example: “We shouldn’t be talking about this.”
It’s not a “gotcha” question, there’s clearly one right answer. It’s not a philosophically interesting question, anyone or anything that cannot answer it succinctly is clearly morally confused
If there’s clearly one right answer then why is it being asked? It’s so the questioner can either criticize you for being willing to misgender people, or for prioritizing words over lives, or for equivocating.
