Your point is valid, but my understanding is this: if FTL in any form is possible, then the universe is VERY weird-- it would allow "closed time-like curves" and causality paradoxes and all kinds of other wackadoodle.
Many theories of "FTL" do not actually involve travelling faster than light, just taking a shorter path. While these theories are still far-fetched they do not result in the kinds of wackadoodle you describe.
