If your end users want a fast site (I would imagine most do), then the faster sites are going to win anyway, with everything else being equal.
At worst, Google is just reflecting that reality (in ~1% of searches where it changes the results).
Personally, I would prefer to compete in a world where everything relevant is "up for competition". You can compete on content, speed, price, or a myriad of other factors. If your complaint is that you want to compete on all of those things EXCEPT speed because that metric is unfair in your opinion, I don't have sympathy for that point of view.
At worst, Google is just reflecting that reality (in ~1% of searches where it changes the results).
Personally, I would prefer to compete in a world where everything relevant is "up for competition". You can compete on content, speed, price, or a myriad of other factors. If your complaint is that you want to compete on all of those things EXCEPT speed because that metric is unfair in your opinion, I don't have sympathy for that point of view.