That's how it's been for at least decades. Democrats voted for Gore from hate of Bush. So this is a structural problem you're stuck with. Unless you come up with a stronger force than a pandemic with 100,000 fatalities.
Ranked choice elections, or something that gives seemingly fringe candidates a chance. Skip primaries for party candidate and use the extra time for more debates.
