How many of those non party voters even care enough to vote in a primary. My guess is very few and if they did then they should switch their non to a party.

My experience with primaries is only the most dedicated vote in them. For example in Nevada you have to sit in rooms in schools and listen to every nut that wants to give a ten minute speech to change your vote.