You absolutely have the right to not board an aircraft if you don't feel safe, and should exercise it. Laurence Gonzales has a story about declining to fly on a DC-10, which saved his life. But even he admits that that decision wasn't rational: you can't judge a decision by what you know after the fact. Even at its worst, a flight on a DC-10 was much safer than his morning commute.

You'll have less luck getting a refund for that decision, but I do appreciate your admitting that the decision is about your feelings, not demonstrable evidence.