I think this should also work with comments as you can easily click the wrong direction by mistake. Anyway, I like the idea of negative voting with a cost. How about:
You need 20 karma to down vote and each vote costs one karma point.
Actually I like the idea - it would prevent (or at least raise the cost of) karma-bombing. I think the threshold should be higher than 20 though - maybe 50 or 100. It would probably be best to introduce it at a high minimum and then lower it after observing activity, like pg said he's doing with the polls.
You need 20 karma to down vote and each vote costs one karma point.