I would hope they weren't downvoted equally. Random acts of meanness are much more poisonous to a site/society than random acts of kindness.

True, but there is a difference between downvoting and not-upvoting. Leaving them alone at 1 point would have let them sink to the bottom just as well and not appeared to be mean-spirited.

I see that as noise, and I prefer to downvote noise. Not voting is reserved for comments with very small contributions or opinions I disagree with and don't seem properly justified. But that's just me.

I agree that downvoting is a harsh punishment, but it's also my opinion that this "noise" should be differentiated from the above cases, so I don't see any way around.