I think you're doing the right thing by retaining the down arrow on comments, however. When someone writes a comment you dislike, there are four things you can do with it:
1. Ignore it
2. Downmod it
3. Write a quick reply
4. Write a thoughtful reply
Option 4 is ideal, of course, if you have the time, but you usually don't. Quick replies are what we want to avoid. That's how flamewars start. Ignoring the post is going to be unsatisfactory for the flamewar-prone. That leaves the down-arrow as a sort of relief valve. So I think its presence can actually reduce nastiness.
1. Ignore it
2. Downmod it
3. Write a quick reply
4. Write a thoughtful reply
Option 4 is ideal, of course, if you have the time, but you usually don't. Quick replies are what we want to avoid. That's how flamewars start. Ignoring the post is going to be unsatisfactory for the flamewar-prone. That leaves the down-arrow as a sort of relief valve. So I think its presence can actually reduce nastiness.