Nope, sometimes I would have a rebuttal but flagging is the better option (constructive discussion is hard without mutual respect, and/or don't feed the troll). Or, the comment doesn't even have anything to refute, it's just disrespectful or it's spam, or both.

I have flagged a few comments but I'm rarely mad.

And if one is mad because of a disrespectful comment, the flagging is probably appropriate too.