In a 1-to-1 situation you are right, a single human is not the scariest. But humans (mostly) dominated the animal kingdom, and with enough time we could probably eradicate every other species if we wanted to.

I guess bacteria and viruses might have a stronger claim for being the 'scariest monster' though. I don't think there's a strong 'living' contender for wiping out humanity apart from those. (Viruses is a bit debatable on the 'living' part though).