My guess would be the difference in perception comes from the fact sport cars tend to be smaller and sit lower, which makes pedestrians and motorist feel safer and less intimidated around them. In addition their general rarity means most people still view them as novelties rather than something to actually take a side on. That being said though it is 100% true performance engines are the absolute worst in terms of economy/emissions/noise; most truck engines are really just oversized economy engines and have the efficiency to match.
Also I tend to think that they are often rather expensive and not as robust. So people who drive them do not want to damage. As repairs tend to be expensive too. So in general they avoid accidents, unless they are going to speed off the road...