Yes, they use an off-the-shelf diagnostic tool. No, that does not make the methodology sound. They use a word like "aggression" which has a qualitative meaning and attempt to assign a quantitative metric to it. To assign a meaning to it, I would have to go to other studies to find out how this links to my various qualitative notion of "aggressive": incidence of violent crime vs. cutting in line vs. correcting people's grammar.
You may say "this is impossible, the bar you are setting means that social science papers can't even use the word 'aggressive' to describe quantitative results". And yes, that's true, they can't. They can say things like "convicted of a violent crime" which we can at least agree means something objective that vaguely correlates to aggressiveness (although this has limits). The media can report this as "aggression" and that's fine too, so long as we know what is actually being measured.
If they are using an existing & proven diagnostic tool to quantify aggression then you can't criticize them for quantifying a qualitative variable (which is perfectly acceptable when done right) unless you are appropriately addressing flaws within that diagnostic tool.
You're correct, use of such a tool doesn't automatically validate their methodology, but it does shift a lot of the focus of any criticism in the direction of that tool and its implementation, which is not a discussion I've seen in the comment thread. Criticizing or affirming the results of the study are pre-mature without that.
Yes, we're not going to get to the bottom of it here. Since most of us aren't social scientists and don't have experience with any particular diagnostic tool, all we are doing is comparing priors, and there's no reason why you should take the priors of some other stranger commenting on the Internet all that seriously.
But I'll say that my priors lean heavily towards the idea that most surveys, even the standard ones, are less meaningful than they appear, for reasons explained here:
Yes, perfectly acceptable. But garbage for the reasons I described above. You can probably tell that I don’t have a high opinion of social science studies, so it’s unlikely that I’ll be convinced that this is one of the vanishingly small number of studies that I’ll concede has any validity at all.
It’s garbage because it tries to assign a quantity to an English word that already means something. They are piggybacking the formal (which is limited in scope and meaning) on the informal (which is deep and reflective of human experience). If Jane Goodall makes an observation about aggressiveness in social situations it’s a judgement that I can easily accept. There’s great value in trying to obtain a detailed qualitative understanding of behavior.
On the other hand it’s one of the great mistakes of the social sciences to assume that quantification is necessary for engaging in science; this results in “metricism” or “scientist”, which cargo cults the trappings of the physical sciences without any of their insightful power.
Of course the conclusion of the article is correct, but I would not consider this evidence for that conclusion. If they had reached the opposite conclusion I would also have dismissed it entirely, so it’s only fair I do the same even though it lines up with my personal beliefs and experiences.
Yes, they use an off-the-shelf diagnostic tool. No, that does not make the methodology sound. They use a word like "aggression" which has a qualitative meaning and attempt to assign a quantitative metric to it. To assign a meaning to it, I would have to go to other studies to find out how this links to my various qualitative notion of "aggressive": incidence of violent crime vs. cutting in line vs. correcting people's grammar.
You may say "this is impossible, the bar you are setting means that social science papers can't even use the word 'aggressive' to describe quantitative results". And yes, that's true, they can't. They can say things like "convicted of a violent crime" which we can at least agree means something objective that vaguely correlates to aggressiveness (although this has limits). The media can report this as "aggression" and that's fine too, so long as we know what is actually being measured.