The other big challenge to this work was the repetition. Reading the answers to the same questions over and over was mentally challenging for people. I don't blame them. Most kids of a specific age aren't too creative when fed a question for a state test. For example, ask them who is a public figure they admire and why, and you're likely to get the bulk of the answers focusing on just a few people (athletes, popular music stars, etc.). For the written assessments, 10% of the student materials were scored twice (by separate people) to ensure accuracy of grades and as a way to identify issues with potential scorers.
I'm curious - do you happen to know whether the essay portions ever offered any actual statistical utility above and beyond the multiple choice ones? As in, do they actually measure anything that the multiple choice questions can't?
I spent several years teaching SAT and GRE classes, I always noticed that I could predict people's essay scores pretty accurately based on their multiple choice scores. Percentile ranks always tended to be very close between the sections, which always made me wonder whether there was any point in having the essay at all. I always suspected that since the overall level of competence was so low on the SAT (sadly, the GRE was not all that much better...), anyone that was writing at the level where actual quality of writing matters was already scoring top marks on the essay, so they were essentially unmeasured by the scale.
I realize that when it comes to setting up these tests, "include an essay" is likely a political mandate, because people for some reason think that essay sections are "more fair" to "bad testers" or "fluid thinkers" or something like that (I'm pretty sure ETS was more or less forced to include an essay for this reason), so it probably wasn't even an option not to include them. But I'd be curious to know whether anyone ever looked into whether they actually told you anything you didn't already know. ETS does not provide this data or analysis, otherwise I'd check it myself as it relates to the SAT.
I'm curious - do you happen to know whether the essay portions ever offered any actual statistical utility above and beyond the multiple choice ones? As in, do they actually measure anything that the multiple choice questions can't?
I believe written assessments were relatively new to standardized tests when I was in the field. My understanding was that the purpose was to assess writing, just like we do with, say, math or reading comprehension. Multiple-choice questions can't evaluate a student's ability to respond to an open-ended question, convey ideas, stay on topic, demonstrate grammar and punctuation, etc.
But I didn't get the impression that adding writing components to the test added anything of practical value. Sure, it's one more score-based judgment that can be assigned to each student and be used to assess schools and districts, but as far as I could tell it wasn't benefitting student education. And it certainly came with a high price tag. But I was definitely not in classrooms, so that's a key caveat to anything I share.
Does writing seem any better? Well, these written assessments have been widespread for about 20 years or so. I don't get the sense from what I see that people nowadays are generally good writers. But my exposure is certainly limited, even if others I speak to seem to agree, and I'm not certain if enough time has elapsed whereby we would broadly see the effects of such a change in educational assessment.
I'm curious - do you happen to know whether the essay portions ever offered any actual statistical utility above and beyond the multiple choice ones?
I don't think statistical utility was the goal - the goal might have just been to reduce math from 50% of the test to 33% of the test.
The (unproven, but widely repeated) story is that after Prop 209, too many of the wrong type of people were getting into UC due in part to high math scores [1]. California proposed dropping the SAT requirement as a result, and the College Board came up with a way to reduce it's demographic impact (and keep CA students paying them).
I'm curious - do you happen to know whether the essay portions ever offered any actual statistical utility above and beyond the multiple choice ones? As in, do they actually measure anything that the multiple choice questions can't?
I spent several years teaching SAT and GRE classes, I always noticed that I could predict people's essay scores pretty accurately based on their multiple choice scores. Percentile ranks always tended to be very close between the sections, which always made me wonder whether there was any point in having the essay at all. I always suspected that since the overall level of competence was so low on the SAT (sadly, the GRE was not all that much better...), anyone that was writing at the level where actual quality of writing matters was already scoring top marks on the essay, so they were essentially unmeasured by the scale.
I realize that when it comes to setting up these tests, "include an essay" is likely a political mandate, because people for some reason think that essay sections are "more fair" to "bad testers" or "fluid thinkers" or something like that (I'm pretty sure ETS was more or less forced to include an essay for this reason), so it probably wasn't even an option not to include them. But I'd be curious to know whether anyone ever looked into whether they actually told you anything you didn't already know. ETS does not provide this data or analysis, otherwise I'd check it myself as it relates to the SAT.