Somehow the idea that this is all caused by people applying to more places doesn't cut it to me. It might account for some of it, but not all of it. A lot of this is probably fairly chaotic and complex. For example, the GI bill probably contributed directly somewhat also, but also probably led to cultural shifts because of the veteran population, which then in turn changed career and educational expectations, but the GI bill itself reflects WWII, which also reflects cataclysmic societal changes (that is, WWII caused many significant things, but was also caused by many significant things).
Speaking as someone who has done research on selection and standard testing, and whose first-hand experiences support what I see from the theory and research, this is kind of horrifying, because it inevitably leads to increases in bullshit.
The accuracy of any of the selection methods we use in education is very poor. It just is. Standardized tests, "holistic review", all of it. It's not useless, but it is poor. It's the sort of thing that works well for a certain level of selectivity, but not past it. Once you go past it, you're selecting on well-impressioned noise, and incentivizing bullshit.
Somehow this all seems related in my mind to the rise of concerns about replicability in science, college admissions scams, and the age of fraud in general (https://www.theatlantic.com/entertainment/archive/2019/04/th...). Something is broken. Probably many things.
> The accuracy of any of the selection methods we use in education is very poor. It just is.
The SAT when combined with the high school GPA (HSGPA) has an adjusted correlation correlation coefficient of 0.56 with first-year GPA, meaning the combined measurement accurately predicts how a potential college applicant will perform in their first year of college 56% of the time. [1]
That's actually pretty good, what other proposed metrics can say their signals match outcomes with 56% validity? How much you liked their essay?
Lower SAT scores have about 63% retention rate for first-year students whereas high SAT scores have about a 95% retention rate [2]. That is, high schoolers with poor SATs drop out of college about 40% of the time in their first year.
Standardized tests have many problems -- obviously -- but no one has developed a less unfair system.
When colleges abandon standardized tests what else are they relying on? Random signals made up by admissions officers? That's worse than job interviewing.
I have no problem criticizing standardized testing, but I feel everyone who does should be obligated to propose a better alternative method with a higher validity rate than 56%.
> The SAT when combined with the high school GPA (HSGPA) has an adjusted correlation correlation coefficient of 0.56 with first-year GPA, meaning the combined measurement accurately predicts how a potential college applicant will perform in their first year of college 56% of the time.
I'm summarizing for a general audience. I could say, r is " the strength of the linear relationship between two variables on a graph" but I'm not sure that helps the average person understand the connection.
If you have a better description, it's more helpful to chime in with that instead of "You're wrong!"
A better summary would be that those two quantities explain about half of the variation, not that they predict accurately half the time.
If you took a random sample of cases, half of them wouldn’t exhibit a direct relationship b/w SAT and first year GPA and half nothing (unless the data is _super_ weird). Instead, SAT would be instructive-ish in predicting first year GPA for all those cases.
Explaining half the variation, and the other half?
The point was to draw a connection for the general audience, not present the most scientifically accurate description of a relationship between two variables -- that's what the links to the research are for.
It's good to communicate for a general audience, but your presentation misleads rather than simplifies.
> meaning the combined measurement accurately predicts how a potential college applicant will perform in their first year of college 56% of the time.
"accurately predicts...56% of the time" implies that half of predictions are 'accurate', which most readers would interpret as 'correct' i.e. knowing SAT + HSGPA allows you to state FYGPA _exactly_ for about half of cases. That's not what the research you cited says. Rather, the square of the multiple correlation R (which is exactly R^2, the coefficient of determination) indicates how much of the variance in the output variable is explained by the input variables. That quantity _must_ be communicated in terms of the strength of the relationship, not accuracy for a given or share of cases as it doesn't tell us anything about a given case. One could say it tells us about 30% (0.56^2, correction from my statement above) of the information we'd need to know to perfectly predict the outcome, or that the relationship is better than random, but doesn't predict perfectly, or ...
Additionally, table 5 of the link you cited indicates the adjust correlation coefficient b/w FYGPA and the combination of HSGPA and SAT is 0.62. None of the numbers in that table are 0.56, so I'm not sure where you pulled that exact number from. I've used 0.56/56% above to be clear which quantity I'm referring to.
Uh...this is exactly what I mean. Your description is 100% scientifically accurate but probably way beyond the average reader.
Again, if you can simplify this in a way more accurate than I have, great, be my guest-- I look forward to reading it.
R^2, coefficient of determination, output variable variance, etc etc -- most readers aren't going to go that deep in the math. For those who do, like you, the links to the actual research is provided.
But so far all I see are data scientists complaining about how my description is not 100% statistically accurate without providing any alternative explanation that doesn't devolve into variance of output variables.
Again, be my guest to show me I'm wrong, but what you wrote above is not something that would be easy to understand for the general audience, IMHO.
The concerning (mis)interpretation of your statement is what I said on the third line above:
> "accurately predicts...56% of the time" implies that half of predictions are 'accurate', which most readers would interpret as 'correct' i.e. knowing SAT + HSGPA allows you to state FYGPA _exactly_ for about half of cases.
This interpretation is easy to arrive at, and clearly does not correspond to a reasonable understanding of the source, even for a general audience.
I provide two suggestions above:
> One could say it tells us about 30% of the information we'd need to know to perfectly predict the outcome, or that the relationship is better than random, but doesn't predict perfectly
OK, but that only makes it more confusing. You say it tells us about 30% of the information we'd need to know...which makes it sound (to the lay person) like there's no connection because 70% of the information is elsewhere!
I appreciate your commitment to academic rigor but sometimes oversimplifying things, even at the cost of mathematical accuracy, is enough for a general audience who aren't going to compute variance of output variables.
This isn't about academic or mathematical rigor - this is about responsible communication of statistics.
You're right that a general audience isn't going to look at the source, nor think about variance of output variables. Therefore, it's the responsibility of us as communicators of statistics to relate the conclusions that can be drawn from the data in a way that first and foremost is not wrong or misleading, and secondly captures the concept as accurately as possible for the audience.
The first principle is the overriding obligation. Your simplification can capture as little of the information and conclusions supported by the data as you want, but it cannot imply or state conclusions that are not supported.
You're getting this response to your statement, from me and others, because your interpretation of the source can easily be read as drastically overstating the character (and strength) of the relationship supported by the data - even if that's not what you intended.
To you this is the major responsibility, but not to everyone.
I'm getting this response to my statement, from you and many other data scientists, who can't accept oversimplifying math, but all of whom have failed to produce a general description in plain English.
In some ways it's like the test-- everyone hates it but no one has a better alternative. You've complained maybe 7 or 8 times in this thread about how scientifically inaccurate my general summary is but have not produced a description that a regular person could understand in 5 words or less, with no technical jargon.
"Something is partially explained by a combination of numbers"?
That says nothing semantic of value. Better to exaggerate the causal relationship and give a sense of meaning than offer meaningless generics like that, because at least the general reader intuits a sense of the universe of the relationship. The above implies nothing.
1) Your description is 0% correct; and
2) It's not a complaint. Maybe you don't know what a correlation coefficient is. That's okay -- I don't know what polyfinite rings are.
The problem is not plain English, the problem is that you want to make claims that are more specific than your numbers allow you to.
"Correlation is a number in [-1,1] where -1 means predictably inversely related and 1 means predictably directly related". What is correlation 0.5? There isn't even an unique definition of "correlation", man.
The best thing you could do is to produce a table of correlation coefficients for comparison. That would communicate some qualitative nuance as to what a 0.5 correlation really means.
You can also try to do something like a bootstrap or kernel density or use a prior distribution to claim that the probability that increment dx leads to an increment of dy or more is... That's more technically involved, but satisfies your desire of saying something specific that correlation coefficients do not allow you to say.
Kernel density, bootstrap, using prior distribution, increment the derivative...all this stuff is so far above and beyond the ordinary person.
You and a couple of others have provided plenty of context for the serious data scientist who wants to understand the exact nature of the relationship between these two variables, but 99% of readers do not understand much less want to read about "kernel density".
Again, my challenge remains open: instead of complaining about inaccuracy, try to render the relationship in plain English.
Lol. I was giving a basic intro for the 99% of readers who don't care about "kernel density" or controlling for output variables -- the fact that you conflate such an attempt with a "quantitative analysis" (which it was not) coupled with your unwillingness or inability to give a plain English description just proves my point.
That's not summarizing. "It's the strength of the relationship" is summarizing. "The combined measurement accurately predicts how a potential college applicant will perform in their first year of college 56% of the time" is just wrong. See Anscombe's quartet for a great example of why it's just plain wrong.
"It's how perfectly you can fit a straight line to them."
You can be mathematically accurate without being mathematically precise. Better imprecise but correct than incorrect but precise.
If you're trying to give a quantitative lay picture of what exactly 0.56 linear correlation means, you need to still be quantitatively right, while the above are quantitative. Pictures and examples can help. "For perspective, 0.56 is about the correlation between <example> and <example>"
Saying there is a quantitative strength to a relationship is, to a regular person, meaningless. Am I .56 in love with my wife?
Can I fit in a straight line to her?
These are not good descriptions. Of course HN is full of data scientists who wildly object to oversimplifying statistical relationships -- luckily you are here to give the detailed mathematical context. But these are not simplified descriptions for a general audience.
> Saying there is a quantitative strength to a relationship is, to a regular person, meaningless.
That the average person would not understand a particular accurate description of a subject does not, in and of itself, make a completely inaccurate alternative description less wrong or even a good simplification.
>The SAT when combined with the high school GPA (HSGPA) has an adjusted correlation correlation coefficient of 0.56 with first-year GPA, meaning the combined measurement accurately predicts how a potential college applicant will perform in their first year of college 56% of the time.
This is a totally incorrect interpretation of what correlation is.
Again, I'm summarizing for a general audience. If you have a better way to describe it that doesn't devolve into polynomials and linear relationships between variables on a graphs it's more helpful to do so than just say, "You're totally incorrect!"
Correlation is not probability. You can't compare them at all. Flipping a coin for each student would produce a correlation of 0, far lower than the correlation of 0.56 cited above. Have a look at some plots of data [1] with different correlation coefficients to see how dramatic it can be. Note the difference between r = 0.00 and r = 0.60. That's about what we're dealing with here.
The SAT reliably correlates with IQ, and of any psychometric variable we are able to measure, IQ has the strongest correlation with long-term socioeconomic success. Insofar as it is useful to funnel smart people into college, using the SAT is a good way to filter them.
I'm curious, as someone who was slightly involved with a documentary [1] exposing pitfalls of standardized testing. Generally the SAT only has shown a weak correlation between test scores and first year (some studies I read showed only first semester, but I don't have them on hand right now) collegiate performance [2].
>Generally the SAT only has shown a weak correlation between test scores and first year [...] collegiate performance [2].
That's not what your cite says. The InsideHigherEd article actually shows a strong correlation between SAT and grades but Aquinas identified a minority % of schools where it didn't. Please carefully read the 3 bullet points again and notice the minority percentages.
Your qualifier of "Generally" in your comment is misrepresenting Aguinis' findings.
I hate that piece because most of what he's referring to has been dealt with for decades. It's like "narcissist who knows nothing about a field doesn't bother to learn anything about it and as a result takes down a strawman to make himself look good."
Taleb should stay in his lane. He sounds like a "get off my lawn" old douche when he talks about anything other than financial markets. His central tenet in that article is that IQ doesn't correlate with wealth and is therefore useless. How enlightening.
There is a Supreme Court ruling that IQ tests are assumed to unfairly disadvantage minority candidates. This was in the context of employment rather than college admissions, but I don't think anyone has been bold enough to test whether it applies.
Broadly speaking, IQ tests tend to include a lot of cultural assumptions and therefore members of the in-group will test higher than members of out-groups of the same aptitude. IQ tests are therefore treated as discriminatory (disproportionate impact) "by default." The administration of a particular test for a specific purpose can usually be OK'd by either showing that particular test is not discriminatory, or by showing that particular test has a measurable correlation to the specific purpose for which it's used. My understanding is that "specific purpose" in the employment context means the individual job description, not just hiring in general.
> There is a Supreme Court ruling that IQ tests are assumed to unfairly disadvantage minority candidates
I wonder if there's an "IQ test" that unfairly disadvantages non-minority candidates. It would be interesting to see the results if the same pool of people (minority and non-minority) take both tests.
SAT presumably tests you have learned certain things in high school and so are ready for university. E.g. you won’t show up to your first university math class needing to learn all of high school math, and thus be unable to complete the class.
IQ is a lot like GDP. It doesn't measure what we need as accurately as we need. If you keep that in mind, though, it can remain a broadly useful metric.
So I agree with you about SAT <-> IQ, and IQ <-> SES. However...
IQ isn't the only correlate of success. I think if you look at it all, there's a large dose of conscientiousness (which standardized testing companies are now going after), and attractiveness/charisma...
... but that's on the individual side of things. There's also a whole host of societal and random stuff that is outside the control of the individual, or maybe is significant to a person, but that studies tend to treat as irrelevant.
Also, saying that the SAT is useful as a selection device doesn't mean it's the only useful selection device, or that as a selection device it's very good. Offhand I don't remember the numbers, but standardized test score probably correlates .45 or so with first-year college grades? Think about that for a second. First, that's a ton of noise. That's not very predictive at all. Second, that's first-year college grades. Change the criterion but it's still the same: your best tool really is a pretty fuzzy predictor.
So now take this very fuzzy predictor, add some other fuzzy predictors that at best get things up to like maybe .6 correlation? Still fuzzy. Now you're going to be really selective on these things? What you're going to end up with is a lot of people who would have done as well but for whom the stars didn't align right at a particular period in their life. But now we as a society have this crazy income inequality, rent-seeking monopolies of all forms, and a general winner-takes all climate, so these small meaningless differences get amplified tenfold.
The conversations about this too have this kind of all-or-nothing quality, like you're forced to choose between "standardized tests are meaningless" or "standardized tests are valid predictors for a large portion of people so we need to treat them as infallible predictors for everyone." The truth is really much greyer than these positions: yes they predict, but they predict pretty weakly, all things considered, and generally for people who fit into a certain box. This all might be fine, except now we've structured our society in part around these oversimplified assumptions, pretending it works when it doesn't really. It might be ok when there's lots of second chances, lots of opportunities for people to get back on their feet from the vagaries of life, and good opportunities in general for everyone, but when resources gets hoarded by fewer and fewer, there's more noise, that's compounded by people gaming the system, etc. etc. etc.
Just as a thought exercise: what do most human traits look like in terms of distribution? They're pretty normal, pretty Gaussian. What does income look like? Not that, not at all. The discrepancy between them should be shocking to everyone.
The difficult solution is increasing the accuracy and “trustability” of k12 assessments. As is, high school educators are not assessing their student in a manner that allows for real judgements of their students’ knowledge/capabilities/creativity/etc. by an outside party. We spend immense amounts of money, time, and effort creating and assessing students in school, but the results are only useful with the context of progressing within those specific courses.
Trustable assessments would go a long way to accurately rewarding merit.
Currently standardized testing better predicts future academic performance than years of GPA, which in a way is the sum of teachers’ views on a student.
But teachers still prefer teachers over standard tests, and they shun conversation with their researching peers. The classroom is their kingdom and that’s how people prefer it.
“Teacher tests” often have different objectives than solely measuring one student’s ability compared to another. Likewise, the investments made in the creation of standardized tests are not likely to be matched by classroom teachers due to logistical constraints even if they have the necessary knowledge to create an equally valid assessment.
The objective should be the creation of a system that allows teachers to retain the flexibility to modify curriculum in a manner appropriate to their specific student population while also maintaining the trust and reliability benefits found in standardized tests.
Maybe it is our expectation for everything to be fair, straightforward and honest that is broken?
These expectations are the root of the current outrage culture. It's as if now that we have these wonderful technological advances we assume every problem should have been solved by now, and get outraged when we realize we don't live in a utopia.
So... I've been programming in R for over 20 years. I've been ready for an alternative for performance reasons for about half that time. Julia seems like a promising alternative, that I wish I could use all the time instead of R but it's just not there; I'd prefer Nim most of all but that's even less well-resourced in terms of libraries. Maybe a zero-cost abstracted offshoot of Rust will eventually come to have a role? Who knows.
R, like Python, has far outgrown its initial scope. I don't think it was initially envisioned to be used the way it is today. But both have been kept in use as costumes for C/C++.
One of the things I've noticed the most in the last 20 years, to your point, is that the language used to be a lot more straightforward and simpler, more predictable. Over the years a lot has been added in a sort of haphazard way, and as a result today you have this kind of Frankenstein language that isn't what it started with.
As for data structures, though, I don't really see R as being that different from other languages. Many of them are the same as in other languages, but just have different names (and I do wish they used similar terminology). Others have been taken up in other languages as people have come to appreciate their utility.
Being a wrapper for C/C++ can only go so far. Eventually you have to write in R (or Python) and the speed shows, if you have enough data to deal with.
I think what a healthy diet is fairly individual-specific. I think there are probably general guidelines that are pretty good for most people, but the details depend on how different people metabolize different things, their taste preferences etc.
I think I feel the best eating vegetarian, whole grains, lots of fruits and vegetables, nuts and legumes, no restrictions on unsaturated fats but little or no saturated fats. I basically try to avoid processed foods, saturated fats, artificial colors, artificial flavorings, and preservatives, and meats (including fish). I try to avoid added sugars too but am not too strict about it, and don't have any restrictions on complex carbs. I've kind of done intermittent fasting before it was a "thing" and I do think that's been helpful/feel better with that at the right times, as has not eating after a certain time at night, and keeping breakfasts light or waiting a bit later.
It's worked for me but I think everyone's optimal diet is probably a bit different. I think in general you can't go wrong with whole grains and nuts, whole vegetables, and whole fruits. The rest of it is kind of tweaking for an individual.
I think the poster was just pointing to the parallels. It's salient to me as a professor.
FWIW, I think the issues are definitely related though. There's lots of STEM hype in academics as well as in commerce which creates a labor oversupply in both.
The problem is that there are genuine advancements in STEM that are interesting and rewarding, from the academic as well as commercial side, but they're sort of buried in a sea of bullshit, and being able to connect with the worthwhile stuff (financially or otherwise) can often be due to random or sketchy reasons. So people see the great stuff going on, and then either promote it (in the case of politicians and public figures) or pursue it (as someone seeking a career) not realizing that throwing more people at something will necessarily produce more of what we like.
So you end up with the Gallipoli that is modern academics.
Speaking as someone who has done research on selection and standard testing, and whose first-hand experiences support what I see from the theory and research, this is kind of horrifying, because it inevitably leads to increases in bullshit.
The accuracy of any of the selection methods we use in education is very poor. It just is. Standardized tests, "holistic review", all of it. It's not useless, but it is poor. It's the sort of thing that works well for a certain level of selectivity, but not past it. Once you go past it, you're selecting on well-impressioned noise, and incentivizing bullshit.
Somehow this all seems related in my mind to the rise of concerns about replicability in science, college admissions scams, and the age of fraud in general (https://www.theatlantic.com/entertainment/archive/2019/04/th...). Something is broken. Probably many things.