The brokenness of academia starts with the exploitation of graduate students by educational institutions, which makes it a very lucrative enterprise to over-hire professors, each of whom is like an incubator for the school, which acts as an investor because they get a cut (upper five figures per year per student) no matter what. It's similar to college sports in a way. It's all driven by money.
Most research is useless. Most professors are unneeded given the size of the problem space. Students see this and in the end, besides the very few true geniuses, it's not the most purely motivated that become professors, but the ones who most aggressively play the game in trumping up their results.
> Most professors are unneeded given the size of the problem space.
I disagree.
First, most subjects are too deep for the professor to be knowledgable about more than just their specialty (Do you expect a single Computer Science teacher to have complete and up-to-the-date knowledge of formal methods, programming languages, and operating system virtualization?) Even then, they will either be out of date or have spent a lot of time keeping up-to-date.
The problem is in many cases, the size of the problem space is much too big for a single group of researchers to have any effect on it. Many 'simple' studies have thousands of factors that can affect the outcome, and almost all of them have too few people, with not enough time and energy to devote to isolating all of those factors. For that reason (and many others), most studies are not replicatable, and dubious at best.
This is the main reason why psychological, sociological, medical, and most other fields of research that aren't mathematical, are considered dubious. Not because their methods are inherently bad, or their fields inherently invalid, but because most studies do not have the manpower available to do a completely formally-correct ideal study, so they have to make-do with what they have, and trust that eventually we will have enough mediocre-evidence studies that together account for enough varying factors on the subject that we can eventually account and iron out the individual flaws through statistical methods.
If research were _truly_ a priority for humanity, and we really dumped all of our effort into scientific research as a society (i.e. governments and companies both prioritized R&D and gave the scientists enough resources to actually do the jobs properly), then we might see these fields as "hard science" rather than "soft science".
But that's like saying, if Jeff Bezos got up and actually objectively used his money properly, he would have billions left over and there would not be starvation or poverty in the modern world. It's an idealistic scenario that is extremely unlikely to happen.
>>> Most professors are unneeded given the size of the problem space.
>most subjects are too deep for the professor to be knowledgable about more than just their specialty
>in many cases, the size of the problem space is much too big for a single group of researchers to have any effect on it.
>most studies do not have the manpower available to do a completely formally-correct ideal study
I agree that the size of the problem space requires a lot of researchers. This is probably why there's so much attention focused on machine learning and big data, since they seem to have the potential to address the problems you've listed here. Of course, there are many technical and ethical issues to be confronted in developing and deploying them.
>If research were _truly_ a priority for humanity, and we really dumped all of our effort into scientific research as a society (i.e. governments and companies both prioritized R&D and gave the scientists enough resources to actually do the jobs properly)
It's not, because the greatest problem for humanity is still subsistence, which requires solving a massive resource distribution problem. There are some governments and some companies that do prioritise R&D and provide enough resources, but they are too few and far between.
>psychological, sociological, medical, and most other fields of research that aren't mathematical, are considered dubious.
>then we might see these fields as "hard science" rather than "soft science".
It seems to me that these fields are "soft" in part because the ethical issues surrounding the surveillance that's needed to collect the data to do a formally correct study are quite formidable.
You can view an undergraduate education (and especially a graduate education) as a significantly negatively paid (or at best unpaid) internship for the job of academia.
This has all the advantages of internships that employer's normally enjoy. The internship enables you to hire significantly better employees than you'd otherwise be able to get in the open market, because you can engage in more vetting and because of the power of defaults.
Many smart people go into academia who would have been much happier and more productive outside of it simply because going to school itself made pursuing a job in academia something much more of a default than it would have been for many people.
Most research is useless. Most professors are unneeded given the size of the problem space. Students see this and in the end, besides the very few true geniuses, it's not the most purely motivated that become professors, but the ones who most aggressively play the game in trumping up their results.