Take the NSF for example. (Pardon a US-centric example.) If you have a record showing that a particular PI has published 10 provably false/fraudulent papers, is the NSF really going to keep funding that person? At some point, the desire to avoid embarrassment and save their own skin may drive the NSF to de-risk by not funding the flagrant frauds.

Imagine if this were a world-class research institute, supported morally and with funding by people like Bill Gates and Elon Musk. Imagine if those people recognize that we can advance science and global progress much farther and faster if we tackle the epidemic of garbage research in academia (which I'd say is over 60%). Imagine running TV ads that say: "No, we are not right-wing nutjobs. There really IS a plague of garbage research."

(In fact, right-wing sponsors should be totally ruled out. As well as clearly left-wing sponsors. We want basically nonpolitical people who are basically trusted, like, again, Bill and/or Melinda Gates, Elon Musk, and so on. So, no Peter Thiel, which is unfortunate, because it would be right up his alley.)

But then you're publishing papers that try to replicate studies and specifically calling out specific frauds and getting their funding taken away, and more broadly arguing for funding agencies and academic departments to stop tolerating the culture of fraud and shoddy work. And you're calling out the worst offenders (in terms of academic departments and funding agencies) and calling on their political sponsors (like Congressmen) to cut their funding if they don't clean it up.

There may be areas of science where the foundation doesn't go, like climate change (maybe race?), because they are too heated. That's OK. It doesn't have to cover everything.

Part of the reason academia is really broken is, people aren't rewarded for proving that results can't be replicated. We need to find a way to incentivize that. The scientific process does not work without it. Also, academia doesn't allow people to just say, "This idea is wrong."

Academia is composed of a million little old-boys-club silos. You don't get into any of these clubs if you are not going to play ball. De facto, you cannot criticize your peers, if you are in academia.

For instance, the NSF has grant-award panels that are composed of researchers in the niche. So if you criticize your peers you literally cannot get NSF funding. Because your funding is literally gated by your "peers" as in "peer review." But if you were gonna be that way, you wouldn't have gotten a tenure-track job in the first place, because tenured professors only want people in the club to become their "peers" and sit on those committees.

I'm concerned I may be kind of overstating my case, but it's hard for me to know exactly how widespread the worst-case version of the problem is. Certainly what I'm describing can and does happen.

You're absolutely on the ball, except that everything is ~100x worse and more hopeless than you're describing :( Far from overstating it, you're understating it.

"If you have a record showing that a particular PI has published 10 provably false/fraudulent papers, is the NSF really going to keep funding that person?"

Absolutely. The problem is: showing fraud according to whom? I can point to dozens of papers by the same authors that I personally consider provably false/fraudulent, funded by the NSF. That isn't going to achieve anything. People can't even reliably get papers retracted, let alone cut bad researcher's funding.

From https://fantasticanachronism.com/2020/09/11/whats-wrong-with...

"If you look at the NSF's 2019 Performance Highlights, you'll find items such as "Foster a culture of inclusion through change management efforts" (Status: "Achieved") and "Inform applicants whether their proposals have been declined or recommended for funding in a timely manner" (Status: "Not Achieved") ... We're talking about an organization with an 8 billion dollar budget that is responsible for a huge part of social science funding, and they can't manage to inform people that their grant was declined! These are the people we must depend on to fix everything."

"with funding by people like Bill Gates"

Bill Gates is one of the primary funding sources for epidemiology, which is more or less dominated by pseudo-science and fake research. The article we're discussing was published in a journal of epidemiology so presumably the researchers polled for data were mostly epidemiologists, and only a few percent of researchers were honest about basic things like whether they'd share their data or not. You can imagine how bad the rest of it gets.

Gates doesn't care. Like all the other sources of funding for public sector research, he wants to be a philanthropist cheerleader for Science™. The act of spending money is the end, not the means. Whether that money is spent well and yields accurate conclusions is neither here nor there to him, as he has too much for that to be relevant.

I mean, you can't get 100x worse than "60% of research is invalid."

> Absolutely

If you get enough pressure on Congress, you can always make changes. I wish you wouldn't be so belligerent.

Society does change, but slowly. You would probably have said slaves will never be free, or women will never vote, or racial integration of schools will never happen.

> Gates doesn't care. Like all the other sources of funding for public sector research, he wants to be a philanthropist cheerleader for Science™.

You don't know that. Unless you've read his secret diary, or something. I suspect he does care. And I think the evidence weighs more heavily on my side of the argument. But you aren't debating the evidence; you're just claiming to know, which you obviously don't.

To make progress here probably need political parties that take up academic reform (or defunding) as a voter wedge issue, and campaign on it for many years. Based on the failure of initiatives that tried to solve these issues so far, I'm very skeptical of any effort that isn't based on some sort of populism. Realistically this means any attempt to dramatically improve standards is going to get tangled up in culture war, as after all, you can't divorce an attempt to raise scientific standards from specific fields and claims (if you do it's admitting that they should be ignored). To get political momentum you need to be able to point to examples of claims that are wrong, and they have to be claims people actually care about.

Re: 60%, that's your personal estimate, right? My impression is that it varies a lot by field. In some fields you don't really get much invalid research at all, it's a curiosity. In other fields 100% of papers are useless because the underlying premises of the field are themselves wrong.

"You don't know that. Unless you've read his secret diary, or something."

I haven't read his secret diary, no. I have talked to someone who worked in epidemiology on malaria research, who described to me how the field is totally distorted by Gates Foundation funding, I've read many of his statements throughout COVID and read a skeptical review of his non-secret book (if you can get past the invective at the start the review is pretty decent):

https://www.eugyppius.com/p/we-must-find-a-way-to-prevent-bi...

... which reinforced the overall impression: Gates is a cheerleader. His approach to find the hierarchically most important people, ask them what they think and then repeat it uncritically, whilst distributing grants to more or less anyone who says they'll make Gates's personal goals come true.

It's also the case that if Gates was reading the outputs of his funded researchers and is as smart as usually claimed, he would have long ago noticed the problems. Yet his book boils down to: what we need next time is way more of all that. He is aware lots of people think Ferguson is a fraud, but thinks that's only because they were misinformed by the press. As far as Gates is concerned Ferguson is great and he repeats Ferguson's defences of his own work verbatim, even though they aren't accurate. As the reviewer points out, it's impossible to believe given what's written in his book that he ever actually read Ferguson's research (I have read it, and his model code, very carefully). Gates' one concession is that the vaccines weren't the silver bullets they were promised to be, but as for everything else - well, he acts as if he's completely unaware that any problems exist.

Now, as you observe, this might be an act. Nobody wants to spend decades lavishly funding people to engage on a noble mission and then one day admit, actually, they were mostly scamming me and we didn't get much out of it. The loss of face would be impossible to handle. Even if Gates did know, we might expect him to act as if he didn't. Still we'd hope he'd find subtle ways to improve things without outright admitting to the problem in plain language. I've never seen any evidence of this.

At least with Gates there's the theoretical possibility he could have a Damascene Conversion and start enforcing rigorous standards. With governments it really does need to become a political issue before anything can happen, as every time people try and improve standards via government, or purely internally inside academia, the new rules seem to be immediately subverted and everyone carries on as before.

I agree that pushing for cultural/political change is a hard way to go given the populism in America. I'm an American. On a personal level, I think the country is so broken that it's not worth fixing, and would like to expatriate.

> Re: 60%, that's your personal estimate, right? My impression is that it varies a lot by field. In some fields you don't really get much invalid research at all, it's a curiosity. In other fields 100% of papers are useless because the underlying premises of the field are themselves wrong.

Yes, it was my estimate. But it's just a number I threw out there. I don't disagree with what you're saying here, but I'm more focused on the structural problems that exist to some (varying) degree in every field. For example, in every field, funding is corrupted. So I wouldn't say there are some fields that don't get much invalid research at all. Outright fraud is not the only kind of "invalid" research. It's still "invalid" if it's honest research but exploring the wrong path because there's some old fart at the funding agency who has a lot of friends who are exploring that particular path, which actually has been played out for 10 years already.

> Blog post about Bill Gates

I appreciate your sharing this. I think it's easy to construct this sort of narrative. That doesn't mean it's true. It might be. I don't know enough to tell.

I really think someone like Bill Gates could see the kind of research validation institute I'm proposing as a "win." I'm not saying all the research he's funding is worthless. I don't believe that, personally. I'm just saying, we can boost the quality and effectiveness of research if we have this kind of institute. Maybe our research process as a civilization gets 2x better. To use military terminology (ugh), it's a "force multiplier." It's not saying all our existing research is garbage. It's picking off the low-hanging fruit, the worst offenders, and thereby improving the signal to noise ratio of research overall. And once you get all the worst offenders, you can look for less low-hanging fruit.

I bet Gates would be happy to admit a lot of the research he funds is "low quality." That isn't a personal indictment against him. He'd probably argue that a lot is "medium quality" and a lot is "high quality." I wouldn't disagree. I'm sure there is some proportion in all three buckets.

As someone who lives outside of America, I really wouldn't be so down on it as a country. I'm lucky to live in a very nice part of the world (Switzerland!) but even so, America is a country and culture I hugely admire. I've spent most of my career working for American firms because that's where the action is, that's where the bravest people tackle the hardest problems. Yes, America is a land of extremes and the lows can be low, but the compensation is that the highs are really high.

"For example, in every field, funding is corrupted. So I wouldn't say there are some fields that don't get much invalid research at all. Outright fraud is not the only kind of "invalid" research."

Ah, I see. Well, I'd say that the funding mechanism makes invalid research possible/easy but doesn't necessarily directly create it. The lights are out but someone still has to misbehave. In some fields there just isn't much incentive to do that, e.g. consider computer graphics or the papers that explore more efficient K/V stores or compilers. You could try and cheat in those papers but why bother? You'd just be undermining any possible future jobs in industry, so I find these papers to be pretty reasonable.

The corruption really seems to kick off in fields that can be twisted into forms of social control in some way, or where there's not much chance of ever getting a good job in the private sector. Social sciences are a great example but public health is the same problem. People see an opportunity to change the world through misrepresenting their science, they see that nobody will stop them, and so they take it. Power is the goal and the apathetic funders are the enabler. You can't change the world or control anyone via compiler research though, so it stays closer to the original ideals.

Now, I agree that if you wider the problem scope to include irrelevant research nobody cares about, then indeed every field has big problems with that. It's probably too much to ask people to care about both invalid and irrelevant research at once though.

"I bet Gates would be happy to admit a lot of the research he funds is "low quality." That isn't a personal indictment against him. He'd probably argue that a lot is "medium quality" and a lot is "high quality." I wouldn't disagree. I'm sure there is some proportion in all three buckets."

Well if you or anyone else can get him to admit that, it'd be a great start.

I was a grad student in CS for 8 years, but left without finishing the PhD. I wasn't in compilers, but that's a reasonable example because I was in some other "niche of a niche." Small community, fairly obscure.

Most of the grad students in my niche really just wanted to be professors. And the way to become a professor was to (a) publish a very high quantity of papers; and (b) make friends with all the senior people in our little niche.

The goal people had wasn't to amass power. It was just to get a tenure-track job. Somebody else in this comment section made a joke about Chinese students. The problem is not at all limited to them. But a tenure-track job at an American university really seems like heaven to someone who's made their way up from the bottom in China, for example. And also to some people from other parts of the world, including America. If publishing tons of low-quality papers is the path to that, and being buddy-buddy with other people, they go for it. It's a "I'll scratch your back, you scratch mine" environment, including regarding "peer" review.

There is a tradeoff between quality and quantity. I was doing empirical research. I couldn't compete with the people doing more pure mathematical/algorithm stuff. They could just spit out papers with some new obscure algorithm (that will never be used anywhere) and a proof of some of its properties. I would have been lucky to have 5 published papers at the end of my grad school career, but a good tenure-track candidate would have like 30.

The senior people, like my thesis adviser, more than enable this kind of behavior. They get grant money basically based on the quantity of papers published. I never understood why my adviser cared so much about grant money. Like, what drove him to put out a super high quantity of crappy papers, to have a ton of students, and get a lot of grant money? What's the point? I never understood it. I mean, he already had tenure. And there were lots of tenured profs I knew who actually just didn't care about grant money and publication count, and didn't do that. Which is great. But you know who all the grant money goes to, and then who has a huge "lab" with lots of students? The ones like my professor who really care about that sort of thing and optimize for it.

I think compilers produces much higher quality than my niche-of-a-niche (which I don't want to name, by the way). But I don't think an area like that is immune to the pressures I'm talking about. It's probably a bigger community (which helps), where the research just has a different dynamic. We could speculate why that might be. But nonetheless I would argue that every area of modern science suffers from the problem I'm describing, to a varying extent between countries and fields and sub-sub fields.

By the way, maybe people go into compilers as PhD students, wanting to go to industry... but the truth is, in computer science, getting a PhD is usually not advancing your career over just putting that many years directly into industry. If your planned route is PhD=>industry, it only makes sense if you want to be in an industry research position and if you care about that more than how much money you make.