I am a scientist, and I don't think anyone knows how to do this. There also is very little skin in the game for this. Bureaucrats want dumb metrics that they can point to to determine success regardless if that metric is meaningful or not. There is a strong pressure to publish quickly, which directly conflicts with a pressure for quality. Many prominent Nobel Laureates, Fields Metal Winners, Turing Prize Winners, and so on have discussed how they themselves could not have thrived in the current environment and its insanity. But who is going to change this? Honestly the ones that are hurt the most are the grad students at the non-top 10-20 universities. Everyone else has found "success" in the system (as in learning how to play that game) and they are highly incentivized to maintain that system and their status. Research is a completely different game today than it was even 20 years ago and we have not adjusted our system accordingly, and worse, many want to pretend that it hasn't changed. An interesting simple example is that research teams have exploded in size (especially at large universities) which makes for a lot of Nobel drama (max 3 per team). But there are also many other simple and more nuanced points that exist, but few want nuance. Either way, I'm absolutely certain that our current metrics do not strongly align with producing good science.

Oh, of course. Most academics don't do it for the money. It's not like they're paid much. Though that can incentivize cheating in another way because they want money. But there will always be people that don't give a fuck and just want to do good research regardless of the metrics being used. I am insistent that to be a good scientist you must be somewhere on the side of "anti-authority." Because your job relies on challenging concepts, and especially well known and widely agreed upon concepts. Generally look for people who are passionate, will rant, but importantly rant with nuance. Those are the people passionate about their research and not the metrics.

The problem is we're just throwing a lot of money down the drain, wasting a lot of time, and generating distrust of the system. Any "crash and burn" is never going to lead to an extinction in any sense. It's just about general society level if we want to do good science or just do noisy science (all science is noisy). But you can never do good science if you remove nuance. This is why I hate that ML (and CS in general) uses conference systems as the main platform for publishing. It is ridiculous to think you can have a good system when it is highly competitive, zero sum, highly time consuming, there's no discussion between authors and reviewers (you may get a one page rebuttal, but your reviewers comments are often disjoint and vague), and you're being judged by those you're competing with. It's just a silly notion to believe this is useful.

The solution is actually not hard. I refer to the larger phenomena as Goodhart's Hell. The solution is to stop using metrics as targets. Metrics are guides. If you don't have a deep understanding of what your metric actually measures, how well your metric aligns to the thing you're intending to measure (never 100%), and if you don't understand the biases to your data, you're fucking doomed to this bureaucratic hell. Noise is inherent to complex systems and aggregation is the bane of evaluation of complex feature spaces. Just remember that its models all the way down and that all models are wrong (despite some models being better than others).

I think that will happen when we swap peer review for peer replication.