> And still people come to me with some bug that they've experienced on their machine, that I cannot reproduce on my machine
But this is the other way around. Have you ever written a program that doesn't run anywhere except a single machine of yours? Would you release it and advertise it and encourage other people to use it as dependency in their software?
If it only runs on one machine of yours, you don't even know if your code is doing something, or something else in the machine/OS. Or in terms of science, whether the research says something about the world, or just about the research setup.
I think you misunderstand the point of scientific publication here (at least in theory, perhaps less so in practice). The purpose of a paper is typically to say "I have achieved these results in this environment (as far as I can tell)", and encourages reproduction. But the original result is useful in its own right - it tells us that there may be something worth exploring. Yes, it may just be a measurement error (I remember the magic faster than light neutrinos), but if it is exciting enough, and lots of eyes end up looking, then flaws are typically found fairly quickly.
And yes, there are often overly excited press releases that accompany it - the "advertise it and encourage others to us it as a dependency" part of it analogy - but this is typically just noise in the context of scientific research. If that is your main problem with scientific publishing, you may want to be more critical of science journalism instead.
Fwiw, yes of course I've written code that only runs on my machine. I imagine everyone has, typically accidentally. You do it, you realise your mistake, you learn something from it. Which is exactly what we expect from scientific papers that can't be reproduced.
> But the original result is useful in its own right - it tells us that there may be something worth exploring.
I disagree. It shows that when someone writes something in a text editor and publishes it, others can read the words they wrote. That's all it shows, by itself. Just like someone writing something on the web only tells us that a textarea accepts just about any input.
And even if it did show more than that, when someone "explores" it, is the result is more of that, something that might be true, might not be, but "is worth exploring"? Then at what point does falsifiability enter into it? Why not right away? To me it's just another variation of making it someone else's problem, kicking the can down the road.
> if it is exciting enough, and lots of eyes end up looking, then flaws are typically found fairly quickly.
If that was true, there wouldn't even be a replication issue, much less a replication crisis. It's like saying open source means a lot of people look at the code, if it's important enough. Time and time again that's proven wrong, e.g. https://www.zdnet.com/article/open-source-software-security-...
> yes of course I've written code that only runs on my machine. I imagine everyone has
I wouldn't even know how to go about doing that. Can you post something that only runs on one of your machines, and you don't know why? Note I didn't say your machine, I said one machine of yours. Would you publish something that runs on one machine of yours but not a single other one, other than to ask "can anyone tell me why this only runs on this machine"? I doubt it.
I think you may be seeing the purpose of these papers differently to me, which may be the cause of this confusion.
The way you're describing a scientific publication is as if it were the end result of the scientific act. To use the software analogy, you're describing publication like a software release: all tests have been performed, all CI workflows have passed, QA have checked everything, and the result is about to be shipped to customers.
But talking to researchers, they see publishing more like making a new branch in a repository. There is no expectation that the code in that branch already be perfect (hence why it might only run on one machine, or not even run at all, because sometimes even something that doesn't work is still worth committing and exploring later).
And just like in software, where you might eventually merge those branches and create a release out of it, in the scientific world you have metastudies or other forms of analysis and literature reviews that attempt to glean a consensus out of what has been published so far. And typically in the scientific world, this is what happens. However, in journalism, this isn't usually what happens, and one person's experimental, "I've only tested this on my machine" research is often treated as equivalent to another person's "release branch" paper evaluating the state of a field and identifying which findings are likely to represent real, universal truths.
Which isn't to say that journalists are the only ones at fault here - universities that evaluate researchers primarily on getting papers into journals, and prestige systems that make it hard to go against conventional wisdom in the field both cause similar problems by conflating different levels of research or adding competing incentives to researchers' work. But I don't think that invalidates the basic idea of published research: to present a found result (or non-really), provide as much information as possible about how to replicate the result again, and then let other people use that information to inform their work. It just requires us to be mindful of how we let that research inform us.
> Replication is one of the central issues in any empirical science. To confirm results or hypotheses by a repetition procedure is at the basis of any scientific conception. A replication experiment to demonstrate that the same findings can be obtained in any other place by any other researcher is conceived as an operationalization of objectivity. It is the proof that the experiment reflects knowledge that can be separated from the specific circumstances (such as time, place, or persons) under which it was gained.
Or, in short, "one is none". One might turn into more than one, it might not. Until it does, it's not real.
more snippets from the above WP article:
> This experiment was part of a series of three studies that had been widely cited throughout the years, was regularly taught in university courses
> what the community found particularly upsetting was that many of the flawed procedures and statistical tools used in Bem’s studies were part of common research practice in psychology.
> alarmingly low replication rates (11-20%) of landmark findings in preclinical oncological research
> A 2019 study in Scientific Data estimated with 95% confidence that of 1,989 articles on water resources and management published in 2017, study results might be reproduced for only 0.6% to 6.8%, even if each of these articles were to provide sufficient information that allowed for replication
I'm not saying it couldn't be fine to just publish things because they "could be interesting". But the overall situation seems like quite the dumpster fire to me. As does software, FWIW.
If you can't even replicate it yourself, what makes you think peers could? We are talking about something not being replicated, not even by the original author. The most extreme version would be something that you could only get to run once on the same machine, and never on any other machine.
But this is the other way around. Have you ever written a program that doesn't run anywhere except a single machine of yours? Would you release it and advertise it and encourage other people to use it as dependency in their software?
If it only runs on one machine of yours, you don't even know if your code is doing something, or something else in the machine/OS. Or in terms of science, whether the research says something about the world, or just about the research setup.