The rise of 'scientism' - until a few years ago I considered myself a 'science type' (excuse the phrase), immersed in logic all my life.
Recently the depth of negative influence, corrupt findings, blind faith and religious-style zeal within the scientific community and pseudo scientific communities (like Reddit) has become absurd and embarrassing.
Science is a method, not a conclusion. I hope it becomes 'open', and fast.
Science has not changed, nor has bad science. There gave always been incentives to produce bad science, just like there has always been a moral incentive to produce good science.
The religious zeal I think comes from fan boys/girls not from scientists themselves.
The increase in retractions can mean either that (1) there's more bad science now than there was before, or (2) we've got better at spotting bad science. Without further evidence, it's difficult to tell which one is the case. Perhaps it's a mix of both.
Dear god of science, I pray that you will lead me to the truth, I pray that your methods, applied with utmost vigilance, will vanquish the darkness. I pray that my faith in you remains unwavering. Amen.
I wonder whether a 'GitHub' for Science wouldn't help things along, just as it's helped the open source software community.
Scientist could have repos with full version control, so you could go back and look at their experiments from day one. All their data would be accessible, as well as their statistical analysis, and you could see the evolution of their written paper.
The repos could start off private, and then opened on the day of publishing, so others couldn't take credit for their work.
> Scientist could have repos with full version control, so you could go back and look at their experiments from day one. All their data would be accessible, as well as their statistical analysis, and you could see the evolution of their written paper.
> The repos could start off private, and then opened on the day of publishing, so others couldn't take credit for their work.
Funny you should say that...
I work at the Center for Open Science, and our founder was quoted in the article. One of the major use cases for our primary product (the Open Science Framework) is to facilitate preregistration.
With an analogy to open-source, my failed projects (like we're talking about here) fall largely into two distinct categories. Those that I never started, or were totally half-baked, or turned out to trivial - the uninteresting or incomplete. 90%. And 5% turn out to be illustrative learning moments.
One of my first OOP projects trying to model the real world in hierarchical classes. It was a good non-contrived example of a need for multiple orthogonal inheritance methods as any one can easily be forced into contortions in modeling even simple structures.
I tried to write a project with my own trig implementations, just because. That'd at least be funny...
But the rest are a few lines of drunk-coding, or something I found a library to do and never developed, or in some other way a waste of time to look at.
So I don't want to hold scientists to standards I wouldn't want. Nor do I want to miss the value in the noise.
So maybe we should have a honor-system somewhat like this - when you first describe to project to anyone looking for funding, or discussed it with more than a handful of peers, or ... then you make a note in your lab book. "Research direction: see if flatworms are really flat because Y - hope it'll give insight into Z". And your internal audit system submits all of these to a pool of peers who pick by interest in the area. If it's chosen, an auditor checks if the idea was ever really developed, dropping it if not, and summarizing whatever is there (or letting you write a paper) if there are results.
It sounds like a lot of work, but considering that everyone is asking for grants and being audited anyways, and this would replace vast swaths of less-pointed simple bookkeeping audit with peer-review of your methods, etc. In other words, likely higher quality results for the funding bodies in less time, and with the additional benefit of being able to publish interesting failure results.
Thus achieving a balance between researcher privacy and ability to try without public mockery on failure, funding efficiency, and scientific benefit.
Very little of what is actually needed to reproduce an experiment goes into academic papers. What you'd really want is a more formal, digital version of the "lab book". Experimentalists are taught, from their earliest days as an undergrad, to keep notes on what they're doing. The idea is to put everything useful in there. Got a new piece of gear with a long startup procedure? Put it in your book. Designing an experiment? Put the design in there. Ordered parts? Make a note of it. Running an experiment and finally found a way to do things that works? Write down the procedure. Got some results from your experiment? Print them out and paste them in your book. Saw something odd? Write it down. The result is a linear log that tracks a lot of stuff, both useful and useless. With experience, you learn to make use of tables of contents and margin notes, but searching through them for something dimly recollected can sometimes be time consuming. Lab books are where the gory details of how to actually do things is stored. The procedures, diagrams, and details in them are simply not conveyed in papers because there's no space for them, and many are viewed to be "common knowledge", although this kind of knowledge is probably only common to a small number of labs in the world.
The problem with these lab books is that they're often the personal property of the scientists and, as such, almost a sort of diary. They're written with yourself as the intended audience and can sometimes be impenetrable for anyone else. If you're working in a commercial lab, as opposed to research in academia, these books might be considered the property of your employer. In this situation you might try to make them more friendly to other human beings, but that often isn't a priority.
Putting things into a digital form would, in general, take a lot longer. Even with good proficiency in LaTeX, complex equations are quicker to just write on paper. Hand-drawn diagrams are also very common in lab books, and the software/hardware to draw easily in a digital form is not yet ubiquitous. Finally, digital storage occasionally dies. A lab book is a precious possession whose loss can cause months or years of difficulty. Physical hard-copy is easier to store long-term. Some people have undergraduate lab books from half a century ago sitting on their office bookshelves.
There is nothing that is technologically insurmountable to moving to a digital lab book format. Microsoft's surface tablets make digital drawing pretty painless. Cloud storage could be maintained long-term. Etc. The benefits of digital lab books could be considerable. They'd be easier to search and distribute. You could include copies in a digital appendix to an experiment (currently, most journals discourage large attachments because they don't like to pay for storage). To my knowledge, nobody has actually put everything that's needed together into a single cohesive program. You'd want a note-taking program with full LaTeX and freehand drawing support. You'd need to be able to paste in tables, graphs, and data files easily. It would need to have a long-term archiving solution and a really good interface. It would need to be as fast or faster than writing on paper for pretty much everything. It would also probably need to be open source, as scientists would probably not trust proprietary software that might make their notes difficult to access at some point in the future. Finally, the hardware to run it on would need to be ubiquitous.
> Very little of what is actually needed to reproduce an experiment goes into academic papers.
This is also an issue where investigators apply custom software to complex systems. Getting source code can be nontrivial, and documentation is often inadequate. And there may be bugs that were avoided rather than patched.
>The problem with these lab books is that they're often the personal property of the scientists and, as such, almost a sort of diary
Further complicating things, individual researchers don't actually own the books. Often, an institution or the lab owns the rights to the book. This can make individuals who are otherwise keen on "open science" hesitant to place copies of this information in a publicly accessible place.
See page 4 of this manual from the NIH (National Institutes of Health) about the ownership of notebooks.
Everything on that page is stuff you're going to find in a grad student's lab books, which are their property. This document is basically a primer for people transitioning into paid research where the lab-books become lab property. A huge proportion of research, especially in less commercializable areas of fundamental science research, is done by students (because they're cheap). This means the kind of lab-book described in your link is probably less common that you might expect!
I'd imagine you can find guides on university websites. e.g. Search for physics, chemistry, etc. lab book format guides, etc.. These tend to be pretty formulaic and oriented towards ease of marking. However, they do establish habits that persist when experiments become life-consuming affairs rather than something that start and finish in a three hour period.
While I think increased accountability is a good thing for the world of science, I'm concerned this will fan the anti-intellectual, anti-science rhetoric.
Science, like every human institution, is flawed. That's not a reason to reject everything about it.
It is annoying to see a retort "scientists always end up being wrong". But I don't think science should sink to the level of fake-confidence "everything going according to plan" of non-science, I think it should embrace its honesty about its mistakes.
Over time people can't help but depend on science's results, and new generations take it to heart. It's actually similar to how people can't help but depend on open source tools/platforms, as much as they hated "free-tards" and said "you get what you pay for" in the past.
Well, the whole point of science is to falsify past theories and produce better theories. So of course the majority of scientific theories will end up being wrong at some point.
So it's pointless to try to rebuke the "scientists always end up being wrong" like of argument. If anything, I think scientists should just bite the bullet. Theory proven wrong? That's proof that science works!
But it is a reason to be skeptical. Further, in recent months and years, we've learned the peer review system is completely broken.
Until it's been reproduced and validated by a few outside - and hopefully disinterested - third parties, we shouldn't accept it as fact but as a theory that could be replaced.
Peer review has never been a designation of fact, just "you must be this tall to enter the fighting pits."
Confusing peer review with fact is a public-perception problem, and a lot of that comes down to various antiscience groups whose work isn't even peer reviewed, let alone tested and challenged.
In computer science, the case where the easier thing to do is to replicate something does happen sometimes.
For example if you create a natural language parser that you claim is more accurate than another, you may need to replicate the other in order to compare accuracies. Sometimes you can just take the authors' reported accuracy, but often it's not that easy (maybe they just tested on English and you're interested in performance in other languages, maybe you want a different metric, maybe you want to compare on some standard set of features, etc.)
This all seems to be 'as it should be.'Science is after all a human endeavor with all the frailties that implies. Errors and fraud are to be expected. What matters is that the process is reasonably efficient at catching such things.
And the push for open data repositories and open science to increase accountability has to be a good thing.
Many of those journals charge very high prices, 4 figures per year IIRC. Perhaps they could earn some of that by verifying and vouching for the integrity of what they sell.
I think those journals charge high prices for the same reason that niche software vendors charge high prices. Namely, the product takes a goodly number of highly educated (and thus highly compensated) people to produce, and they don't have that many customers. So if the business is going to work at all, the prices have to be high. Furthermore, they are taking steps to verify the integrity of what they sell. And of course, the more steps they take, the more the journals are going to cost...
I don't know about that industry, but generally prices don't depend on the sellers' costs. Prices are set to maximize profit, that is to maximize (volume * net profit). Sometimes that's a huge markup (e.g., the $5 soda at the movie theater), sometimes that works out to a loss (e.g., the 2-year-old smartphone on clearance). There are many complexities of course, such as loss-leaders, regulated markets, public goods, exploitative pricing, etc.
I've read that the journals realized that they had a captive audience, i.e., that college libraries had no choice but to subscribe in order to support their faculty, and the publishers raised prices dramatically. There was a backlash from some schools at the time.
These days, scientific publishers are bundling everything together into one package, commonly known as the "Big Deal". Institutions pay a single fee for access to hundreds or thousands of journals.
"I don't know about that industry, but generally prices don't depend on the sellers' costs." What you're saying is true only if the revenue at the optimal price is higher than the seller's costs. If the revenue at optimal cost is lower than the seller's costs, then generally the product won't be produced. Therefore if the product is still being produced, the seller's costs put a lower bound on the revenue. If the volume is fixed, this means a lower bound on the price. If the volume is small, this lower bound can be high.
You are aware that 98% of the highly educated people (i.e. the authors and the reviewers of papers) are not compensated at all? Sure, they're paid to do the research by their employer, but they never see a dime from the journal.
I am aware of that---I've written and reviewed several peer-reviewed papers. Maybe many of the journals are making money hand over fist. But I've become suspicious of arguments of the form "that seems like a high price, therefore they must have huge margins". Sometimes the price is high because the market is small.
Is there any chance that there will be some sort of google-books-style centralization of experimental datasets from researchers which will then be subject to scrutiny from machine learning models trained to suss out faked data, at which point we'll uncover lots of shenanigans from our scientific past?
> It also includes moving the goal posts: that is, mining the data for results first, and then writing the paper as if the experiment had been an attempt to find just those effects. “You have exploratory findings, and you’re pitching them as ‘I knew this all along,’ as confirmatory,” Dr. Nosek said.
Why is this a problem? If the experiment's design is not in conflict with the new findings, why complain?
Let's say I flip a million different fair coins 20 times each. Then I analyze my findings and see that coin number 54 of them was heads all 20 times. I present my results to my peers saying, "Coin #54 is defective! The chances of this happening by chance are on in a million!" But I'd expect one of them to be heads every time flipping that many coins. It just happened to be coin #54.
The problem is we're not matching up our statistical analysis to our testing when we look for findings then naively test hypotheses based on those very findings.
Two-up ("Come in, spinner!") is a coin-flipping gambling game where people bet on whether a punter will come up double-heads or double-tails. The house makes it's money when neither come up five times in a row (with some variations).
Would using Bayesian statistics fix that? If the prior for a coin being defective to produce only heads is more than one in a million, then 54 is probably defective. If it's less than that, then your analysis doesn't conclude it's defective anyways.
Bayes can be hacked by ommitting info, but not by things like this, I believe.
It's not a bayesian versus NHST thing, it's just about doing the right test. You can do this kind of thing easily enough in a NHST kind of way. But yeah I think doing it in a bayesian manner is much easier to interpret and explain, and so requires less training, which is a HUGE boon for the crisis at hand.
You can do the test with NHST only if you know what the researcher had in mind when experimenting. That can lead to absurd results at times, such as your example. With Bayes, as long as no lies are told, your inference doesn't depend on the researcher's private thoughts.
You only need to know how the experiment was performed, same as you do for bayes. If they present the whole dataset, nothing is changed. If they cherrypick and only show the data for the coin that happened to be all heads saying that was their whole experiment, no amount of stats will help you, bayesian or otherwise.
No, if the coins are independent, the Bayesian is not fooled even when seeing only that coin. The argument was in my post above.
The inference in Bayes only depends on the data, and the flips of other coins doesn't make a difference if independent. Frequentist testing can depend on things like stopping rules and hypothesis tested, which aren't correlated to the actual truth and therefore should have zero effect on inference.
They can cherry pick and only show some flips of that coin, but then they really need to be outright lying or you'll ask why only some flips were reported.
imh's explanation is good, but it's also worth considering a very common example of this. Consider two groups of people where one group engages in some activity and another does not. Measure a dozen things about those people. Then report only on the one that positively (or negatively) correlates with group membership as if it was the only thing you measured.
This generates an unrealistically high statistical significance (unrealististically low p-value) because the question should not be, "What are the odds these two things are correlated by chance" but "What are the odds that any of of these things is correlated by chance with this other thing?"
The significance of the result is far below the reported p-value from an analysis that assumes only one thing was investigated. You frequently see incorrect p-values of this kind even when the researchers acknowledge they tested for multiple things.
Because it conflates fit and prediction. If I take a graph of the DJIA over the 20th century then fit a polynomial to it it's intellectually dishonest to turn around and claim that I have a predictive model of the stock market. I have a fitted function, nothing more.
This is similar to the sharpshooter fallacy. If I shoot at the side of the barn then draw a bullseye wherever I hit I can pretend to be a good shot.
Because if one takes a large dataset with many variables and starts mining for correlations it's roughly equal to trying to get a random number generator to produce numbers in some specific pattern. More numbers, the likelier the result.
This is the reason experiment repeatability should be considered of crirical importance especially for 'gooey' disciplines like medical and psychological research where there is little more than statistics to go with. Was the experiment just mininf random noise for correlation - 'simple', just repeat the experiment. If results can't be repeated it's more likely the original authors just botched the experiment, massaged their data to look how they wanted or just got lucky with a random number sequence.
Recently the depth of negative influence, corrupt findings, blind faith and religious-style zeal within the scientific community and pseudo scientific communities (like Reddit) has become absurd and embarrassing.
Science is a method, not a conclusion. I hope it becomes 'open', and fast.