"""
The Bayesian-frequentist argument, unlike
most philosophical disputes, has immediate
practical consequences. Consider that after
a 7-year trial on human subjects, a research
team announces that drug A has proved bet-
ter than drug B at the 0.05 signifi cance level.
Asked why the trial took so long, the team
leader replies “That was the first time the
results reached the 0.05 level.” Food and Drug
Administration (FDA) regulators reject the
team’s submission, on the frequentist grounds
that interim tests of the data, by taking repeated
0.05 chances, could raise the false alarm rate
to (say) 15% from the claimed 5%.
A Bayesian FDA regulator would be more
forgiving. Starting from a given prior distri-
bution, the Bayesian posterior probability of
drug A’s superiority depends only on its fi nal
evaluation, not whether there might have
been earlier decisions.
"""
Is that right? At each next trial Bayesians should feed the probability from the previous one as prior. Assuming that the first two trials did not bring the required results - then the prior to the third one should be rather small.
It's wrong. Not because the stated probability at the point you stop the experiment is wrong, but because the (stupid) rule is that we'll approve the drug if the probability that the drug is better than the alternative is above some arbitrary threshold value. If you run experiments a fixed number of trials, you'll get a variety of conclusions with different strengths. If you stop as soon as you are in the "barely passing" zone, you'll get a lower number of failing result, a higher number of barely passing results, and none at all that do better than barely passing.
The way I read this, the first two trials did not bring enough evidence towards the drug working, but they did provide some evidence. In the end, all three experiments taken together provide pretty massive evidence that the drug actually works, way below 0.05.
Of course, this all comes crashing down if the first two experiments happen to provide contrary evidence (that is, evidence the drug does not work). This would cancel out the results of the final trial somewhat, and not taking this into account is clearly cheating by publication bias.
Is that right? At each next trial Bayesians should feed the probability from the previous one as prior. Assuming that the first two trials did not bring the required results - then the prior to the third one should be rather small.