> the best we can achieve is to stop early, as soon as the trial has a clear outcome
That must be really hard: if you wait for 95% confidence, you are selecting for 5% noise. If you repeatedly re-measure for 95% confidence, you strongly select for random noise.
Not many medical advancements provide such an anomalously strong signal (98% survival versus 30% survival).
> If you repeatedly re-measure for 95% confidence, you strongly select for random noise.
You can't use standard methods for early stopping - as you rightly point out, you get gibberish if you naively keep peeking at a growing data set. Instead, you have to use statistical methods that explicitly adjust for the repeated sampling in early stopping trials. This does make early stopping more complicated to analyse than a trial with a pre-determined duration.
That must be really hard: if you wait for 95% confidence, you are selecting for 5% noise. If you repeatedly re-measure for 95% confidence, you strongly select for random noise.
Not many medical advancements provide such an anomalously strong signal (98% survival versus 30% survival).