One advantage of PRNG assignment is that you can reproduce your assignment procedure whereas this is impossible with true RNG. If you use true RNG how could you ever prove to anyone that that was what you actually did?
You could say how you produced the random numbers and which random numbers came out. What you can't do however, is then prove that you actually did that and only that and not for example re-rolled the dice till the results looked right to you.
But you have the same issue with a PRNG, just boiled down to the seed value. Disprove that I didn't chose seed 42 by random, you can't.
The only way to protect against this is registering the study with a third party beforehand and letting them dice out the numbers as soon as the hashed data is there.
By registering your plan beforehand, someone can check if you altered it to cherrypick or p-hack your results after the fact. By letting them chose the random numbers you protect against you rerolling the numbers when inconvenient. By letting them only see the hashed data you protect against them rerolling/altering based on their interest and against you swapping out data point indicies after the fact, since they can rehash the actual values and check if it matches their selection.
For many experimental designs this could be overkill, but registering studies is becoming more common in medical studies. I doubt they go as far as I described with the random numbers.
Not so. Assuming I don't need a whole bunch of runs in the final paper I can use the zero seed for the published work. I don't think anyone will contest that.
We're talking about randomizing experiment participants into treatment and control group... What you're saying is equivalent to "I can use the same order of assignments for every experiment I do"...
I thought we were speaking more generally. In the specific case of assigning experimental subjects to groups at the time the experiment is performed you'd want a "no tricks up my sleeve" number for the key such as a hash of the date string in ISO standard format (ie yyyy-mm-dd). Your RNG is then AES_ECB( key=sha3(date_string), plaintext=serial_number ) using the serial number of the participant.
If you need to use a rejection method to achieve a uniform distribution you can do so via plaintext=( ( serial_number << 32 ) | sample_counter ).
By adhering to such a scheme it becomes extremely difficult for anyone to reasonably accuse you of underhanded tricks via RNG manipulation.
I thought where you were going with his was "that realized the best way to dispose of their nuclear waste was to dump it in the deep past." I’d read that novel.
reply