What about reproducibility of the compiler binaries? I got the impression that model training itself isn't deterministic across training hardware? Since the model is embedded in the binaries, then the binaries aren't really reproducible if the model isn't. How big is the data used to train the model? How costly is it to train the model?
> The TensorFlow model is embedded with XLA AOT, which converts the model into executable code.
Taking the TF model to executable code should be deterministic.
Generating the TF model might not be (depends on implementation and hardware).
But it's unclear from this if this is a problem in practice - it's not uncommon for non-deterministic models to end up producing the same output because you perform thresholding/quantizing or some analogous process to convert it into a classification style output.
I.E. here, you are generating operations, which you can setup as a classification problem: Given this input and this history what is the next operation.
And of course you can always got back to your saved model and generate the code from that.
While the exact scores for the next operation might be non-deterministic you always end up with the highest scoring one being the same.
