They had to largely use alpha fold for the data part of the transformer scaling laws so not quite a bitter lesson, but still interesting.