This is huge, was just checking today on what would take someone to get a model similar to Llama, since Meta did not share Training code or Dataset.. Looks like they have figured out how to make the dataset ,Main Problem here is pre-processing them. Second step is to make the code to train model and final one do it cheaply.
Yeah. They will use Frontier at Oak Ridge, also known as the most powerful supercomputing system in the world. Maybe it can run some expensive LLM training for once rather than it's typical diet of physics simulations and quadratic gene-gene interaction models :)
