Model Stats - Architecture: LLAMA-like model with multi-query attention - Object...

		notsahil on Sept 4, 2023 \| parent \| context \| favorite \| on: Refact Code LLM: 1.6B LLM for code that reaches 32... Model Stats - Architecture: LLAMA-like model with multi-query attention - Objectives Fill-in-the-Middle, Chat - Tokens context: 4096 - Pretraining tokens: 1.2T - Finetuning tokens: 40B - Precision: bfloat16 - GPUs 64 NVidia A5000 - Training time 28 days