> The tokenizer covers the entire dataset. Well, this is only trivially true. Yo...

		astrange 34 days ago \| parent \| context \| favorite \| on: What GPT-OSS leaks about OpenAI's training data > The tokenizer covers the entire dataset. Well, this is only trivially true. You can feed binary data to the LLM and it probably has tokens that only cover single bytes of that.