I’m sorry that I came across so different from how I meant to. I was intending to evoke a tone more similar to one I had recently here[0]. That is my fault and I should work on my communication.
> I admit that I don't know what an embedding is but I know it's something that is created by a neural network's output layer (or the post-processing of the output layer).
The embedding layer comprises the first layers of a transformer model, definitely not the “output layer”. It is typically trivially easy to isolate the output of the embedding layer because it generates the “actual” input to the LLM. The LLM has a certain number of input parameters (typically 784) and some length of text usually won’t fit precisely into 784 parameters (it will be longer or shorter than that). All the embedding layer does is map arbitrary length text to a single 784-dimension vector, which actually can be passed to the input of the transformer.
Ever since the paper “Attention is all you need”, the embedding layers have always been the first layers of a transformer to act on the text input. Most of the output is also sent back to the embedder, but this is only so that it can be used as input again to the “attention” layers.
> I admit that I don't know what an embedding is but I know it's something that is created by a neural network's output layer (or the post-processing of the output layer).
The embedding layer comprises the first layers of a transformer model, definitely not the “output layer”. It is typically trivially easy to isolate the output of the embedding layer because it generates the “actual” input to the LLM. The LLM has a certain number of input parameters (typically 784) and some length of text usually won’t fit precisely into 784 parameters (it will be longer or shorter than that). All the embedding layer does is map arbitrary length text to a single 784-dimension vector, which actually can be passed to the input of the transformer.
Ever since the paper “Attention is all you need”, the embedding layers have always been the first layers of a transformer to act on the text input. Most of the output is also sent back to the embedder, but this is only so that it can be used as input again to the “attention” layers.
0: https://news.ycombinator.com/item?id=40878317