I don't think that's a great analogy.

LoRAs tend to be adapters bolted onto to systems by people other than the system designers, and they are low rank factorizations.

There is nothing low rank or adapter here.

Kind-of. You could theoretically use LoRA for this, in fact, but it probably wouldn't have enough capacity to make it a proper substitute of the attention mechanism. Instead a full MLP is trained as input chunks get processed.