I am new to Amdahl's law, but wouldn't a rearchitecture make it less relevant. For example if instead of growing an LLM that has more to do in parallel, seperate it into agents (maybe a bit like areas of the brain?). Is Amdahls law just a limit for the classic LLM architecture?
I don't think it can ultimately be escaped but the cited Vegh et al exactly proposes that, the bioinspiration, as a means to surpass those limitations.
However, in this article I contend that those limitations have posed little adversity in the field given the success of the latest models. As a result, it may be a bit premature to be concerned about it.
