In a previous generation, the enabler of all our computer tech innovation was the incredible pace of compute growth due to Moore's Law, which was also "top-down" from very well-funded companies since designing and building cutting edge chips was (and still is) very, very expensive. The hype was insane, and decisions about what chip features to build were made largely on the basis of existing vendor relationships. Those companies benefited, but so did the rest of us. History rhymes.
Should probably change this to "was appearance of incredible pace of compute growth due to Moore's Law," because even my basic CS classes from 15 years ago were teaching that it was drastically slowing down, and isn't really a "law" more than an observational trend that lasted a few decades. There are limits to how small you can make transistors and we're not too far from it, at least not what would continue to yield the results of that curve.
The corollary to Moores law, that computers get twice as fast every 18 months, died by 2010. People who didn't live through the 80s, 90s and early 00s, where you'd get a computer ten times as fast every 5 years, can't imagine what it was like back then.
Today the only way to scale compute is to throw more power at it or settle for the 5% per year real single core performance improvement.
- Running up single-core performance through increasingly sophisticated core design and clock speed (which is now at the 5% per year point mentioned)
- Going wider by throwing more SMT, more cores, and larger caches at the problem.
Assuming here that x86 was the last major architecture that was going for high single-thread performance at all costs, the first phase lasted us a good 30 years-- from the 4004 to the flameout of Netburst.
We could consider the second phase starting when they started delivering the P4 with Hyperthreading, and its true-dual-core predecessors shortly thereafter, so we're now about 20 years into that era.
The difference is once you bought one of those chips you could do your own innovation on top of it (i.e., with software) without further interference from those well-funded companies. You can't do that with GPT et al. because of the subscription model.
You can do some, but many of them have license restrictions that prevent you from using them in certain ways. I can buy an Intel chip and deliberately use it to do things that hurt Intel's business (e.g., start a competing company). The big AI companies are trying very hard to make that kind of thing impossible by imposing constraints on the allowed uses of their models.
Eh, if this is true then IBM and Intel would still be the kings of the hill. Plenty of companies came from the bottom up out of nothing during the 90s and 2000s to build multi-billion dollar companies that are still dominate the market today. Many of those companies struggled for investment and grew over a long timeframe.
The argument is something like that is not really possible anymore given the absurd upfront investments we're seeing existing AI companies need in order to further their offerings.
Anthropic has existed for a grand total of 4 years.
But yes, there was a window of opportunity when it was possible to do cutting-edge work without billions of investment. That window of opportunity is now past, at least for LLMs. Many new technologies follow a similar pattern.
You completly forgot about the invention of the home computer. If we would have all been loging into some mainframe computer using a home terminal your assessment would be correct.
