Like ASIC Botcoin miners did. There are parallels here in how it might just pan out.

Interesting point. That said, the AI model space is rapidly evolving, while bitcoin's hashing problem is static. This makes it significantly more risky to make a large capital investment in dedicated HW when it's unclear if it will be able to run the next big model architecture. For instance, if this had been built + released a year ago, before SOTA models used MoE , then it would rapidly have become obselete.

Outside of hardware/implementation optimizations, and position embedding choice - has the SOTA transformer architecture evolved that much?

Llama-2 code appears to be about the same as gpt-2.

You can look at https://github.com/ggerganov/llama.cpp/blob/master/llama.cpp... for examples of the different layers in a number of different models, and further down in the code for their implementations. tldr, yes they are very similar. I can see lots of value in something that can just run these models. Even if you just supported llama2 there are tons of options available.

Oh man, all those years back I made a choice between antminer and butterfly labs. I backed the wrong horse.

BFL mined with customer hardware and basically didn't ship units to customers until there was no profit in running one.

Crypto ASICs are a super weird edge case IMHO in chips, strictly speaking it's not rational to sell them if they are very profitable. It only makes sense if the customer has a different risk profile than you; or the customer can somehow get power more cheaply than you; or you have some kind of scam going on; or you couldn't get capital except by presales and are unusually honest.

Note that an additional profit-making option for crypto ASIC producers is to secretly over-produce and compete with your customers and you are unlikely to get caught doing this.

That isn't correct. If the ASIc manufacturer produce them for X can use them directly and it only costs Y to operate them for Z profit. Then you can price them as Z-Y-profit_margin > X. The lower the operating cost, the higher your profit margin per chip if you sell it. Selling the chip might have a 50% profit margin and mining has a 10% profit margin. If you wanted to get into mining you would build a holding company owning both types of companies.