You can set how much of your system ram your gpu gets in windows and linux

Useless for LLMs. Limited system memory bandwidth kills it. GPU will be mostly idle waiting for memory access. The H200 has 4.8 TB/s bandwidth. System memory is below 90 GB/s

If you have a use case for an h200 class gpu you are probably making enough money off of it to not care about $40k and the 900gb/s bandwidth mentioned in another comment would seem absolutely pedestrian in comparison. If these things were really that compelling of a performance per dollar ratio you wouldn’t be able to buy them. They’d be sold out because people would be filling datacenters with them.

M2 Ultra sits between a lower end Intel setup and an H200 for this purpose.

I reject this all-or-nothing black and white childish framing of it.