There is a huge amount of vendor financing available for this exact reason.

Hetzner presumably have been making use of this for years already.

It's far more likely they simply began raising prices and realized the market would tolerate it.

It's not better quality: 59.3% vs 59.4% fp16 on AIME 25

0.1% is within margin of error. Depending on the performance boost, it might be worthwhile taking a minuscule quality hit.

I think it very much is worth it!

But the point was that quality didn't magically increase.

> Venture capitalists & private investors are sucking all of the possible growth and future upside from these companies and then dumping them on retail investors when there's nothing left.

A lot of the money that is deployed by VCs comes from pension funds and asset managers that ultimately manage money for the average Joe.

Is there any evidence of what is the share/volume of such assets involved?

I haven't read TFA yet but a common technique is speculative decoding where a fast draft model will generate X tokens, which are then verified by the larger target model. The target model may accept some Y <= X tokens but the speedup comes from the fact that this can be done in parallel as a prefill operation due to the nature of transformers.

So let's say a draft model generates 5 tokens, all 5 of these can be verified in parallel with a single forward pass of the target model. The target model may only accept the first 4 tokens (or whatever) but as long as the 5 forward passes of the draft model + 1 prefill of the target model is faster than 4 forward passes of the target, you will have a speedup while maintaining the exact output distribution as the target.

> They paid a billion dollars for a vibe coded mess just for the opportunity to associate themselves with the hype.

Lol no they didn't. It wasn't even an acquihire. They just hired Peter.

Maybe they are paying him incredibly well, but not a billion dollars well.

> It's not any company, its Meta and the channels they administrate come with a set of responsibilities and principles

Sorry, which laws stipulate these special responsibilities and principles?

> or if the model might actually have emitted the formatting tokens that indicate a user message.

These tokens are almost universally used as stop tokens which causes generation to stop and return control to the user.

If you didn't do this, the model would happily continue generating user + assistant pairs w/o any human input.

This does not solve the problem at all, it's just another bandaid that hopefully reduces the likelihood.

Yes, it is written for a specific audience.

That is not a reason for snark.

As other commenters have noted, it’s well written.

> LLMs are inherently non-deterministic.

This isn't true, and certainly not inherently so.

Changes to input leading to changes in output does not violate determinism.

> This isn't true

From what I understand, in practice it often is true[1]:

Matrix multiplication should be “independent” along every element in the batch — neither the other elements in the batch nor how large the batch is should affect the computation results of a specific element in the batch. However, as we can observe empirically, this isn’t true.

In other words, the primary reason nearly all LLM inference endpoints are nondeterministic is that the load (and thus batch-size) nondeterministically varies! This nondeterminism is not unique to GPUs — LLM inference endpoints served from CPUs or TPUs will also have this source of nondeterminism.

[1]: https://thinkingmachines.ai/blog/defeating-nondeterminism-in...

Yes, lots of things can create indeterminism. But nothing is inherent.

Quoting:

"But why aren’t LLM inference engines deterministic? One common hypothesis is that some combination of floating-point non-associativity and concurrent execution leads to nondeterminism based on which concurrent core finishes first."

From https://thinkingmachines.ai/blog/defeating-nondeterminism-in...

Yes, lots of things can create indeterminism. But nothing is inherent.