> CI runs use loop devices on shared ephemeral VMs (one VM per filesystem): compare shapes and ratios, not absolute MB/s. Each job records a host-calibration anchor — see the table.
I think if you're not using baremetal for such tests, it's likely that the results are simply not comparable at all? What if another tenant is also using the disk?
It's a fair point but it's also possible the person running the tests has a dedicated test hypervisor for this , so that different configurations of filesystems and VMs can be created and destroyed quickly in an automated manner.
If it's something as simple as a KVM hypervisor that only runs 1 test VM at a time (with no other load from anything else other than the basic systemd daemons, ssh daemon etc running on the hypervisor), the results could be very close to bare metal.
I can see it being very time consuming and annoying to do repeated manual bare metal OS installs and new partitioning/filesystem creation for such a large variety of tests.
The author does also say that performance isn't really the main thing but rather, data integrity:
> I can see it being very time consuming and annoying to do repeated manual bare metal OS installs.
Well don't do that then. There's lots of other options. Probably the simplest is a single bare metal install on a simple filesystem on one device. run the filesystems under test on other storage dedicated to testing.
You could also boot into a network install and use local storage exclusively for testing.
Yes exactly. The issueThe epherrality of the VMs isn't an issue, it's the _shared_ part that's the concern here. Going by the fact that the kernel is listed as "kernel 7.0.0-1012-azure" I feel like it's a fair risk that there may have been noisy neighbours.
Obviously they could and probably were noisy neighbours. I'm not trying to hide that fact.
Initial step for every benchmark is test of underlying device to at least reject completely unlucky cases.
Also after almost 600 runs average is probably more or less correct...
Also take a look at tests on real hardware. There are not many of them but there are some. I pointed to them in my first answer.
For the main linked benchmark there are no dedicated test hypervisors. It's all based on GH runners with all the limitations and quirks that came with it.
In this case, given that the author's own disclaimer (above) already disclaims the numeric readings, I'm not sure how it's possible to make any inference on "shapes and ratios" derived from the numeric readings.
@farlight assuming that you're the creator do you think you'd be able to rework the HTML/CSS? I'm sure you've got good data but speaking on behalf of my eyeballs, the results page is... hard to read!
> 2G of random garbage is written directly onto one member device (behind the filesystem's back, offset 1G — python injector; uutils dd mis-seeks on dm devices), caches dropped, then a full scrub: btrfs scrub -B, zpool scrub + wait, bcachefs scrub, md/lvm sync-action 'check' (which can only COUNT mismatches — no checksums to know which copy is right).
I'm not sure that nuking 2G of the underlying block device is a recoverable error on any filesystem that I'm aware of? Can you confirm if any ofthe filesystems really came out of the other side in a usable state after scrubbing?
-----
> Trivial-op p99, idle (ms) # A trivial operation — one 4k write + fsync every 200ms (like a shell appending history or an editor updating its swap file) — run alone for 10s. p99 of the fsync completion
In fact, if it's OK for me to ask, are any of the metrics tht you used standard industry metrics? It looks like several of the tests are bypassing the kernel's page cache? -- which I worry may fall into the trap of "I modified the system to be unrepresentative of reality and then tested it".
----
> kernel 7.0.0-1012-azure
Can you confirm if you tested on a bare metal machine? were you the only tenant?
> I'm not sure that nuking 2G of the underlying block device is a recoverable error on any filesystem that I'm aware of?
ZFS and btrfs were designed from the start to handle this, by using checksums on every piece of (meta)data and redundancy to return the same data as was stored to the kernel, and rewrite the bad data.
I've tested my own machines running ZFS by random writes out of band from the filesystem/kernel and it has always found and fixed them.
I'd actually recommend some excellent books on the "philosophy" of data presentation: The first that comes to mind is "the visual display of quantitative information" by Edward Tufte seems to be freely available online, and the other one on my mind is "how charts lie" by Alberto Cairo (which doesn't seem to be freely accessible)
But if it helps, just some "initial gut feel observations" from me:
* It's definitely not possible to find issue with the the _sheer amount_ of results, but there's just far too much for a human to absorb, all presented at once
* Overall text size is quite small, and difficult to read
* The page doesn't make a strong statement of _what_ is under test: the first words are: "modern-fs-benchmark Multi-device CoW filesystems under workloads classic benchmarks skip" -- which defines the webpage in terms of what it is _not_, without stating what benchmarks are actually present.
* The first line of teh page contains run statistics that probably eithre want to b at the bottom, or just don't need to be in the webpage at all: "latest run 2026-09-18 18:50:45 UTC, kernel 7.0.0-1012-azure, 593 runs recorded · 145 trend points shown"
* A significant proportion of the free text is caveats. There's nothing wrong with being transparent about limitations, but they may be a sign that there might be alternative ways to present the data, or that the data may be flawed (depending on the caveat)
* Theres several categories that I think have been invented for the purpose of collation, but I don't think are defined on the page. I think "Overall Core" and "Core I/O" aren't explained, which means by definition it's impossible for a reader to understand the score table.
* And as we're all aware right now, current Claude models are currently struggling to write coherent English. There's several incoherent sentences on the page. It's a Claude issue.
Speaking as someone who consumes this, I appreciate how it's laid out. As a developer, we can often see at a glance where the bottleneck is if we have enough data laid out - IOW, data overload for you is me feeling like a kid in a candy store.
Sometimes there are ways to make things easier without dumbing them down, but way too many people conflate the two; I get nervous when non engineers say "I've studied this, it should be easy".
ah, yes that's obviously fine; I'm definitely coming fromt he perspective of "bored guy on a Saturday evening who happened to stumble across someone else's wall of charts on HN"
I think the Phoronix test suite does pretty well for the end user who wants something easy to digest; the exact choice of benchmarks is sometimes odd but the harness itself is quite good.
At some point I'd like to get our own automated pts runs going, since Michael is not consistent with what hardware he tests on and he hasn't been consistent with getting them out.
Thankyou for putting the last three months of my life into words <3
> and the explanations of the agent are convincing enough that surely, it knows better than you.
I feel this in my bones. I also get to watch the misalignment feedback loop close itself when the next agent sees that security rules aren't followed because of a hallucinated 20 line justification in a code comment, and then it decides that the project _is_ a demo and then confidently writes even more security holes into the codebase.
Then when you catch the issue, the agent pushes back against the fix because it would need a schema change and production DB migration.
hmm, assuming that this article is part written by claude and part human-written, can anyone help me find a rule of thumb for "how to know if the article is worth reading"?
Because on the one hand, the prose and the presentation is painful (narrating irrelevant points, nonlinear X-axes, ambiguous chart labels, etc etc),
But on the other hand, the result that I'm assuming the author means to communicate ("on these evals, generation quality seems fairly good") sounds worthwhile to share?
Because I really struggle with this question at the moment. Am I allowed to draw an adverse inference that "if the writeup presents irrelevant text side by side with the data, then this may be a sign that the author does not understand the task that they are attempting to write up"?
I wrote this blog post myself, with AI for proofreading (typos and grammar, but not style). There were a few singular sentences for which I had a writer's block, but not much besides that.
So, if there are irrelevant remarks, these are mine. :)
Charts are vibe-coded - but it took quite a bit of hand-holding to get something decent. And the logarithmic scale for model size is my conscious choice (against Claude's initial ideas).
IMO, whether or not an LLM was used in the writing process doesn't really matter and I think it's a bit annoying that articles are being dismissed out of hand because of that.
The line is "Is this an interesting and accurate article that concisely makes it's case".
LLMs love to burn paragraphs writing about nothing which is why it's generally poor writing. Humans can do the same thing if they are trying to make very little information feel more substantial.
I say, stop trying to determine if an LLM was used and start judging based on your subjective measure that you'd have used before LLMs became widespread.
(If it helps, I ask my own question of myself too -- I mostly don't write code by hand any more as I find that an LLM writes it faster and with less bugs -- Is that therefore proof that my time was never worth my paycheck? I hope not but at the same time I would actually be proud if I had got away with being an accidental charlatan/fraudster at my employer's expense during my entire career)
-----
Similarly, if what I said really is true, I would be implying that LLMs are charlatan/fraudster detectors (to some statistical level). And I refuse on principle to believe that that is actually the case.
My main problem — which I am sure being middle-aged compounds — is that I struggle to retain information that an LLM has written or produced. I cannot explain why but it is a consistent problem.
In a week’s time I might remember the substance of your comment and some of its shape as a matter of course. Nothing LLM-written that I see today will stick, no matter how curated it was.
This is a complaint I hear others voice and one I have myself. I think it comes down to the text being poorly written. Sure, LLMs are good at a lot of the surface indicators of good writing: they have a wide vocabulary, they use grammatical sentences, they break up the text into paragraphs, sections, and lists. But they're terrible at organizing the text and marshalling a concept to get it across to a specific audience. The section and paragraph breaks are meaningless, the metaphors are unenlightening, the rhythm is exhaustingly uniform. Pre-LLM, you'd find this kind of text in marketing materials, corporate PR, and heavily padded sophomore essays.
Possibly an inevitability, since LLM text is specifically generated to be low entropy. It’s smooth, optimized for minimal perplexity.
And if you don’t find any of the word choices a little bit perplexing, your brain can just gloss over it. There is no meaning here. It’s like smalltalk.
Interesting — do you happen to have any suggestions for something I could read about this?
It really bugs me, this aspect of LLMs, and yet I find myself sounding like a lunatic when I describe it, until others chime in and say they have the same experience.
Perhaps it's just a bias? You are already negatively biased against LLM writing so you disregard stuff you read when you suspect it's an LLM. This could also be a selection bias. It may be that you generally struggled to retain information but you are more aware of it when LLMs are involved.
I don't think so, no, because it extends to LLM-generated text I want to recall.
I use LLMs to generate starter/tutorial material. I may hate the way Claude writes but I absolutely don't hate the way Gemma 4 writes. But I have to continuously consult it in a way that I do not with human-written text, which gets its message across in a more persistent way I find less troubling.
(This non-memorability extends to AI images and video.)
You are right that there are some confounding factors in my life but while I was worried about middle age affecting recall, I find actually I am still remembering stuff humans wrote pretty much fine.
ETA: there is one thing that I have noticed that does affect recall that is specific to LLMs: watching text roll out word-by-word in LLM chat, I think, damages recall. It's engaging the wrong kind of memory and focus.
So I now let it generate and I'm trying to find decent ways to format it e.g. as PDF, to give it the best chance.
This could be generational; there were many studies twenty years ago that suggest that people a decade or so younger than me who grew up with full colour books and magazines and multimedia can read less-linear text layouts more comfortably, for example, so it's not out of the question that there's something generational going on here too.
Sure, but if you're going to publish it, at least run it through an edit prompt and tell it to remove clickbait "Its not X, its Y" rubbish. Like literally calling examples clickbait in the prompt has given me better results. Interestingly, I have a lot less trouble with the first draft with Qwen than with Opus.
Yes, LLMs make it a lot easier to produce a lot more garbage. That's not an exception to my point. If something is well written then it doesn't waste the reader's time.
Just ask your freaking agent to read it for you and extract the information. That's what it's all about. Why would I be reading these articles other than information?
Claude, read Love in the Time of Cholera for me and summarize the information contained. You are an expert book reader and understander. Make no mistakes.
I normally don't appreciate snarky comments on HN, me being but a simple curmudgeon wanting pure information and the occasional opinion. but this one actually makes a good point, even if it's not stated explicitly. Reading this book is a wonderful experience because of the way it's written. Reading the plot summary (from an AI or otherwise) yields nothing but the dry husk of that experience.
Don't know why you're getting faded out, because having worked on exactly this engine, software mitigations were put in place. That's exactly your `anomaly detection system`
It was a belt-and-braces fix, because the primary fix was of course to manufacture the engines correctly in the first place.
The "lots of temperature sensors" thing is actually easier said than done; jet engines already have lots of temperature sensors, and they will turn the engine off if a huge overtemperature is detected, but it's not a free win to add more of them. It increases the risk of a faulty sensor shutting down a perfectly good engine in flight, which from a safety-critical perspective is a borderline disaster condition all by itself.
(technically you usually cross-correlate temperature sensor readings with pressure sensor readings to prevent sensor faults turning engines off, but still, extra sensors isn't always a free win.)
It's a valid shift to move onto actually trying to read the article critically (which I don't mean in an insulting way -- If you assume a writing has something worthwhile to tell you, reading it critically is how you learn the worthwhile thing)
In this article, _I_ get unstuck right at the very first paragraph:
> Same driver, same track. The LLM is the star. Seventeen leading models driven round the identical 28-realworld task lap — one harness, same verbatim prompts, deterministic grading — and the results go on the board.
It jsut doesn't make much sense to me. At best, I think it can be glossed as... "I made an arbitrary benchmark which I'm not going to explain, and I plotted the results."
------
Getting my own opinions out, this is blatant slop. It claims to be "deterministic grading", but then almost the _entire_ webpage is editorialization. Examples:
* "If you only run one model, run glm-5.3"
* "opus-5 posts the best rubric on the default panel"
* "deepseek-v4-pro is nominally cheaper still at $0.0029 [...] treat it as a batch-only option."
Yeah, I'm with you on this, I think this is just what fable/opus-5 slop looks like now...
- "A frozen verify.py accepts the claim" (what does it mean to freeze a python script?)
- "which trains the recipe eight times on fixed seeds it can't touch" (what does it mean to not be able to touch a seed)
- "One other detail is that we gave an estimation of the speedrun noise in program.md that was slightly too large. 62 out of ~100 runs measured it themselves instead of trusting our number" (What does it mean for a "run" to "distrust" a noise measurement)
- "One important disclaimer is that our benchmark has a lot of variance" (Actually this one makes sense, but congratulations for burying the lede that your entire article is bogus.)
- "Almost every model finds the same winning ideas. What separates the best traces is what an experiment leaves behind" (This implies the the graph would show every model finding a plateau in whatever metric the experiment is measuring, but I don't see every model scoring the same in the graph)
Those damn invented terms are so annoying. We can't see their reasoning anymore unfortunately but I wonder if it's terminology they invent for themselves early in their reasoning token burn and by the time they're outputting a final message they've used that term talking themselves so many hundreds of times that they start assuming the meaning of the term is common knowledge
Yes this is an extremely well known result for exactly the reason you guessed. It's not just abcktracking, asking an LLM to present a conclusion and then justify is also an excellent way to provoke hallucination as the model con concts "any justification that plausibly justifies the words it's already said".
This is the actual reason why openai _invented_ reasoning models, to give them time/space to work out a solution, rather than having to magic a correct solution out of thin air from token 1.
It's less important now that all models do reasoning, but it's still almost always better to make the output come out last rather than first.
Is it too cynical to read the first quote as "when openai catch up, we will restore fable 5 to subscription plans"?
In other words, they're saying "We know have a monopoloy and we will take all of your money until someone offers a cheaper competing product"?
And "We intend to do this as quickly as we can" read as a non-promise then as much as it does now?
---
Well, I am assuming that this is all based on the fact that Anthropic have enough compute capacity to serve fable 5 now and in the future, and they're "only" limiting it in the coming week to get richer quicker. Since compute hardware is presumably relatively un-fungible, I'm assuming Anthropic isn't offering a week of fable-5 for everyone on rented hardware that they're paying exporbitant fees for?
In other words, my cynical read is "if they can serve it under the terms of subscription plans for a week", then they could serve it under the same terms for a month?
The word "monopoly" is overused. They simply have the best model at the moment but no real monopoly-style moat or ability to crush their competition through monopoly tactics. I don't support their bait-and-switch, but if you aren't happy, don't use Anthropic. Also, as soon as someone comes up with a better model, transition won't be that painful for most users.
Late to this post, but my impression was that later models would be more efficient per task? Wouldn't they save compute released fable 5, maybe capping the effort, if it is actually a better model?
I'm totally feeling you here as well. Fable is definitely quicker to first output token on `claude.ai` (ie less internal reasoning tokens being generated), and given how much more expensive decode is than prefill, I'm sure that must pay for itself pretty nicely, on top of any architectural changes that they must have made since the opus 4 architecture was locked in.
See the link to Reddit in my comment below for more details, but the TLDR is that (as a programmer) even when you use Mythos ... Mythos will hand the work off to Opus!
Whether it happens to 5% or 50% (or even 0.5%) of users, I think everyone should feel extremely cynical when a company offers a huge freebie of a product ... and fails to mention that any % of their customers will actually get yesterday's model.
They did talk about it at length in the post, and most of the guardrails were already there in the first release. To be clear, it shouldn't have anything to do with the user. It's the prompt, and if it switches, it'll tell ya.
I mean for many of us this doesn’t even increase the cynicism. Ie it can’t get worse. These AI labs are already believed to be shifting behavior of these models at runtime so you never really know what you’re getting. Stability is not a thing with any provider, if you need that you have to run it locally.
I’m not offended by that because I’m already maximally offended lol.
I think if you're not using baremetal for such tests, it's likely that the results are simply not comparable at all? What if another tenant is also using the disk?
reply