Maybe the CRC32 implementations in the smasher suite just aren't that fast?
[1] claims 15 GB/s for the slowest implementation (Chromium) they compared (all vectorized).
> The 32 bit hash of CRC32 is too low for file checksums. xxhash is definitely an improvement over CRC32.
Why? What kind of error rate do you expect, and what kind of reliability do you want to achieve? Assumptions that would lead to a >32bit checksum requirement seem outlandish to me.
From SMHasher test results quality of xxhash seems higher. It has less bias / higher uniformity that CRC.
What bothers me with probability calculations, is that they always assume perfect uniformity. I've never seen any estimates how bias affects collision probability and how to modify the probability formula to account for non-perfect uniformity of a hash function.
It doesn't matter, though. xxhash is better than crc32 for hashing keys in a hash table, but both of them are inappropriate for file checksums -- especially as part of a data archival/durability strategy.
It's not obvious to me that per-page checksums in an archive format for comic books are useful at all, but if you really wanted them for some reason then crc32 (fast, common, should detect bad RAM or a decoder bug) or sha256 (slower, common, should detect any change to the bitstream) seem like reasonable choices and xxhash/xxh3 seems like LARPing.
> both of them are inappropriate for file checksums
CRCs like CRC32 were born for this kind of work. CRCs detect corruption when transmitting/storing data. What do you mean when you say that it's inappropriate for file checksums? It's ideal for file checksums.
Uniformity isn’t directly important for error detection. CRC-32 has the nice property that it’s guaranteed to detect all burst errors up to 32 bits in size, while hashes do that with probability at best 2^−b of course. (But it’s valid to care about detecting larger errors with higher probability, yes.)
There’s a whole field’s worth of really cool stuff about error correction that I wish I knew a fraction of enough to give reading recommendations about, but my comment wasn’t that deep – it’s just that in hashes, you obviously care about distribution because that’s almost the entire point of non-cryptographic hashes, and in error correction you only care that x ≠ y implies f(x) ≠ f(y) with high probability, which is only directly related in the obvious way of making use of the output space (even though it’s probably indirectly related in some interesting subtler ways).
E.g. f(x) = concat(xxhash32(x), 0xf00) is just as good at error detection as xxhash32 but is a terrible hash, and, as mentioned, CRC-32 is infinitely better at detecting certain types of errors than any universal hash family.
This seems to make sense, but I need to read more about error correction to fully understand it. I was considering possibility that data could also contain patterns where error detection performs poorly due to bias, and I haven't seen how to include these estimates in probability calculations.
> I'm pretty sure xxhash is a straight upgrade compared to CRC32.
Unclear; performance should be pretty similar to CRC32 (depending on implementation), and since integrity checking can basically be done at RAM read speeds this should not matter either way.
> If you had enough paper and ink and the patience to go through it, you could take all the training data and manually step through and train the same model.
But you could make the exact same argument for a human mind? (could just simulate all those neural interactions with pen and paper)
The only way to get out of it is to basically admit magic (or some other metaphysical construct with a different name).
We do know that they are different, and that there are some systematic shortcomings in LLMs for now (e.g. no mechanism for online learning).
But we have no idea how many "essential" differences there are (if any!).
Dismissing LLMs as avenues toward intelligence just because they are simpler and easier to understand than our minds is a bit like looking at a modern phone from a 19th century point of view and dismissing the notion that it could be "just a Turing machine": Sure, the phone is infinitely more complex, but at its core those things are the same regardless.
I'm not so sure "a human mind" is the kind of newtonian clockwork thingiemabob you "could just simulate" within the same degree of complexity as the thing you're simulating, at least not without some sacrifices.
You absolutely have a point, I just don't see how this is functionally different from western/US policy, especially from the perspective of e.g. BRIC nations:
We have ample evidence that US intelligence siphons data from literally every meaningful company it can tap, is willing to share that data with partners abroad and uses such things without even public sanction against targets picked by the president (see Venezuela).
Sure, the US is still the devil you know, but if Americans want to claim the moral high ground then at least credible pretending is required, and under the current administration we wont even get that.
This is exactly why its so hard to compare though; government contracts, emission credits and direct subsidies are all quite different and weighting them is highly subjective.
While Americans very frequently complain that the Chinese state subsidizes various industries, I am astonished that they do not see any similarity with the fact that I never heard of any really big investment project in USA, e.g. the building of any new big factory or new company headquarters, that was done otherwise than after receiving very substantial tax reductions of various kinds from the local government of the place chosen for the project. In many parts of Europe those kinds of tax reductions would be illegal, being considered a form of state aid for a private company.
And yet virtually all European lawmakers get $ from governments threatening to cut jobs.
Many countries actively lose money for those jobs, Serbia is an example. They go to extreme lengths to underbid competition for stellantis factories and get a net negative impact.
If you can't survive without taxpayers paying the bills, just die ffs.
I think people generally vastly underestimate the negative effect of "nativism"/anti-immigration rethoric on desirable foreign workers. Even if those desirable foreigners are completely unaffected by actual policy.
I personally know at least two highly qualified STEM workers that went back to Europe after Brexit/Trump2 and from what I can tell this was at least 70% pure "spite" (instead of being affected by actual anti-foreign regulation or somesuch).
I used to live in Cambridge. Only one person I knew there was pro-Brexit, call him C. In the run-up to the referendum, I'd already told C I was looking to move out of the UK due to the entirely native political choices of the UK. The domestic political nonsense was necessarily only going to get less constrained by a policy decision (Brexit) specifically about getting less constrained.
One of my last memories of C was that a group of us were in a local pub discussing it, one said they were worried Brexit would make Cambridge smaller. C shouted "Good!".
Given my plans, obviously when C shouted "Good!" at the idea of Cambridge shrinking, I was shocked, took it somewhat personally. Then the referendum came, and I was both angry with him and far too busy, so I stopped talking to C entirely.
Other people report that C was very confused by this, did not understand at all.
I completely disagree with your base premise: I think EU leadership (national + EU) performed on an average level over the last three decades, and the EU as an insitution did even better than I would've expected given its limitations/constraints.
If you could pick policies with the unfair benefit of hindsight (while staying somewhat democratically acceptable), what would a "perfect" government have done differently in the last twenty years?
I actually don't think they performed very well. I just look at the EU GDP that's now half the size of the US compared to bigger than US 20 years ago, and at standard of living of middle working class, which has slowly been going downhill in the last 15-20 or so years for a lot of people(worst of all southern europe) and at a more accelerated rate in the last 3-4 years. All due to their decisions and policies and I hold them accountable for it.
Like you don't need the unfair power of hindsight to know that tying your energy independence to Russia (your military adversary and the reason NATO exists) was a bad decision back then, or that staying dependent on US military and tech was bad for sovereignty, or that pursuing unrealistic climate goals was a bad decision, or that opening your borders to millions of unvetted people from unstable regions with high crime and low education was bad, since many people have been saying all these were bad decisions 20 years ago but they were ignored because the gravy train was still running and the EU political elite never game much of a shit about what the peasants though anyway. And now that the gravy train has stopped and the piper has to be paid, our leadership class are trying to gaslight us and deflect the blame for their recklessness at best or just suppress voices of dissent at worst.
And even if we were to assume they performed super well, that doesn't mean I should now swallow tyrannical laws designed to suppress our freedoms while they give themselves exemptions, just because different people from a different era who are no longer in power made some good decisions 30 years ago under the same umbrella of the EU since the EU-EC of today as an org is a vastly different beast than the EU of 30 years ago. There wasn't even a common currency and central bank 30 years ago.
You are exactly not answering the question though.
It is easy to complain about energy dependence, struggles with immigration/integration and precarious national budgets, but the conditions for those weren't caused by recent decision in my view (instead, namely, lack of local oil/gas, unstable north africa/middle east and bad demographics with too many old people).
All those problems are really costly and difficult to solve. Sure, the EU could've tried a super-scale Messmer plan 20 years ago, and could then maybe rival current renewable power with nuclear output, but this would have been orders of magnitude more costly.
Hard cutting immigration, Japan style, would probably have led to comparable economic stagnation from insufficient workforce (see post 1990 Japan).
It is unclear to me if averting demographic change would've even been feasible at all, and even if it was, it would have come with a plethora of undesirable side-effects (e.g. insanely high youth unemployment).
I personally think that environmental policies are the absolute bare minimum.
If you want to prevent worst-case global climate effects (say, +4°C global average temperature rise within the century) then what we are doing right now is not even enough.
>You are exactly not answering the question though.
WHich question did I not answer?
>weren't caused by recent decision in my view
No, but they could admit it was a fuckup of the political class, and tell us how they're planning to fix it, instead of gaslighting us for complaining about it and doing more mistake that further damage our economy. We can't stop the music chair song from playing just yet. Just one more round I promise.
How about the recent EU Mercosur trade deal? They're fuckign EU farmers and makign us dependet on foreign food imports. This is a totally new strategic blunder, they can't keep blaming the past anymore.
>Hard cutting immigration, Japan style, would probably have led to comparable economic stagnation from insufficient workforce (see post 1990 Japan).
This is disproven and untrue but has been repeated so many time it became the poster child argument of pro-illegal immigration EU propaganda. That open borders will magically save our economy. It didn't. Case in point, Poland's economy has been growing like crazy so IT IS doable to grow your economy without importing illegal migration and associated issues.
Regarding Japan's economy, is not fucked due to not importing millions of uneducated third worlders to live on welfare like Europe did, it got fucked starting from 1985 when the US forced them to sign the Plaza Accord in order to counter their massive trade deficit US had with Japan(and other powerful export economies), which increased the value of the Yen making Japanese imports suddenly expensive and uncompetitive. Then to combat this, the Japan central bank fucked up themselves by printing money like crazy with ultra low interest rates which instead of stabilizing the exports, caused a massive speculative real estate and golf membership(yes that's true) bubble which imploded in 1990, dealing the final blow to their economic growth since they ran out of levers to pull other than suppressing their labor into a losing race to the bottom with other low wage manufacturing economies of Asia. I recommend you read more on the Plaza Accord, it's pretty eye opening.
Depressing economic situations is what leads people to stop having kids, not the other way around.
Ever since the 1980s, Germany and other European models already knew population trends were gonna be fucked in the future, all the way back then. Did they take any measures and do anything to help the European population have more kids and prevent this? No. But they have the nerve to blame us for not having kids in a stagnating economy with crazy real-estate prices then gaslight us that salvation lies in illegal immigration instead of fixing the local issues preventing the locals from having kids, then introduce speech control laws for anyone who criticizes this.
>It is unclear to me if averting demographic change would've even been feasible at all, and even if it was, it would have come with a plethora of undesirable side-effects (e.g. insanely high youth unemployment).
Good point. If you already have high youth unemployment, why do you need to import illegal migrants? Just to lower their bargaining power further?
>I personally think that environmental policies are the absolute bare minimum. If you want to prevent worst-case global climate effects (say, +4°C global average temperature rise within the century) then what we are doing right now is not even enough.
yeah, the environment is important, but all economic sacrifices EU has made to combat climate change, are being absorbed by US, CHina and India to grow their economies, so the planet is still getting more fucked like before except now we made ourselves poorer for it. Great success. And even Europeans will stop caring about the environment when they won't be able to have a job, or afford to pay rent or heating anymore.
The "what policies should a perfect government have enacted instead, with hindsight". But your last post adds a lot on that front.
> Depressing economic situations is what leads people to stop having kids, not the other way around.
Hard disagree on this. Just consider two poster children for continuous positive economical growth: Poland and Vietnam. Did that economic outlook effect positive population growth? Clearly no (for both cases, and it's not even close).
My personal view: Easy access to contraceptives (people want to fuck more than raise children), and realignment of economical incentives: Children are no longer the default retirement plan (nor are they needed by the parents themselves as cheap workforce).
Both of those factors are icky to counteract for a modern democracy.
> If you already have high youth unemployment, why do you need to import illegal migrants? Just to lower their bargaining power further?
No, because those did not occur synchronously. The imported, lower-skill workforce was most needed earlier (pre 2000s), with the big youth unemployment problems mostly occuring later (2000s and after).
Immigration rates are also much easier to control than local birth rate, and, most importantly, dont suffer from a two-decade lag.
Restricting foreign workers would have most certainly been a big economic hit (this is somewaht obvious; because foreign workers are mostly raised and educated for free), especially for countries like Germany, even with perfectly boosting population growth at the perfect times to compensate.
> Case in point, Poland's economy has been growing like crazy so IT IS doable to grow your economy without importing illegal migration and associated issues.
Poland has been playing catchup inside a huge, wealthy, low-barrier market. Would their growth have been comparable if wages had started at a German level in 1990? IMO clearly not.
> all economic sacrifices EU has made to combat climate change, are being absorbed by US, CHina and India to grow their economies, so the planet is still getting more fucked except now we made ourselves poorer for it
Two big problems with that view.
First: The planet "getting fucked" is not binary. If you want to limit warming to a somewhat reasonable degree (and the worst-case timelines are not reasonable), then action has to be taken at some point, and pretty obviously the biggest culprits need to get the ball rolling, or no one is ever gonna do anything.
Second: The EU has already fucked the planet much harder than any developing nation, China included, despite being not even half the population of India or China. You could make a case that the US is slightly worse, but thats completely irrelevant; the relative moral net-obligation on the EU to act is clearly pretty large already for a long time now, and a few outliers (US, oil states) being even worse does not absolve the Europeans from anything.
Lastly, those efforts to combat climate change clearly had already huge global effects. Or do you honestly think that massive Chinese buildout of solar/wind power would have happened without those technologies getting developed, refined and proven in Europe over the last decades?
I don't necessarily disagree with your outlook completely: I think a bunch of things could have been done better, especially refugee handling, and possibly immigration vs economy tradeoffs. But still: a lot of those decisions had to be made without hindsight, and I don't think expecting much better than what we got is reasonable.
You also have to consider that lots of those decisions were made in a different time (with different values/outlooks): It was much harder to let refugees become homeless, executed or starved when a lot of Europeans saw thair own past actions (=> colonialism) as a big driver for those crises (and I'd argue that this only really changed, somewhat justifiably, post Arab spring).
pick policies with the unfair benefit of hindsight (while staying somewhat democratically acceptable), what would a "perfect" government have done differently in the last twenty years
So, they'd have to be things that were democratically acceptable at the time.
For example: Germany in that period was never no way going to accept nuclear power. Their leadership regrets it in hindsight, but at the time, forcing it on the people would have been undemocratic.
> They're fuckign EU farmers and makign us dependet on foreign food imports.
20 years ago the biggest problem with the EU's farming system was massive overproduction.
Like, "newsworthy scandal" levels of overproduction.
> Poland's economy has been growing like crazy so IT IS doable to grow your economy without importing illegal migration and associated issues.
Poland mainly missed out on the downside of the global financial crisis, rather than being special otherwise. Few percent difference between Poland, Germany, Japan, Europe collectively, and the USA all around the same level.
> is not fucked due to not importing millions of uneducated third worlders to live on welfare like Europe did
Europe did not in fact import millions of uneducated third worlders to live on welfare. This would have been a self-evidently stupid thing to do, which is why that is not what happened.
Europe did take around two million asylum seekers in total, before the pandemic. Important thing about asylum: they get sent back as soon as their homes stop being warzones, or sooner if they're deemed to have been taking the piss. Right now there's about 4 million Ukrainians, who would probably count as "second world" given the etymology; do you want to count them as "uneducated"? I wouldn't. But then, I have Ukrainian neighbours.
Economic migrants, who are important for the economy, are a bigger group. Mixing up asylum seekers and economic migrants because they're both "migrants" is as much of an error as declaring that all Canadians and Mexicans are "Americans" because they're from the continent of America.
> Depressing economic situations is what leads people to stop having kids, not the other way around.
If this were so, even royal families would not have had any kids before 1850, there certainly wouldn't be a massive population boom in e.g. India where they've only recently connected (almost) everyone to the electricity grid.
> Ever since the 1980s, Germany and other European models already knew population trends were gonna be fucked in the future, all the way back then. Did they take any measures and do anything to help the European population have more kids and prevent this?
I was born in 1983. I remember being warned of overpopulation, there was literally zero public concern about a demographic crisis, and even in the last few years people are mostly warning this will affect us by the time I reach pension age.
I also remember ongoing press campaigns in the UK demonising single mothers.
> instead of fixing the local issues preventing the locals from having kids, then introduce speech control laws for anyone who criticizes this.
That's a new one.
You think there's a law banning people from criticising the lack of support for families? Have you seen, like, any election campaign ever? One reliable theme throughout, no matter how effective the policy would be if examined closely, is at least one party saying they support families.
> EU has made to combat climate change, are being absorbed by US, CHina and India to grow their economies, so the planet is still getting more fucked like before except now we made ourselves poorer for it.
China's also going green. India… isn't, but the pain point hits them much sooner than we expect it to hit us, so they probably will. Like, it becomes reliably lethal to work in parts of India before it's expected to make heatstroke deaths more than a passing headline in Europe.
The US was going green, then Trump happened. He's against renewables and doesn't believe in climate change, while also wanting to invade Greenland for reasons that only make sense if one or both of those are good bets; he insists on keeping coal plants open when the owners of those plants don't want that because gas is cheaper; he's lying a lot in general, but specifically by saying China doesn't use the renewables they're exporting. He's all over the place, wildly incoherent, and is mad enough he could lead to WW3 where none of this matters anyway (P(WW3 this year due to him)~=0.05).
Just because of supply/demand alone?
If sitting in a comfy office pays as well as it does, why would people take care of children or build houses for way less?
I think this is just Baumols cost disease in action: you really cant have amazingly well paying jobs (like in SF generally) AND super low paid laborers without some kind of class system/feudalism/etc.
They're still very good for finetuned classification, often 10-100x cheaper to run at similar or higher accuracy as a large model - but I think most people just prompt the large model unless they have high volume needs or need to self host.
[1] claims 15 GB/s for the slowest implementation (Chromium) they compared (all vectorized).
> The 32 bit hash of CRC32 is too low for file checksums. xxhash is definitely an improvement over CRC32.
Why? What kind of error rate do you expect, and what kind of reliability do you want to achieve? Assumptions that would lead to a >32bit checksum requirement seem outlandish to me.
[1] https://github.com/corsix/fast-crc32?tab=readme-ov-file#x86_...
reply