Interesting and well-written article. Thanks to the author for writing it. Replacing Spark with these single-machine tools seems to be on the hype, and Spark is not en vogue anymore.
The author ran Spark in Fabric, which has V-Order write enabled by default. DuckDB and Polars don't have this, as it's an MS proprietary algorithm. V-Order adds about 15% overhead to write, so it does change the result a bit.
The data sizes were bit on a large size, at least for the data amounts I see daily. There definitely are tables in the 10GB, 100GB, and even in 1TB size range, but most tables traveling through data pipelines are much smaller.
FYI I had V-Order and Optimzed Write disabled in the benchmark. The only wrote diff was that I enabled deletion vectors in Spark since it’s supported which the other two don’t.
I had pinched ulnaris nerve in both elbows a few years ago. The symptoms were pretty much the same as you describe. It took two operations and more than a year to recover. The symptoms started to creep back this year, probably because I started to hit the keyboard more. Luckily I found a book about trigger points [1] and releasing the triceps trigger points, especially #3 removes the symptoms. The book is godsend; I've found help from it to so many other issues too.
> The Supplement section of the letter states that the study was conducted from February to October 2020. The year reported was an error; the study was conducted in 2021. The Journal is in the process of correcting this error.
Quite the opposite. Covid is dangerous for elderly and people having severe medical condition, much, much less dangerous for young and healthy. Flu is more dangerous to young people than Covid.
Flu also causes hospitalizations in comparable amounts to Covid[1]. Note that we have a vaccine for flu, and it is still causing almost a million hospitalizations in bad seasons.
Yes, I bore that in mind when I made the statement. Especially the younger people the GP talked about do not bother getting flu shots. If everyone got vaccinated, the yearly influenza waves would result in way fewer deaths.
Since you can get both influenza and Covid-19, even at the same time, German health authorities have strongly recommended getting flu shots and started the vaccination campaign earlier this year. It's good advice. Get your flu shot!
I got a really hard pneumonia few years ago. It took several days in hospital to find an antiobiotic which actually worked, and I was in pretty bad shape in ER when it finally started to work.
It took 10 weeks to recover back to work, and over a year before lungs were back to normal, confirmed by x-ray.
Pneumonia can be deadly, even for healthy young adult.
"However, these masks would provide the wearers little protection from microorganisms from others persons who are infected with respiratory diseases. As a result, we would not recommend the use of homemade face masks as a method of reducing transmission of infection from aerosols."
Completely agreed! Excess deaths are reaching all time lows in many European countries. Most of the excess deaths - as in Italy - were probably due to antibiotic resistance.
But this is something you can't say, not at least in HN. You have to agree and obey the official truth.
It is mind boggling how otherwise sensible people panicked and refuse the see the truth. I really hope vaccine would come soon, just to calm people down.
The author ran Spark in Fabric, which has V-Order write enabled by default. DuckDB and Polars don't have this, as it's an MS proprietary algorithm. V-Order adds about 15% overhead to write, so it does change the result a bit.
The data sizes were bit on a large size, at least for the data amounts I see daily. There definitely are tables in the 10GB, 100GB, and even in 1TB size range, but most tables traveling through data pipelines are much smaller.