"durability" isn't voodoo. Consider if dpkg updates libc.so and then you yank the power cord before the page cache is flushed to disk, or you're on a laptop and the battery dies.
And if yank the cord before the package is fully unpacked? Wouldn't that just be the same problem? Solving that problem involves simply unpacking to a temporary location first, verifying all the files were extracted correctly, and then renaming them into existence. Which actually solves both problems.
Package management is stuck in a 1990s idea of "efficiency" which is entirely unwarranted. I have more than enough hard drive space to install the distribution several times over. Stop trying to be clever.
Not the same problem, it's half-written file vs half of the files in older version.
> Which actually solves both problems.
it does not and you would have to guarantee that multiple rename operations are executed in a transaction. Which you can't. Unless you have really fancy filesystem.
> have to guarantee that multiple rename operations are executed in a transaction. Which you can't. Unless you have really fancy filesystem
Not strictly. You have to guarantee that after reboot you rollback any partial package operations. This is what a filesystem journal does anyways. So it would be one fsync() per package and not one per every file in the package. The failure mode implies a reboot must occur.
> It's called being correct and reliable.
There are multiple ways to achieve this. There are different requirements among different systems which is the whole point of this post. And your version of "correct and reliable" depends on /when/ I pull the plug. So you're paying a huge price to shift the problem from one side of the line to the other in what is not clearly a useful or pragmatic way.
> You have to guarantee that after reboot you rollback any partial package operations.
In both scenarios, yes. This is what dpkg database is for, it keeps info about state of each package: whatever is it installed, unpacked, configured and so on. It is required to handle interrupted update scenario, no matter if it was interrupted during package unpacking or in the configuration stage.
So far you are just describing --force-unsafe-io from dpkg. It is called unsafe because you can end up with zeroed or 0-length files long after the package has been marked as installed.
> This is what a filesystem journal does anyways.
This is incorrect. And also filesystem journal is irrelevant.
Filesystem journal protects you from interrupted writes on the disk layer. You set some flag, you write to some temporary space called journal, you set another flag, then you copy that data to your primary space, then you remove both flags. If something happens during that process you'll know and you'll be able to recover because you know in which step you were interrupted.
Without filesystem journal every power outage could result in not being able to mount the filesystem. Journal prevents that scenario. This has nothing to do with package managers, page cache or fsync.
Under Linux you do the whole write() + fsync() + rename() dance, for every file, because this is the only way you don't end up in the scenario where you've written the new file, renamed it, marked package as installed and fsynced the package manager database but the actual new file contents never left the page cache and now you have bunch of zeroes on the disk. You have to fsync(). This is semantic of the layer you are working with. No fsync(), no guarantee that data is on the disk. Even if you wrote it and closed the file hours ago. And fsynced package manager database.
> There are different requirements among different systems which is the whole point of this post.
Sorry, I was under assumption that this thread is about dpkg and fsync and your idea of "solving the problem". I just wanted to point out that, no, package managers are not "trying to be clever" and are not "stuck in the 1990s". You can't throw fsync() out of the equation, reorder bunch of steps and call this "a solution".
Pretty sure kernel doesn't have to fsync on close. In fact, you don't want it to, otherwise you're limiting the performance of your page cache. So fsync on install for dpkg makes perfect sense.
I didn't say it synced. The file is simply "written" and available at that point.
It makes no sense to trust that fsync() does what it promises but not that close() does what it promises. close() promises that when close() returns, the data is stored and some other process may open() and find all of it verbatim. And that's all you care about or have any business caring about unless you are the kernel yourself.
