Thank you, thank you. I haven't reviewed this, and I may or may not use it, but wow was freebsd-update from 13.0 to 14.0 slow, especially since I run my freebsd systems off spinning drives, and most of them have src.
I did some looking at freebsd-update, and it's quite the shell script, which makes it daunting to consider changing. All the forking and whatever doesn't seem great; using data structures should be a lot simpler!
But my systems running freebsd-update spend an awful lot of time on I/O, and not so much time on cpu, as I recall. I think the general flow for a replaced file is gunzip to the update dir, then install to the destination. This means all the files are written twice, and I thought there was a third write that I can't remember at the moment. Install doesn't have an option to decompress, but it'd be super handy if it did --- then you could write the file only once to the destination directory and move it into place.
I'm never a fan of quite as many files in a directory as freebsd-update gets up to either. Given that everything is already named by hashes, it's not too hard to have 16 or 256 directories named 0-f or 00-ff that get a slice of the files. Large directories have gotten better over time, but they're still slow.
There's some ability to exploit the embarassing parallelism available too, but that's hard in a complex shell script, so some other environment makes sense. I didn't look, but hopefully that's a knob in rustdate ... Sometimes you'd want it and other times one thing at a time is better, even when it's slow.
One feature I know update can do that I want to look at more but haven't had time is the zfs boot environment stuff --- seems like with the right arguments you could do the update on an inactive environment and reboot to switch, instead of the default way which does the install on the live environment but you could reboot to switch back. I'd be much more patient if the live environment continued working and I could reboot at my convenience once the process finished. Especially if I might be able to run pkg upgrade in the boot environment before the reboot. If rustdate supports that too, that'd be neat.
> I did some looking at freebsd-update, and it's quite the shell script, which makes it daunting to consider changing. All the forking and whatever doesn't seem great; using data structures should be a lot simpler!
It's fairly well broken down, and pretty well organized. I think the most difficult part is that so much of it relies on shell nice-isms that are a pain to re-implement in something like C (although, I guess you could just call fetch and the like directly via 'exec' or 'system'; rather than using their libraries to manage things directly).
I did some looking at freebsd-update, and it's quite the shell script, which makes it daunting to consider changing. All the forking and whatever doesn't seem great; using data structures should be a lot simpler!
But my systems running freebsd-update spend an awful lot of time on I/O, and not so much time on cpu, as I recall. I think the general flow for a replaced file is gunzip to the update dir, then install to the destination. This means all the files are written twice, and I thought there was a third write that I can't remember at the moment. Install doesn't have an option to decompress, but it'd be super handy if it did --- then you could write the file only once to the destination directory and move it into place.
I'm never a fan of quite as many files in a directory as freebsd-update gets up to either. Given that everything is already named by hashes, it's not too hard to have 16 or 256 directories named 0-f or 00-ff that get a slice of the files. Large directories have gotten better over time, but they're still slow.
There's some ability to exploit the embarassing parallelism available too, but that's hard in a complex shell script, so some other environment makes sense. I didn't look, but hopefully that's a knob in rustdate ... Sometimes you'd want it and other times one thing at a time is better, even when it's slow.
One feature I know update can do that I want to look at more but haven't had time is the zfs boot environment stuff --- seems like with the right arguments you could do the update on an inactive environment and reboot to switch, instead of the default way which does the install on the live environment but you could reboot to switch back. I'd be much more patient if the live environment continued working and I could reboot at my convenience once the process finished. Especially if I might be able to run pkg upgrade in the boot environment before the reboot. If rustdate supports that too, that'd be neat.