My 3080 can turn a 16 sample Euler_a @ 512^2 in about 1.5s (9.7 iterations/s). I've found you can yield pretty good results in txt2img with the settings. And once you've found a good image you can further iterate in img2img with loopback at approximately the same rate.
It's worth noting that I'm on a 5800X as well, I'm sure.
I guess it depends on how you do it. Depending on how I've set things up, I've found that more samples isn't necessarily better (usually just different). I suppose it's an optimization problem. I have found that I can pretty reliably look at a 1 sample image and kinda guess where the earliest iterations are going to go, and that might be the most appropriate workflow, actually, but beyond that it seems a couple of samples can drastically alter outputs, and likewise with prompt editing. Whereas with img2img there's a lot more control, I pick an input, I can force it to strictly abide to the image and the parameters I want, and as someone else said in- and outpainting are nice as well.
I guess I'm just manipulating probabilities in my favor?
It's worth noting that I'm on a 5800X as well, I'm sure.