I would suggest you stick with it. Docker compose is very powerful once you get the hang of it and it can be used to deploy straight to AWS etc.

But no, you're not a bad person if you give up. Remember that each time you restart a container it goes back to it's original state. Apt update should be done during the image build.