Direnv plus nix will go a long way on linux and macos. Windows should be using wsl anyway.

If you can get an image with just those two tools installed then each repo can specify what things it needs and when you cd into the repo it will install all the tools you need.