I've never felt that icky. I don't really have any secrets in my dotfiles, though I have in the past. In that case, I just encrypt the private stuff using my SSH private key (stored in 1password) and age (via sops-nix).