no need to re-encode the video if you're only manipulating the audio, and audio is so small compared to video you could just have it lossless (or effectively lossless)

amusing I made the same mistake when batch digitising tapes, did about 20 before realising the audio was out of sync