I second this. Smaller models indeed may be much better positioned for fine-tuning for the very reason you point out - less noise to begin with.