If you compiled it and it used these files and none of those changed, you don't need to compile it again.
When you add a new file, it clearly doesn't have an output file, so it needs to be compiled. Where it gets a little less clear is when you've got something that depends on all the input files --- if you just use the compiler's dependencies, when you add a new input file, Make (or whatever) wouldn't care, because last time it compiled (or linked), that file wasn't used, because it didn't exist.
So you still need some rules that are based on directory scans, if that's how you want to roll.
It'd be moot after they've been compiled. You're saying compile to get the information to decide if you should compile.