Plus the default is to launch a different instance of the compiler for every .c/.cpp file, so that state has to be written out to disk and reread by the next one.
If I ever did another C or C++ project (which would probably only be at gun–point), I would go the opposite route and have only a single compilation unit. I would have a single primary source file that included all of the others, and none of the others would include anything. Then I could run the compiler a single time on just that one file and it would be as fast and as simple as possible. Well, I don’t know of any C or C++ compilers that can spread their work over multiple cpus (since we usually just run multiple instances simultaneously with make -j), but other than that it would be as fast as possible.