Spend some time reading The Old New Thing. Microsoft's ability to deal with buggy applications is world class. It's one of the core competencies of the Windows team. There are not too many libraries.
kmontrose is right. App compat is an application-level thing, not a library-level thing. Applications can load libraries dynamically even if they didn't want to (shell extensions are the classic example here).
Do you want explorer.exe to think the OS is named Windows 10 because it loads a library that requires the compat fix?
An example with made-up names: You have an application a.exe that loads b.dll and c.dll. Both b.dll and c.dll contain a call to getOSName(). getOSName normally returns "OS Foo", but c.dll is known to use the result of getOSName() incorrectly and it is decided that all calls to getOSName() from c.dll should return "OS Bar" for :reasons:.
How do you get the OS to do this without returning "OS Bar" to b.dll as well?
Remember that function pointers can cross DLL boundaries.
1. You're advocating for the OS compat shim applier to run on every call to load a DLL into the process space, not just every call to start a process. The former can be many times the latter.
2. You're suggesting the OS change third-party code (if only in memory) and said third parties will be okay with this. (App compat shims in Windows wrap the OS function by modifying the import table, not the callsites.)
3. You assume "the relevant function" is something that exists. It's perfectly valid to generate functions at runtime. The shim would have to patch all the places that generate code, all the places that generate the places that generate code, and so on.
4. (Function pointers can cross DLL boundaries.) b.dll contains a function that returns the result of getOSName(). b.dll passes this function pointer to c.dll. c.dll executes it and gets the unshimmed name.
Edit: As an example of (3), consider that c.dll is a Java JAR that contains JNI calls to getOSName(). The shim would have to patch the Java interpreter that interprets that JNI call bytecode so that it instead generates a function call to getOSNameCompat. Then it'd have to shim the Java JITter that eventually generates native code from the bytecode so that the native code calls getOSNameCompat.
And don't forget the Java interpreter and JITter aren't even in c.dll, they're in the process that's loading c.dll, so the shim isn't even limited to the same file that necessitated the shim in the first place.
And what is the Java interpreter / JITter anyway? c.dll only requires to be loaded by java.exe, but java.exe is provided by Oracle Java, OpenJDK, and any other Java vendors. Would the shim have to know how to patch each of those depending on which gets used to load c.dll, just because c.dll needs a shim?
