If anything you might want to explicitly do -fno-lto to not use LTO when you have fat objects.
It does more than optimize.