One major reason why it is cheaper to make these clothes overseas is because we have such a stringent regulatory regime for worker and environmental protections, and using foreign manufacturers is essentially a loophole through those regulations. Whether those regulations are useful or not is entirely beside the point: if they are good, then moving manufacturing over to lower regulation countries is bad and we should impose tariffs on all of those goods. If those regulations are bad, then we should just repeal them and let domestic manufacturers compete on an even playing field.
1. The regulatory burden on manufacturing is much higher here than foreign countries
2. Those foreign countries have a deliberate policy of maintaining a trade surplus in order to improve their own industrial capacity at the expense of our own
I would absolutely expect that even in a more level trade environment there would still be some foreign manufacturing of clothes. But what we are seeing is not the happy accident of the free market, but the result of very deliberate government policy.
We have better things to do, and others will do it faster, cheaper, and better than we can.