1. The regulatory burden on manufacturing is much higher here than foreign countries
2. Those foreign countries have a deliberate policy of maintaining a trade surplus in order to improve their own industrial capacity at the expense of our own
I would absolutely expect that even in a more level trade environment there would still be some foreign manufacturing of clothes. But what we are seeing is not the happy accident of the free market, but the result of very deliberate government policy.
As such, Americans don't make many of their own clothes.