I don't think that matters in terms of whether it is even enforceable. I could sign a document allowing management to take my first born son but them doing so is not legal. "But he signed it!"
You could also sign an NDA. And then you'd be bound by that NDA to not disclose the information it is protecting. Should penalties from that should also be non-enforceable because it would limit what you can say?
Consider that you have no agency if a gun is pointed at you, and that you do have agency if the gun is a water pistol. In your mind, does everything in between exist in a spectrum, or do they fall into one of the two buckets into which the above two scenarios fell? I.e. is your conception of agency binary or continuous?
A significant difference is that it is not legal to own slaves or trade in slaves. That makes contracts for the sale of slaves inherently unenforceable.
It is not illegal to refrain from speaking. If you want to enter into a contract that requires you not to speak under certain circumstances there is probably nothing inherently unenforceable about it.
I'm pretty sure for example it would be legal for the Diogenes Club [1] to expel you if you violated their no talking rule, if someone would actually make a real Diogenes Club.