This restriction comes up quite a lot in discussions about "modernizing" ham radio. I predict that if encryption was universally allowed, the ham bands would quickly become saturated with encrypted channels as hobbyists and not-hobbyists start using the bands for their personal communications networks.
The two main problems with allowing encryption are that
1. The HF bands often have worldwide propagation with very low power levels, which turns local interference into national or international interference, and
2. With encrypted communications it would be impossible to enforce other amateur radio laws, such as forbidding commercial uses.
I think that allowing encryption on dedicated slices of the UHF bands would be a more realistic approach.
I think the biggest reason not to allow encrypted data is the second point you mentioned. The bands will begin being used for purposes other than what the rules of ham radio allow.
There are some individuals apart of groups who would definitely use encryption on the ham bands as a means to have communications taking place that are not allowed by the rules. The biggest one would be communications with a connection to politics. Met plenty of these types who are jealous of the ham band and scorn the rules because of that, so they keep themselves on CB and FMRS.
Also like you mentioned, it is now free radio space for commercial entities to try to use.
There's a difference between modernization and monetization. Commercial traffic can remain banned. License requirements remain in place. And we can have a standard way to transmit digital callsign before an encrypted packet. There are volunteers right now who do a great job of policing the use of ham bands, and FCC does actually initiate enforcement actions including 5 figure fines.
If the transmission consists of <CALLSIGN><ENCRYPTED PAYLOAD> then there is no way to determine if the payload is commercial in nature. One particular example is business owners using amateur radio spectrum for business communications. Or some wise-guy using amateur spectrum for his high frequency trading to beat the speed of copper.
1. HFT firms have definitely tried to perform HFT communication on HF bands before, see [1].
2. IIRC there have been some undecipherable transmissions in ham HF bands that were suspected to be associated with HFT, but I can't find a source right now.
3. While you don't have any additional latency from routers and serialization delays in the path across the pond, you need to keep in mind that long-range HF communications rely on bouncing the signal on the F1/F2 layers of the ionosphere (an atmospheric layer with lots of charged particles/ions that act as a mirror to HF signals). If you're comparing HF transmission with a hypothetical straight-line fiber connection (assuming it the propagation speed in the fiber is comparable to the atmospheric propagation speed [2]), the fiber connection would win.
Additionally, bandwidth is an extremely scarce resource in HF bands (compared to cellular networks or other common VHF/UHF systems). There's a reason people still use CW (morse code) or even PSK31 [3] instead of SSB (voice) [4] in HF bands: You need around 500 Hz for CW signals, around 3000 Hz for SSB signals. PSK31 only needs 31.25 Hz. My point is: While you might beat latency of fiber connections and routers along the path, the throughput is going to be quite low for HF transmission.
[2] Since you mentioned copper: Copper can be quite a bit slower than fiber depending on cable type, since the propagation speed of signals in copper cables depends on the velocity factor. Wikipedia gives a few examples: https://en.wikipedia.org/wiki/Velocity_factor
[3] PSK31 stands for Phase Shift Keying at 31 Bd. The short explanation is that PSK31 is a bandwidth-optimized text transmission mode.
[4] SSB stands for Single Sideband. The short explanation is that SSB is a bandwidth-optimized AM voice mode.
This restriction comes up quite a lot in discussions about "modernizing" ham radio. I predict that if encryption was universally allowed, the ham bands would quickly become saturated with encrypted channels as hobbyists and not-hobbyists start using the bands for their personal communications networks.
The two main problems with allowing encryption are that 1. The HF bands often have worldwide propagation with very low power levels, which turns local interference into national or international interference, and 2. With encrypted communications it would be impossible to enforce other amateur radio laws, such as forbidding commercial uses.
I think that allowing encryption on dedicated slices of the UHF bands would be a more realistic approach.