To be fair to Android, this is a limitation of the MTP protocol and not android.
To mount your storage as a mass storage device then the host device (your computer in this case) does raw sector read/writes to the device, the host device provides the filesystem services. For this to work it has to be completely unmounted from the phone as obviously having the block mounted in two filesystems at once would corrupt everything very badly.
Android used to split storage into various partitions, which is why this used to work - It was able to unmount the partition and let your PC manage it. This meant any apps using that partition needed to be stopped, etc etc. It was a pain, and I can totally understand why they moved away from this approach.
Personally I prefer the new way, yes using MTP has some limitations as you've noticed but it does mean the storage can remain mounted on android while your PC accesses it.
I can't think of any modern operating system that lets a foreign system mount it's already mounted filesystem over USB without going through some kind of server such as MTP, NFS etc.
In the case of plugging a "phone", as a device, into a USB host computer, the USB device (the phone) can present a filesystem endpoint to the host, and allow read/write access. The OS of the phone then passes these read/writes through to its mounted filesystem, with whatever mapping and access controls to the mounted filesystem it wishes to implement.
Thus the USB connection doesn't require that the raw filesystem of the phone be mounted by 2 hosts at the same time.
This already happens with every USB "stick" you plug into a host computer. The memory in the USB stick is accessed by firmware on a CPU inside the memory stick, which then presents that memory to the host as a USB storage class device. The firmware may not have a linux or iOS OS, but it does perform mapping to preserve and remap sectors to alleviate flash endurance issues, perform secure mounting, and other features.
There's no technical reason android can't do this.
p.s. MTP clients accessing an android device are a major PITA! Especially, ironically, for a linux OS USB host...
I think the issue comes with trying to present a USB mass storage API to the host where you allow the host to write to any arbitrary byte offset of the mass storage device without the structure that is imposed by a file system.
If the host and guest both get presented with the same “array of bytes” mass storage interface, then they will compete and potentially mess up each other’s reads and writes, let’s say if they both treat that array of bytes as an ext4 file system and try to write a file system metadata to the same physical location at the same time.
Of course you have have a “virtual file system” exposed over USB, but isn’t that exactly what MTP is? The point is that USB mass storage is not a virtual file system.
Android used to split storage into various partitions, which is why this used to work - It was able to unmount the partition and let your PC manage it. This meant any apps using that partition needed to be stopped, etc etc. It was a pain, and I can totally understand why they moved away from this approach.
Personally I prefer the new way, yes using MTP has some limitations as you've noticed but it does mean the storage can remain mounted on android while your PC accesses it.