It might as well. If there is no definition of an ABI, nobody is going to build the tooling and infrastructure to detect ABI compatibility between releases and leverage that for the off-chance that e.g. 2 out of 10 successive Rust releases are ABI compatible.
You can have binary dependencies with a stable ABI; they're called C-compatible shared libs, provided by your system package manager. And Cargo can host *-sys packages that define Rust bindings to these shared libs. Yes, you give up on memory safety across modules, but that's what things like the WASM Components proposals are for. It's a whole other issue that has very little to do with ensuring safety within a single build.
