cargo fetch gets most of the way there. But you would need to build them as well to actually get good performance.

This is the upstream issue in question: https://github.com/rust-lang/cargo/issues/2644

Cargo fetch and/or cargo-chef kind of does these things, but not in a super intuitive and easy to use way.