The smallest unit of compilation in Rust is the crate, not individual files unfortunately. That's why you'll see larger projects like Wezterm or uv use workspaces to try to tame compile times.
You're right that the crate, not the file, is the smallest unit of compilation.
My largest rust project is still fairly small at around 8k lines across two dozen files. Still, it builds and links in 30 seconds on my workstation which is several years old and not particularly fancy.
I could see this beginning to become an issue around 100k lines of code, but I'd think most people would be looking to split into multiple crates before that just for readability.