Using communally audited abstractions over unsafe code means you’re using unsafe a lot.
If there was some way to prove that the abstraction is safe, then that would be fine. But the inadequacy of communal auditing is the reason why C has security issues.
The area of Rust code that is unsafe is much, much smaller than the amount in equivalent C code, making it much more tractable to audit. I won't pretend that it's perfect, but it's not remotely comparable to C.
But if you can manually identify an invariant inside an abstraction it can greatly improve performance for callers/users, additionally, tools like Kani use comprehensible macros to facilitate automatically proving safety of `unsafe` code. Not to mention built in linting, package management, docs, and FP that rust/std provides. Lots has been said about unsafe rust, but the most basic libc tools require the whole cascade of upstream callers to check safety, it's basically backwards from the ground up from a resources and an outcomes perspective.
Use of unsafe is very rare (except for FFI to C where it's unavoidable). I've written tens of thousands of lines of Rust and used `unsafe` exactly once.
In fact, there are exactly two "unsafe" blocks in all of my Rust projects, and the second one is not even needed anymore because of the language and ecosystem improvements, but the project is basically abandoned, so I'm probably not gonna fix it. There's just no need for unsafe in the vast majority of code.
I don't know where Rust critics get their statistics; probably from picking their noses, judging by their arguments. Most don't seem to even have read the official docs, the bare minimum required to form any conclusions at all about the language. I guess they don't read much Rust code and think there is no way we can write even semi-decently performing high-level code without resorting to unsafe hacks or calling C, because that's the way it's done in other languages.