It's unsafe in roughly the same way that Rust developers call C and C++ unsafe, right? The advantage is that it's a special, relatively uncommon mode, and it's explicitly marked. But I don't think you can minimize the reality that it's unsafe without also minimizing one of the major selling points of the language.
Because only 10% of your code isn't verified by the compiler but instead needs extra review and tests doesn't eliminate the benefit that 90% of your code is verified by the compiler.