IIRC https://cyclone.thelanguage.org/ did this (among other things -- I think it's also where Rust lifetimes began their, err, life) and ended up having rather poor performance for pointer-manipulation heavy code.

I don't know if things would be better now with more optimizing compilers, but I remember being somewhat surprised by this at the time, since you'd expect most of the overhead from passing fat pointers around to get optimized away.