If SIM Swap doesn’t work, you can always attack SS7. There’s also nothing you can do about that.

So stop using your phone number as an authentication factor. It’s trivial to pwn for any actor determined-enough.