SIM swapping lets you take over a phone number. It does not let you clone the keys stored in your physical phone’s secure storage / enclave / key storage.

So SIM swapping makes SMS verification vulnerable (that just depends on controlling the phone number), but doesn’t fundamentally affect iPhone/Android passkeys.

Using phones as physical security keys != using SMS for 2FA.

There are two separate problems - the first one is to make sure others don't have the access you don't want them to, while the other is to make sure that you/others have the authorized access.

SIM swapping means someone else might be able to have my phone number, but I'll eventually get this number back via legal means. So the number itself is something I own (at least in my country). Now if a site assumed this and phone numbers were meant to be constant, it'd mean I could always get my account back.

But of course this depends on which problem I consider more important. It's better to lose access to my FB account forever than to allow someone to gain access to it for a moment, because that might cause harm. Similarily, it's better to lose access to my bank account until I have to visit them in person than to lose all the money, but in this case the weakest link is probably not the key itself.

I'm quite concerned that phone numbers I don't own or addresses I no longer live at have legal "owners" and their friends/sublets who could decide to impersonate me with accounts that still have these old details, and its kind of hard to separate that from my new details having been the illegitimate hack.

I think most people wouldn't risk that unless they are pretty messed up and an average legal system to deal with that unless it is pretty messed up. A past residence in a mediocre city in the US provides all the crack heads needed for such a torturous comedy.

sim swapping doesn't move the authenticator app to the attackers phone, only your phone number.

this is why SMS-based 2nd factor has been considered insecure for years.