But when removing an item it’s not necessary to visit every element, just re-arrange the pointers (and in CPython, modify the reference count of the item that was removed).
Yes but the problem there is not that your list is backed by a linked list or an array, it's that every single value is boxed and needs dereferencing. The difference between no hops and one hop totally trumps one hop vs two hops.