It doesn't require an attacker to have physical possession of the phone. They just need you to plug your phone into one of their USB ports. Something like a public charging station would be perfect for this.

An IMEI and serial are still unique identifiers that can be used for tracking.