Real world performance differences between v4 and v6 are more likely to be influenced by different routing and network manipulation for v4 vs v6 than the larger address size.
If your v4 goes through NAT and v6 doesn't, that's a big thing.
If you have different peering and transit providers in v4 and v6, that's a big thing.
If overhead from address sizes was really a big deal, we'd see work to push larger MTUs and working MTU discovery, but that kind of stalled a while ago. 1500 works for a lot of people, and many major sites drop effective MTU by 20 or so and that makes more things work, and then it gets swept under the rug. (OTOH, I think Android may have finally gotten MTU probing enabled after many years of shipping it disabled; Apple has had very effective probing, at least on iOS for a long time)
If your v4 goes through NAT and v6 doesn't, that's a big thing.
If you have different peering and transit providers in v4 and v6, that's a big thing.
If overhead from address sizes was really a big deal, we'd see work to push larger MTUs and working MTU discovery, but that kind of stalled a while ago. 1500 works for a lot of people, and many major sites drop effective MTU by 20 or so and that makes more things work, and then it gets swept under the rug. (OTOH, I think Android may have finally gotten MTU probing enabled after many years of shipping it disabled; Apple has had very effective probing, at least on iOS for a long time)