20 bytes / 1 Gbps = 160 ns per hop. That's 0.016 ms additional latency over 100 hops.
On the internet, most links are faster than 1 Gbps and most paths are shorter than 100 hops, so that's a conservative estimate.
If you're sending lots of 10-byte payloads, then IPv6 requires (40+10)/(20+10)=166% as much network capacity, but are you really filling up an expensive link with VoIP traffic?