Okay, but my point there was saying "Google has millions of servers" isn't relevant, we're not looking at the entire company.
Even with a few addresses per VM, how many racks do you need to put into the same shared-compute mass? One data center is the upper limit, but it doesn't have to be the entire data center.
Back of the envelope math using modern hypervisors that can fit loads of vms in a single U
- let’s put 500 vms on a single one. with 128c256t CPUs it’s easy
- say you can fit 30 of those in a single rack (the common rack is 42U) due to power constraints
- and place 10 of those racks
That’s 500 x 30 x 10 = 150000 nodes to address. With 10 racks you already blow past the scaling limits of the common datacenter switch when it comes to MAC addresses. Here are the limits for Cisco’s Nexus 9000 series, a very common datacenter switch: https://www.cisco.com/c/en/us/td/docs/switches/datacenter/ne...
Plus layer 2 switches when they don't know the destination port will flood. With so many hosts that would be absolutely horrific.
I've seen computers at moderately sized LAN parties (talking 100 nodes, far from the large or even massive events) that were literally crippled by the broadcast traffic. At some point the flooding and layer 2 discovery (ARP) would do the same as well.
Limiting the broadcast domains with layer 3 really makes the Internet possible. Sure, you can have less overhead and simply do layer 2 only, and really it is completely possible. It's just such a rare use case that it in practice isn't important enough to actually do.
Okay, but my point there was saying "Google has millions of servers" isn't relevant, we're not looking at the entire company.
Even with a few addresses per VM, how many racks do you need to put into the same shared-compute mass? One data center is the upper limit, but it doesn't have to be the entire data center.