I have a datacenter with so many machines (blade computers) in it that dnsmasq couldn't keep up and we ended up having really bad UDP storms. That wasn't fun to debug at all since we had no idea that dnsmasq would have scaling issues like that. Even worse is that the logging output is abysmal and made things even harder to diagnose.
Switched to another DHCP server and the storms went away, but I am not entirely happy with it because of the way it handles leases, it doesn't guarantee that each MAC keeps the same IP over time.
Now, I'm always on the lookout for a good DHCP server... tried another one that was golang, but it wasn't working on my network for some weird spec reason, I didn't have time to debug it and the authors didn't seem that interested in helping out.
Did you try Kea[0]? There are a variety of ways to ensure the same MAC addresses get the same IPs even after the initial lease expires.
I'm not surprised you had issues with dnsmasq - I wouldn't use it for anything other than a run of the mill /24 (or maybe a few of them on VLANs). Don't get me wrong - it's perfect for what it was originally intended for and I use it in at least half a dozen places. Just not for an application and network like yours.
Kea is what we ended up with. It has a few feature differences from dnsmaq that we had to work around a bit. Not the end of the world, but not fun when you're under pressure.
Now that I've studied the source code of dnsmasq a bit more because of the issues we had, I have to agree with you. We had run it with about 3k blades and it was fine. It was fine at about 9k blades too... but once we hit around 10k, it just fell over.
The frustrating part was that it just didn't log what was up. We used netdata and discovered the UDP queues were filling up. Tried mucking with tuning the sysctls and that didn't work either.
Switched midstream to Kea, which was a giant pain because we had a hard time dealing with duplicate IP addresses (imagine rebooting 10k+ PXE booted blades during a UDP storm).
Frankly I'm AMAZED you made it to 9k leases with dnsmasq. Not only from a performance standpoint but the overall logging, lack of instrumentation, limits in configuration, etc. As you say (from reviewing the source) it's really not intended for anything other than small home/SMB networks, memory and CPU constrained devices, etc. It's perfect for that and it's my go-to in those scenarios. A 10k host network is a completely different animal and from the down-votes my comment received it doesn't seem like many here on HN have been in scenarios like these either.
Your comment about the logging is spot on as well. I think dnsmasq and I think "small flash device where the logs will never get stored anywhere and no one can or will view them anyway". As you can probably tell I've had issues getting meaningful logging and debug out of dnsmasq as well...
I don't envy your situation in the slightest!
Note to mods - I'm not complaining about downvotes. I'm using the downvotes as a sign there's little understanding on HN of network ops at scale. People may love dnsmasq on a home router (I do) but plenty of network admins like OP and I have watched it fall over in other scenarios.
Both myself and my super talented devops coworker hadn't ever run with that many individual machines in a single location. 4-5k was both our maxes. It worked fine before, we didn't expect it to fall over.
dnsmasq has some really nice features that kea doesn't have. just finding a single blade in the datacenter, means a naming system, and it was easier with dnsmasq, and the config was easier to create. We ended up having to use ansible to generate the kea config file because we also keep machines subnet separated based on physical location.
HN has all sorts of people, I never try to quantify it. We don't know what we don't know.
We've done some basic benchmarking at +5k leases/second.
If you want to go down the rabbit hole with us of looking at using Dora and giving us feedback I'd be glad to chat further cdarby[at]bluecatnetworks[dot]com
Thanks. I've saved your project and if we are ever in the mood to switch, we will definitely give you a try first. Keep up the good work, you're solving a real problem.
On IPv6 this would be a solved issue. With SLAAC the router just broadcasts Router Advertisements and all the machines just automatically configure their addresses, which are fully predictable based on their MAC address (if you have privacy extensions disabled). The router has no state to keep and as such the network can be scaled up "infinitely". Just pick a random /64 prefix and set up radvd (or systemd-networkd) to send the router advertisements.
Sure. The thing is that we also have a physical location based naming system which is somewhat tied to the IPv4 address and we use that system across multiple data centers. We also have a single daemon running on each box that helps with the naming. It would complicate things (ie: more code to write) to have one dc (where we had these issues), on IPv6. We hadn't had issues before, so it never really made sense to use IPv6 other than for the novelty of it. IPv4 works surprisingly well and the issue really wasn't at that layer. Who knows too.. maybe radvd or systemd-networkd also have other issues we don't know about.
Switched to another DHCP server and the storms went away, but I am not entirely happy with it because of the way it handles leases, it doesn't guarantee that each MAC keeps the same IP over time.
Now, I'm always on the lookout for a good DHCP server... tried another one that was golang, but it wasn't working on my network for some weird spec reason, I didn't have time to debug it and the authors didn't seem that interested in helping out.