It wasn't actual WoL. It was broadcasts. For some reason it woke the Mac up when WoL was switched on. We're taking 2015 or so so I don't have the packet logs anymore but I was able to replicate it by resending them.
So the fault is really the Mac's. It should only accept real well formed WoL packets with its own Mac address.
OK, from the comments I read so far I realize that I may have been lucky with an older laptop. And I did not mention that I also turned off wake timers, realized that now. If this does not work on modern hardware that’s a real bummer.
20% of the time it works.
40% of the time it won't wake and I have to kill it (screen stays dark even though keyboard indicates it is awake)
30% of the time it just immediately wakes
10% of the time it wakes at like 2am and power cycles my monitor every 30 seconds waking me up