Looks like the slowdown is actually at sshd process startup time, not authentication time. So it's back to being completely impossible to network-probe for.