Frustratingly, this has also been happening in Apple's open source xorg server for like a decade. Whenever I unplug from my external monitor I just kill it and restart with `xinit`.

https://www.xquartz.org/