If X11 had a future I would've recommended xmonad to you. It's advertised as a tiling WM configured in Haskell, but in reality it's more of a library to make your own WM, as it abstracts most of the low-level details but still lets you change almost anything. It's still quite focused on tiling, yes, so it probably wouldn't be a good fit for you, but the idea of having an abstraction layer above low-level X11/Xlib stuff enabling you to easily build a custom WM is, in my opinion, absolutely awesome.
I find it really sad that in the Wayland world all the window managers are coupled with compositors (and possibly even more than that), which makes it that much harder to roll your own. I wish there was a generic compositor/input server/whatever with some sort of window management RPC interface that would allow running a separate window manager in a subprocess. This would make it so much easier to port xmonad over to the Wayland world and take it from there...
(I'm aware of waymonad's existence, but I believe the process barrier between compositor and WM is really practical, especially during development where the WM rapidly changes, sometimes crashes, but the session still survives.)
I find it really sad that in the Wayland world all the window managers are coupled with compositors (and possibly even more than that), which makes it that much harder to roll your own. I wish there was a generic compositor/input server/whatever with some sort of window management RPC interface that would allow running a separate window manager in a subprocess. This would make it so much easier to port xmonad over to the Wayland world and take it from there...
(I'm aware of waymonad's existence, but I believe the process barrier between compositor and WM is really practical, especially during development where the WM rapidly changes, sometimes crashes, but the session still survives.)