The desktop is just there to launch and switch between applications. I want it to stay simple and to keep out of my way. I've not tried Xubuntu yet, but it sounds/looks like it will be better at satisfying these requirements for me.
"The desktop is just there to launch and switch between applications."
If that's how you look at the desktop, then you should definitely look at Lubuntu. It's just windows and a panel. When you look at an Xfce desktop, it looks like Xfce. When you look at Lubuntu, it's just windows and a panel.