to me it was having to carry the fuel for the landing burn. Also, an empty booster is very fragile, they're basically metal balloons. I figured the landing legs, extra support needed, and the required fuel would make it infeasible in the end.
> an empty booster is very fragile, they're basically metal balloons.
If you think about it that's not really the case. Post launch the boosters are empty and must sustain their own weight and that of the payload sitting upright on the launch pad prior to fueling.
During the launch the full force of the engine is exerted on everything and it must withstand that force even as the tanks empty. If the boosters were not able to withstand these forces then as the rocket climbed into the sky you'd see it start to compress like a tin can the higher into the atmosphere it climbed.