There's an opposite incentive to keep the value of the improvements high (i.e. the house itself and any rental income that might come from it). Degrading the value of the land is hard to do without degrading the value of the improvements at the same time. But the reverse is not true -- increasing the value of the improvements causes only a negligible increase the value of the land (and tax burden). So e.g. tearing down the house and building a quadplex to rent out would not appreciably increase the land value, so the land owner would profit from the additional rental income without having to pay very much more tax.

Of course, if everyone in the neighbourhood does the same thing, the land values will rise considerably. But that's a policy success -- the tax system has collectively incentivized land owners to increase the housing supply to meet demand.