> Most people also like to come back to warm house so heating is running during the day

Also if you have pets you don't want it to be miserable for them. You might make it a couple degrees warmer/colder but not too much.

Also if you have pets you can't really turn off the heat (or AC in the summer for desert areas), and children usually get home in the afternoon while people are still working.