This has been brought up in the past and Apple did try to cater to that market with their smaller phone and it seemed the market response was not great, they ultimately had to drop the iPhone mini because it was not worth it.
I’m sure there are many out there like me, but I used to prefer smaller phones until I used a larger phone for one generation; after trying to move back to a smaller phone, I just missed the real estate of a larger screen.