Note that since linux 3.8 [1] ext4 can embed small files into inodes thus using less space and allow faster reads. Using that feature and adjusting inode size you can manipulate which file sizes are inlined.
OK interesting, so this capability is on by default then? I didn't change the default inode size, but it looks like I could SAVE space by increasing it (from 256 or something). Since I have 1.91M files less than 2048 bytes, I guess I could increase it enough to store most/many of those.
And I guess it halves the IOPS for serving those files, since with a cold cache you need to read block holding the inode and block holding the data.
[1] http://kernelnewbies.org/Linux_3.8#head-372b38979138cf2006bd...