Yes, adjusting margins for every icon's amount of padding doesn't seem to be the right solution. I guess the solution would be to have icons without any padding, and where they should be shown centered inside a standard-sized box, that can be done in the layout code.
It's going to take quite a bit of work to switch everything over.