I'm in no way a researcher or even an enthusiast of machine learning, but I'm pretty sure that I came across a paper posted on HN a few days ago that did exactly what you and the parent poster are describing, figuring out what pixels contributed most to some machine learning algorithm. I'll try and see if I can find it.
Edit: yep, found it.
SmoothGrad: removing noise by adding noise, https://arxiv.org/abs/1706.03825
Web page with explanations and examples
https://tensorflow.github.io/saliency/
I couldn't find the HN thread, but there was no discussion as far as I remember.