People have used genetic algorithms to evolve a set of images to fool DL machine vision models. Your point stands, we don't know of a deterministic method that I know of that can generate images optimized to fool DL models, and that'd indicate a black box model.