Financial IT is interpretable, maybe, but is the financial system itself? You have a lot of agents taking actions that you can't really interpret from outside - unless you say something vacuous like this person made this trade because they think it was good, at which point you may as well say this ai model made this decision because it thought it was good.
What does "the financial system" mean? Obviously there's a level you can address with that term where, just like with ML, interpretability becomes an open question. But on most every level that is genuinely comparable to the role an ML model plays in a software product, finance is plenty interpretable.
