What does "the financial system" mean? Obviously there's a level you can address with that term where, just like with ML, interpretability becomes an open question. But on most every level that is genuinely comparable to the role an ML model plays in a software product, finance is plenty interpretable.