AI explanation problem

AI / machine learning is quite useful but not being able to see the justification of a particular classification outcome is quite annoying for me. As Mr. Tizhoosh puts it in Introduction to Digital Pathology:

After weeks of training of some sort of agent I can just tell you "yes" but I cannot tell you "why"... The pathologist writes a full scale report and he/she has to justify it.

See Explainable artificial intelligence.