Yes, it's certainly a regulatory headache, also doctors don't like the black box approach because they like being the ones making the decision. One sees this in military planning too: the user needs to be able to tell their unit commander why we should put lives on the line, more than because that's what the computer says.

Abstractly (ha) it makes sense that reason why a thing is analogous to another, the essence of the analogy, may live in a higher layer. Recognizing that it matches a new concrete thing feels like a tough problem, though, perhaps, not categorically harder than, say, image classification. The predictive aspect further challenges the matching: the new instance may not yet have all the parts, the missing piece of the partially fitting analogy becoming a prediction. I don't yet have the experience to know what deep neural networks are capable of.