Journal Article
Machine Learning

Definitions, methods, and applications in interpretable machine learning

William J. Murdoch(University of California, Berkeley), Chandan Singh(University of California, Berkeley), Karl Kumbier(University of California, Berkeley), Reza Abbasi-Asl(Allen Institute for Brain Science), Bin Yu(University of California, Berkeley)
October 16, 2019Proceedings of the National Academy of Sciences2,106 citations
arXivPDFDOI

2.1k

Citations

55

Influential Citations

Proceedings of the National Academy of Sciences

Venue

2019

Year

Abstract

Machine-learning models have demonstrated great success in learning complex patterns that enable them to make predictions about unobserved data. In addition to using models for prediction, the ability to interpret what a model has learned is receiving an increasing amount of attention. However, this increased focus has led to considerable confusion about the notion of interpretability. In particular, it is unclear how the wide array of proposed interpretation methods are related and what common concepts can be used to evaluate them. We aim to address these concerns by defining interpretability in the context of machine learning and introducing the predictive, descriptive, relevant (PDR) framework for discussing interpretations. The PDR framework provides 3 overarching desiderata for evaluation: predictive accuracy, descriptive accuracy, and relevancy, with relevancy judged relative to a human audience. Moreover, to help manage the deluge of interpretation methods, we introduce a categorization of existing techniques into model-based and post hoc categories, with subgroups including sparsity, modularity, and simulatability. To demonstrate how practitioners can use the PDR framework to evaluate and understand interpretations, we provide numerous real-world examples. These examples highlight the often underappreciated role played by human audiences in discussions of interpretability. Finally, based on our framework, we discuss limitations of existing methods and directions for future work. We hope that this work will provide a common vocabulary that will make it easier for both practitioners and researchers to discuss and choose from the full range of interpretation methods.

Analysis

Why This Paper Matters

This paper addresses a critical gap in the rapidly growing field of interpretable machine learning: the lack of a unified definition and evaluation framework. As machine learning models become more complex and are deployed in high-stakes domains like healthcare and finance, the ability to interpret model decisions is essential for trust, debugging, and regulatory compliance. The authors provide a much-needed conceptual foundation that helps practitioners navigate the confusing array of interpretation methods.

By introducing the PDR framework (predictive accuracy, descriptive accuracy, and relevancy), the paper shifts the focus from purely technical metrics to the human audience that ultimately uses interpretations. This human-centric perspective is often overlooked in technical discussions and is a key contribution that makes the framework practical for real-world applications.

Technical Contributions

The paper makes several key technical contributions:

  • Definition of interpretability: Interpretability is defined as the ability to explain or present model behavior in understandable terms to a human.
  • PDR framework: Three desiderata for evaluating interpretations:
    • Predictive accuracy: how well the interpretation matches the model's predictions.
    • Descriptive accuracy: how faithfully the interpretation represents the model's internal logic.
    • Relevancy: how useful the interpretation is for a specific human audience and task.
  • Categorization of methods:
    • Model-based methods: inherently interpretable models (e.g., linear models, decision trees, sparse models).
    • Post hoc methods: applied after training (e.g., feature importance, partial dependence plots, LIME, SHAP).
    • Subgroups: sparsity (few features), modularity (decomposable components), simulatability (human can simulate model).
  • Real-world examples: Demonstrates how the PDR framework can be applied to evaluate interpretations in domains like genomics and image classification.

Results

This paper is conceptual and does not present quantitative experimental results. Instead, its main result is the PDR framework itself, which provides a structured way to evaluate and compare interpretation methods. The authors illustrate the framework with case studies, showing how different interpretation methods (e.g., sparse linear models vs. post hoc feature importance) can be assessed along the three PDR dimensions. The paper has been highly influential, with over 2100 citations, indicating its broad adoption as a reference in the interpretability literature.

Significance

The broader impact of this work is substantial. It provides a common vocabulary that enables clearer communication between researchers and practitioners. By highlighting the role of human audiences, it encourages the development of interpretation methods that are not only technically accurate but also practically useful. The categorization of methods helps newcomers quickly understand the landscape and choose appropriate techniques for their problems. The paper also identifies limitations, such as the lack of quantitative evaluation metrics, which has spurred further research into benchmarking interpretability methods. Overall, this paper is a foundational reference that has shaped the direction of explainable AI research.

interpretabilitymachine-learningpdr-frameworkmodel-basedpost-hocsparsitymodularitysimulatability
Attention Mechanisms

Related Papers