# Fewer Than 1% of Explainable AI Papers Validate Explainability with Humans
**Authors**: Ashley Suh, Isabelle Hurley, Nora Smith, Ho Chit Siu
> ashley.suh@ll.mit.edu0000-0001-6513-8447MIT Lincoln LaboratoryLexingtonMAUSA
> isabelle.hurley@ll.mit.edu0009-0006-4031-9018MIT Lincoln LaboratoryLexingtonMAUSA
> nora.smith@ll.mit.edu0000-0001-5255-9226MIT Lincoln LaboratoryLexingtonMAUSA
> hochit.siu@ll.mit.edu0000-0003-3451-8046MIT Lincoln LaboratoryLexingtonMAUSA
(2025)
Abstract.
This late-breaking work presents a large-scale analysis of explainable AI (XAI) literature to evaluate claims of human explainability. We collaborated with a professional librarian to identify 18,254 papers containing keywords related to explainability and interpretability. Of these, we find that only 253 papers included terms suggesting human involvement in evaluating an XAI technique, and just 128 of those conducted some form of a human study. In other words, fewer than 1% of XAI papers (0.7%) provide empirical evidence of human explainability when compared to the broader body of XAI literature. Our findings underscore a critical gap between claims of human explainability and evidence-based validation, raising concerns about the rigor of XAI research. We call for increased emphasis on human evaluations in XAI studies and provide our literature search methodology to enable both reproducibility and further investigation into this widespread issue.
Explainable AI, Trustworthy AI, Interpretability, Human-Centered AI journalyear: 2025 copyright: rightsretained conference: Extended Abstracts of the CHI Conference on Human Factors in Computing Systems; April 26-May 1, 2025; Yokohama, Japan booktitle: Extended Abstracts of the CHI Conference on Human Factors in Computing Systems (CHI EA â25), April 26-May 1, 2025, Yokohama, Japan doi: 10.1145/3706599.3719964 isbn: 979-8-4007-1395-8/2025/04 ccs: Human-centered computing Empirical studies in HCI ccs: Computing methodologies Cognitive science ccs: Computing methodologies Philosophical/theoretical foundations of artificial intelligence
1. Introduction
The growing accessibility of artificial intelligence (AI) promises significant benefits for our society (Dash et al., 2019; Rolnick et al., 2022; Ly et al., 2017), as well as a number of critical challenges (Board, 2019; Cath, 2018; Borenstein and Howard, 2021). A pressing obstacle in the field is the lack of explainability, trustworthiness, and transparency in AI technologies (Rudin, 2019; Saeed and Omlin, 2023; Rudin and Radin, 2019; Holzinger et al., 2017). With AI systems being deployed to diagnose medical conditions, navigate vehicles on the road, and forecast natural disasters, understanding a modelâs behavior, recommendations, and reasoning has become imperative (Edwards, 2021). Yet, habitually used yet nondescript metrics like âaccuracyâ and âconfidenceâ fail to capture the whole story of an AI model nor how it will be interpreted by humans in real-world contexts (Suh et al., 2024).
The potential for skepticism and hesitation in trusting AI has driven the community to develop techniques in interpretable and explainable AI (XAI), concepts that are generally interchangeably used (Lipton, 2018), along with transparent and trustworthy AI While a precise definition for interpretability is not universal, Doshi-Velez and Kim define it as the ability to explain or present AI/ML systems in understandable terms to a human (Doshi-Velez and Kim, 2017). We refer to these methods collectively as research in explainable AI (XAI).. Common XAI approaches include rule-based reasoning, visualizations of deep learning processes, text-based explanations, and example-driven counterfactuals (Saeed and Omlin, 2023). Altogether, the developers of these explainability methods argue that AI models are now more usable, interpretable, and effective for human end-users than their non-explainable âblack boxâ counterparts.
However, research has repeatedly shown that explainability methods are often never tested with actual end-users (Wells and Bednarz, 2021; Nauta et al., 2023; Miller et al., 2017) â leading to mismatched user expectations, unused models, and diminished trust in AI (Koo et al., 2015; Algorithmia, 2021; Siu et al., 2021; Buçinca et al., 2021). A small survey conducted by Miller et al. (Miller et al., 2017) revealed that, from 23 papers published at the 2017 Explainable AI workshop at the International Joint Conference on Artificial Intelligence, only 4 articles referenced relevant social science research, only 1 of them built a model on this research, and none of the 23 included any empirical testing. Wells and Bednarzâs (Wells and Bednarz, 2021) found that 17 out of the 25 papers they surveyed on explainable AI for reinforcement learning did not include any user testing. Similarly, Nauta et al. (Nauta et al., 2023) surveyed 300 papers published at top AI/ML conferences and found that 1 in 3 papers evaluated their methodology exclusively with anecdotal evidence, and only 1 in 5 papers evaluated with actual users.
Our recurrent observation of explainability claims that lack human evidence motivated this late-breaking work. We sought to understand precisely how many papers across all XAI literature are promoting explainable, trustworthy, transparent, etc. methods without empirical testing. While there are many surveys of the XAI literature (Saeed and Omlin, 2023; Wells and Bednarz, 2021), what they tend to focus on is questions about what methods the field is using for explainability, what metrics or quantitative experiments are being carried out, or in some cases, what the state of the empirical testing is like. Furthermore, Miller et al.âs 2017 survey (Miller et al., 2017) â while distinct from these surveys â was self-described as a âlight scan of the literature,â only reviewing 23 papers. Similarly, Wells and Bednarzâs (Wells and Bednarz, 2021) reviewed 25 papers, while Nauta et al. (Nauta et al., 2023) reviewed 300.
We build on these related works by significantly expanding the scope of reviewed papers to include all relevant XAI literature and preprints in the Scopus database (Burnham, 2006). We collaborated with a professional librarian who helped us gather over 18,000 XAI papers and filter them based on claims about human explainability. The research question we aimed to answer was, âOf all the XAI papers that claim human explainability, how many validate those claims with empirical evidence?â Shockingly, we found that only 0.7% of XAI papers in the last decade test their methods with humans evaluating the explanations. Based on our literature review, the field is nearly entirely devoid of empirical evidence about its central claims of human explainability. For the remainder of the paper, we outline our methodology (which includes our exact Scopus database search for others to reproduce our work), the results of our literature survey, and a call to action for the broader community.
2. Methods
Table 1. Keyword search for gathering relevant literature using the Scopus database. The left column describes the high-level search strategy in natural language. The middle column shows the keyword search for scraping all XAI-related papers (e.g., papers that include terms related to âexplainabilityâ, âinterpretabilityâ, âXAIâ, etc.). The last column scrapes the same set of XAI papers but adds constraints related to human end-user testing (e.g., papers including âhuman-subjectâ, âend-userâ, âexperimentâ, âfeedbackâ, âinterviewâ, and so on).
| XAI papers AND Human end-user testing AND English | TITLE-ABS(((explainab* OR understandab* OR âinterpretab*â OR transparen*) W/1 (âartificial intelligenceâ OR âAIâ OR âmachine learningâ OR âMLâ)) OR XAI) AND LANGUAGE (english) | TITLE-ABS(((explainab* OR understandab* OR âinterpretab*â OR transparen*) W/1 (âartificial intelligenceâ OR âAIâ OR âmachine learningâ OR âMLâ)) OR XAI) AND TITLE-ABS((human OR âhuman-subjectâ OR user OR âend-userâ) Pre/1 (experiment* OR test OR trial OR evaluat* OR validat* OR feedback OR interview)) AND LANGUAGE (english) |
| --- | --- | --- |
| NOT Review papers | AND NOT (TITLE(review OR survey OR overview) OR DOCTYPE(cr OR re)) | AND NOT (TITLE(review OR survey OR overview) OR DOCTYPE(cr OR re)) |
We worked with a professional librarian over the course of three months to refine and generate advanced searches that could identify all existing XAI literature, as well as XAI literature with human-subject testing. This process involved iterating over relevant XAI search criteria (Miller et al., 2017), reviewing the results of the keyword search, and refining the keywords to include or exclude relevant or irrelevant papers. This process was repeated multiple times.
For the literature search, we selected the Scopus database for its broad coverage and ability to include the published and preprint literature. We wanted to include preprints in our search as much of the AI field publishes their results on arXiv, OSF, or another preprint platform due to the speed at which the field develops. However, we did not include review articles, workshop papers, or non-English papers. We discuss this limitation further in Section 5.
A total of 18,254 XAI papers were identified (13,839 published papers, 4,394 preprints), we refer to this set of papers as All XAI papers. We then filtered to a smaller subset of 253 papers which contained terms consistent with human subjects participating in testing, feedback, validation, interviews, and so on. This subset is referred to as papers with Claims about humans for the remainder of this work. It is important to note that papers that fall into the âclaims about humansâ category are papers that contain specific terms like âhuman feedback,â âend-user evaluation,â âuser interview,â and so on â see the right column in Table 1. We provide the full keyword search for anyone that would like to reproduce or refine our same literature review.
2.1. Scoring Criteria
The final subset of papers was manually reviewed by the authors. We use the validation criteria from Miller et al. (Miller et al., 2017) and give each paper in review (i.e., papers in the âClaims about humansâ category) one of two scores: 1 when the paper itself validates explainability claims with humans, and 0 otherwise. In the future we plan to code the broader set of XAI papers using Miller et al.âs data-driven criteria, which scores a paper based on whether it cites or implements work from the social sciences. However, we note that Miller et al.âs data-driven criteria â which was determined in 2017 â may be outdated. We discuss further in Section 5. For now, we limit our scope to only the use of humans in a behavioral study to evaluate the XAI method (i.e., the validation criteria) for simplicity.
Each article in the final review set was scored by 3 people in total with XAI backgrounds. To assign a validation score, we checked if any human experiments were conducted to test the proposed XAI method. Disagreements in scoring were broken collaboratively.
We observed several reasons to assign a 0 in our validation scoring. They include:
- The paper claims it is explainable-by-design, so it does not require human experiments.
- The paper contributes a framework or evaluation method that claims to be explainable but does not test the framework or evaluation method with humans.
- The paper claims human explainability because it used data derived from humans in it its training of the AI model or XAI method but does not evaluate with humans. We discuss this further in Section 4.2.
- The paper conducted a quantitative study comparing to human-annotated data or human labels (e.g., as a baseline), but does not provide human validation.
Finally, we excluded 16 papers from the final set that were agreed upon as off topic. Off topic papers did not present an XAI method but rather were meta-reviews of the field of XAI, philosophical arguments about explainability, or were summaries of several papers by particular graduate students.
3. Results
Table 2. Results of scoring 253 explainable AI papers that contained keywords consistent with humans participating in testing, providing feedback for, etc. an XAI technique or system (see Table 1). Our validation criteria follows Miller et al.âs (Miller et al., 2017), where papers receive a â0â if they did not validate with humans and a â1â if they did.
| Validation | 109 | 128 | 16 |
| --- | --- | --- | --- |
In total we scored 237 XAI papers that were both on topic and had terms consistent with claims about human explainability. The breakdown of our scores is shown in Table 2. From the 237 papers, the difference between those that actually conduct a human evaluation and those that do not is very small (54% of papers provide an evaluation, 46% do not). If you consider this number against the larger body of explainable AI work, it implies only 0.7% of XAI papers provide evidence from a human evaluation. Not only does this mean that nearly half of the XAI papers that directly claim human explainability never test this claim with humans, but that for every 1,000 XAI papers that do not explicitly mention humans, fewer than 8 of them provide explainability validation with empirical evidence.
The distribution of our down-selection results are shown in Figure 1a, with a zoomed in version of the last three bars shown in Figure 1b â note the change in the x-axes. âAll XAI Papersâ is the entirety of XAI literature that came back from our Scopus search (Table 1, middle column), âClaims about humansâ is the subset of those XAI papers that use keywords related to human-subjects, end-users experiments, tests, trials, etc. (Table 1, right column), âOn topicâ is a smaller subset of those papers that we deemed on topic (were not meta-reviews of the literature, survey papers, etc.), and âValidatedâ is the smallest subset of those papers that actually provided empirical evidence of human explainability. The same data is presented in Table 3, where each row is a subset of the one above, corresponding to the previously described search and review refinements.
<details>
<summary>extracted/6278551/figures/lit_review_percent.png Details</summary>
### Visual Description
## Funnel Chart: XAI Paper Validation
### Overview
The image is a funnel chart illustrating the filtering process of XAI (Explainable AI) papers, starting from all XAI papers and narrowing down to validated papers based on specific criteria. The chart visually represents the reduction in the number of papers at each filtering stage.
### Components/Axes
* **Y-axis Labels (Top to Bottom):**
* All XAI papers
* Filter for claims of human explainability
* Claims about humans
* Filter for papers we deem on topic
* On topic
* Filter for evidence of explainability
* Validated
* **X-axis Scale:** 0% to 100%, with increments of 10%.
* **Data Series:** The chart displays a single data series represented by horizontal bars. The bars are colored in a dark teal.
* **Flow Direction:** Vertical, indicated by downward-pointing arrows between the stages.
### Detailed Analysis
The funnel chart shows the progressive reduction in the number of papers as they pass through each filter.
* **All XAI papers:** The initial bar extends to approximately 100% on the x-axis.
* **Claims about humans:** After filtering for claims about humans, the bar reduces to approximately 5%.
* **On topic:** After filtering for papers deemed on topic, the bar reduces to approximately 2%.
* **Validated:** After filtering for evidence of explainability, the bar reduces to approximately 1%.
### Key Observations
* There is a significant drop in the number of papers after each filtering stage, particularly between "All XAI papers" and "Claims about humans."
* The final "Validated" category represents a very small fraction of the initial "All XAI papers" category.
### Interpretation
The funnel chart demonstrates that only a small percentage of XAI papers meet the criteria of making claims about humans, being deemed on topic, and providing evidence of explainability. This suggests that many XAI papers may not directly address human-centered explainability or may lack sufficient evidence to support their claims. The large drop-off at the initial filter ("Claims about humans") indicates that a significant portion of XAI research does not explicitly focus on human explainability. The subsequent filters further narrow down the pool, highlighting the stringent requirements for validation in this context.
</details>
(a) Comparison of all XAI literature we reviewed.
<details>
<summary>extracted/6278551/figures/lit_review_zoomed.png Details</summary>
### Visual Description
## Horizontal Bar Chart: Claims Validation Process
### Overview
The image is a horizontal bar chart illustrating a filtering process applied to claims about humans. The chart shows the proportion of claims remaining after each filtering step. The filtering steps are: initial claims, claims deemed "on topic," and finally, claims validated for evidence of explainability.
### Components/Axes
* **Y-axis Labels (Top to Bottom):**
* Claims about humans
* On topic
* Validated
* **Filtering Steps (Top to Bottom):**
* Filter for papers we deem on topic
* Filter for evidence of explainability
* **X-axis Title:** (Implied) Proportion of Claims
* **X-axis Scale:** 0.0% to 2.0%, with increments of 0.5%
### Detailed Analysis
The chart presents three horizontal bars, each representing a stage in the filtering process. All bars are the same color: a dark teal.
* **Claims about humans:** The bar extends to approximately 1.4% on the x-axis.
* **On topic:** The bar extends to approximately 1.3% on the x-axis.
* **Validated:** The bar extends to approximately 0.6% on the x-axis.
### Key Observations
* The proportion of claims decreases with each filtering step.
* The largest drop occurs between "On topic" and "Validated," indicating that a significant portion of claims deemed on-topic lack sufficient evidence of explainability.
### Interpretation
The chart demonstrates a funnel effect, where the initial pool of claims about humans is significantly reduced after applying filters for relevance and evidence of explainability. The data suggests that while a substantial number of claims are initially made, only a small fraction are both relevant and supported by explainable evidence. This highlights the importance of rigorous validation processes in research and analysis. The drop between "On topic" and "Validated" suggests that explainability is a significant bottleneck in the validation process.
</details>
(b) Zoomed in comparison of filtered XAI papers.
Figure 1. Distribution comparing all XAI literature from our Scopus search (Table 1), including those we scored. âAll XAI papersâ is the superset that encompasses all papers with keywords related to explainability, interpretability, etc. âClaims about humansâ papers are a subset of those that were filtered based on keywords related to human explainability. âOn topicâ papers are a subset of those that we filtered to exclude meta reviews, surveys, etc. âValidatedâ papers are a subset of those that provided empirical evidence of explainability. On the right a zoomed in version of the last three bars from the left figure are shown, note the change in x-axes.
\Description
Two subfigures depicting bar charts side by side. The first subfigure on the left shows a bar that represents 100% of the x-axis, where the x-axis ranges from 0% and 100%. The bottom three bars below the first bar are extremely small in comparison. The right subfigure shows the same bottom three bars but with a smaller x-axis from 0% to 2%.
Table 3. Table representation of our down-selection results. Our down-selection criteria is described in Section 3 and partially illustrated by keyword search in Table 1.
| Has explainability-related keywords | 18,254 | 100% |
| --- | --- | --- |
| Has claims about human explanability | 253 | 1.39% |
| Were on-topic (not surveys, etc.) | 237 | 1.30% |
| Were validated on human subjects | 128 | 0.70% |
If we consider the inverse of the percentages above, we see that over 98% of XAI papers do not appear with human evaluation keywords, and over 99% of XAI papers do not contain empirical human evaluation. There is naturally some difficulty in being precise with inclusion or exclusion criteria, but the results are nonetheless striking. Even if our search criteria were dramatically incorrect, say, an order of magnitude off, a ten-fold increase in validation rates in the literature would only result in 7% of works being tested with humans. We discuss refining our search criteria in the future in Section 5.
Since our work is inspired by Miller et al.âs (Miller et al., 2017), we provide a few relevant insights. Of the 253 papers in our search, only 11 cited Miller et al. Of those 11, 6 provided a human evaluation and 5 did not, roughly matching the overall rates of human evaluation in the literature. Thus, even the papers that point to a need for such evaluations do not perform them. That said, in general, the number of citations for Miller et al. has increased since publication (roughly 15 additional citations between 2018-2019 and 100 additional citations between 2023-2024).
4. Discussion of Results
4.1. Studies with Humans
<details>
<summary>x1.png Details</summary>
### Visual Description
## Bar Chart: Distribution of Human Subjects in Validated XAI Papers
### Overview
The image is a bar chart showing the distribution of the number of human subjects used in validated Explainable AI (XAI) papers. The x-axis represents the range of the number of subjects, and the y-axis represents the number of papers falling into each range.
### Components/Axes
* **Title:** Distribution of Human Subjects in Validated XAI Papers
* **X-axis:** Range of number of human subjects in intervals. The intervals are:
* \[2, 152]
* (152, 302]
* (302, 452]
* (452, 602]
* (602, 752]
* (752, 902]
* (902, 1052]
* (1052, 1202]
* (1202, 1352]
* (1352, 1502]
* (1502, 1652]
* (1652, 1802]
* (1802, 1952]
* (1952, 2102]
* (2102, 2252]
* (2252, 2402]
* (2402, 2552]
* (2552, 2702]
* (2702, 2852]
* (2852, 3002]
* (3002, 3152]
* (3152, 3302]
* **Y-axis:** Number of papers. The scale ranges from 0 to 100, with increments of 10.
### Detailed Analysis
The bar chart shows a highly skewed distribution.
* **\[2, 152]:** 91 papers
* **(152, 302]:** 16 papers
* **(302, 452]:** 3 papers
* **(452, 602]:** 1 paper
* **(602, 752]:** 0 papers
* **(752, 902]:** 1 paper
* **(902, 1052]:** 0 papers
* **(1052, 1202]:** 0 papers
* **(1202, 1352]:** 0 papers
* **(1352, 1502]:** 0 papers
* **(1502, 1652]:** 0 papers
* **(1652, 1802]:** 0 papers
* **(1802, 1952]:** 0 papers
* **(1952, 2102]:** 1 paper
* **(2102, 2252]:** 0 papers
* **(2252, 2402]:** 0 papers
* **(2402, 2552]:** 0 papers
* **(2552, 2702]:** 0 papers
* **(2702, 2852]:** 0 papers
* **(2852, 3002]:** 0 papers
* **(3002, 3152]:** 0 papers
* **(3152, 3302]:** 1 paper
### Key Observations
* The vast majority of validated XAI papers use a small number of human subjects (between 2 and 152).
* The number of papers decreases rapidly as the number of subjects increases.
* Only a few papers use a large number of human subjects (over 452).
### Interpretation
The data suggests that most XAI validation studies rely on relatively small sample sizes of human subjects. This could be due to various factors, such as the cost and difficulty of recruiting participants, the complexity of the experiments, or the specific research questions being addressed. The skewness of the distribution highlights the need for careful consideration of sample size and statistical power when designing and interpreting XAI validation studies. The presence of a few studies with very large numbers of subjects suggests that some research questions may require larger-scale data collection efforts.
</details>
(a) Distribution of subjects across XAI papers that validated with a human evaluation.
<details>
<summary>x2.png Details</summary>
### Visual Description
## Bar Chart: Distribution of Human Subjects in Validated XAI Papers
### Overview
The image is a bar chart illustrating the distribution of the number of human subjects used in validated XAI (Explainable AI) papers. The x-axis represents the range of the number of subjects, and the y-axis represents the number of papers falling within each subject range. The total number of papers analyzed is N = 2,152.
### Components/Axes
* **Title:** Distribution of Human Subjects in Validated XAI Papers (N=[2,152])
* **X-axis:** Number of Human Subjects (binned into ranges)
* \[2, 7]: 11
* (7, 12]: 4
* (12, 17): 7
* (17, 22]: 8
* (22, 27]: 8
* (27, 32]: 5
* (32, 37]: 4
* (37, 42): 4
* (42, 47): 1
* (47, 52]: 6
* (52, 57]: 3
* (57, 62): 5
* (62, 67): 2
* (67, 72]: 3
* (72, 77]: 1
* (77, 82]: 0
* (82, 87]: 2
* (87, 92]: 0
* (92, 97]: 0
* (97, 102): 2
* (102, 107]: 1
* (107, 112]: 0
* (112, 117]: 2
* (117, 122]: 5
* (122, 127): 0
* (127, 132]: 0
* (132, 137]: 2
* (137, 142]: 0
* (142, 147]: 3
* (147, 152]: 2
* **Y-axis:** Number of Papers (ranging from 0 to 12)
* **Bar Color:** Blue
### Detailed Analysis
The bar chart shows the frequency of papers using a specific range of human subjects. The highest frequency is for papers using between 2 and 7 subjects (11 papers). The number of papers decreases as the number of subjects increases, with some ranges having no papers at all.
### Key Observations
* The most common range of human subjects used in validated XAI papers is between 2 and 7.
* There is a general trend of decreasing frequency as the number of subjects increases.
* Several ranges of subject counts have no corresponding papers (e.g., 77-82, 87-92, 92-97, 107-112, 122-127, 127-132, 137-142).
* There is a slight increase in the number of papers using a larger number of subjects in the range of 117-122.
### Interpretation
The data suggests that most validated XAI papers tend to use a relatively small number of human subjects. This could be due to various factors, such as the complexity of the experiments, the availability of subjects, or the specific research questions being addressed. The decreasing trend indicates that studies involving larger numbers of human subjects are less common in the field of validated XAI. The peaks at certain ranges (e.g., 117-122) might indicate specific types of studies or experimental designs that require a particular number of participants. The gaps in the distribution could be due to the nature of the research or the specific criteria used for validation.
</details>
(b) Subset of distribution counts, up to N=152 and bin size=5.
Figure 2. Distribution of counts for human subjects involved in an evaluation, experiment, interview, etc. for validated XAI papers. It is important to note that 13 papers did not report their human subject count and 3 papers approximated their count. In the bottom figure, a subset of distribution counts up to N=152 and a bin size of 5.
\Description
Two subfigures depicting histograms, one on the top and one on the bottom. The first histogram shows the distribution of participant counts for the entire range of data, while the bottom histogram represents a zoomed in view from participant count 0 to 152.
We find that out of the 237 papers that claimed some form of human explainability, only 56% of them provided an evaluation with humans. Of course, this number drops down to 0.7% when compared to the larger body of XAI work in which empirical evidence is not explicitly mentioned as a keyword in the paper, but for which explainability is a central topic and often a central claim of a new AI technique. Even if we extrapolate the fraction of off topic papers in the review set to the entirety of the XAI set, and exclude the same fraction from the overall number of papers under consideration, the percentage of papers with human evaluations as a part of this new âon-topicâ set is still approximately 0.7%.
One point worth discussing is the quality of the empirical evaluations in those 128 human-validated papers. In some cases, the participants in the evaluation were never directly discussed. Seen in Figure 2b, 13 of the papers that involved a validation did not report their human subject count and 3 papers reported an approximate count. Furthermore, in many of the papers, the authors do not disclose the subjectsâ expertise, workplace, education, or relationship to the authors. We found ourselves asking: who are the evaluation subjects? If we know nothing about these usersâ backgrounds, or potential motivations to call an XAI technique âgood,â one might argue this introduces an unfair bias into the validation.
Furthermore, many of the papers recruited college students as evaluation participants. Prior research has questioned the validity of such experiments (Wells and Bednarz, 2021) when the end-users of these XAI techniques are intended to be cybersecurity experts, pilots, doctors, and so on. Of course, it can be difficult to assemble domain experts to test methods and tools, especially domain experts who you are not working on the tool with. Perhaps that is why we observed a number of papers only having two or three domain experts as part of the evaluation (Figure 2b). Regardless, it is difficult to trust generalizable claims about explainability when the participant pool is very small or dissimilar to the target users.
Another observation was the type of evaluation conducted. Many of the papersâ evaluation criteria consisted of: âDo you think this is explainable â yes or no?â or âWhich is more explainable: our explainable method, or this other baseline explanation that is clearly worse (e.g., contains grammatical errors or has logical flaws)?â It is important that evaluation scenarios are crafted realistically to their intended target use case. Many questions related to the risks, potential failures, or edge-cases remain unanswered when explainable AI techniques are not properly studied with the correct end-user group nor represent a realistic evaluation scenario.
4.2. Studies without Humans
The studies that propose new XAI techniques but lack human testing almost universally show results of algorithmic performance, and at times a few exemplar outputs. While these elements are important and may provide circumstantial evidence, they are insufficient if a central claim is that a method is âexplainable.â If we think about the production of a possibly-explainable artifact as a signal being sent, it is straightforward for us to realize that it takes confirmation that the signal was received and understood correctly to confirm a claim of explainability. Otherwise, there is no difference between âexplainableâ AI and any other kind of AI, and the many examples of possibly-explainable methods are simply sending signals into a void.
We previously noted that the field still tends to use terms such as âexplainable,â âinterpretable,â âtransparent,â and so on, interchangeably (Lipton, 2018). Rudin (Rudin, 2019) makes the distinction between âexplainableâ as a post-hoc treatment of an otherwise black-box model, âinterpretableâ stemming from model design that is grounded in expert domain understanding. A small number of the papers that received a validation score of 0 did develop their models with the help of domain experts, thereby meeting Rudinâs âinterpretableâ definition, but did not perform any formal evaluation with such experts â neither with the group that helped develop them or with a broader group. Even in these cases, it remains an important step that the evaluation of human understandingâbeyond the typical one or two domain experts who helped develop these modelsâis included. While it is more likely that signals crafted with the help of domain experts are meaningful to their intended audience, a validation that such signals were received and understood is still necessary, as previous work has shown that it is not always the case. For example, domain experts can be vastly overconfident in their understanding of XAI methods that they helped craft (Siu et al., 2023).
For the many other papers with a validation score of 0, there is an often implied or at times explicit claim that an explanatory method is âintuitively obviousâ due to the inclusion of some property such as the use of natural language or conciseness of explanation. While these properties may seem self-evident, they are neither inherently clear nor universally effective, as even brief texts in oneâs native language can often be confusing or indecipherable. These properties are perhaps useful, but not inherently sufficient or necessarily effective (Buçinca et al., 2021).
5. Future Work
5.1. Beyond this Late-Breaking Work
This late-breaking work intends to shed light on the lackluster empirical evidence in the field of explainable AI. However, we consider this research to be a âprequelâ to future work that will require refining our literature search, Miller et al.âs scoring criteria (Miller et al., 2017), and our analysis of the resultant literature.
Generalizability. There are multiple opportunities to build upon our XAI literature review to improve the generalizability of our findings. Future work should examine identifying and analyzing literature not included in our search criteria: review articles, workshop papers, and Non-English papers. Including these works in an analysis would not only help the community gain a deeper understanding of empirical testing in XAI, but would also mitigate the inherent bias of studying English-only works. Our literature search could also be refined to distinguish different types of XAI research, resulting in a potentially smaller set of âAll XAI papersâ that are deemed more on-topic. For example, there are cases of XAI work that do not formally evaluate with humans due to their focus on the use of explanations for scientific discovery, or to understand the mechanisms by which some system works. These are typically works in the sciences, and it is true that broader evaluation of human-interpretability is likely unnecessary here if they directly present what principles have been discovered, or in many cases rediscovered, since those principles can be checked directly by the relevant domain experts. We posit these cases form a small minority of the work in the field; however, we would like to consider additional cases that should not be included in our broadest literature search.
Scoring criteria. Another important direction of research would be to rework Miller et al.âs scoring criteria (Miller et al., 2017) from 2017, specifically the data-driven criteria. Miller et al. rewarded XAI papers with the highest score of a 2 if the paper: (1) explicitly referenced a work in the social sciences or in a computer science venue, so long as the paper was not centrally about AI; and (2) incorporated a method from the social sciences in the paperâs contributed XAI method. We find that the data-driven criteria is somewhat unclear, and it is difficult to understand which computer science papers âcountâ versus âdo not count.â Should XAI papers that cite other XAI papersâwhich do include empirical evidenceâreceive a higher score? What if a new XAI method incorporates the results of a prior XAI paper but does not test this new method with human subjects? The validation score we used may also need to be reworked, as the sole presence of an evaluation may not be equivalent to a paper that presents a âgoodâ evaluation (as discussed in Section 3, some evaluations did not report a participant count or participant expertise).
Comprehensive analysis. It is imperative that a deeper analysis is conducted of our final set of âon-topicâ XAI papers. We briefly touched on the number of human subjects across all validated papers (Figure 2b), but some questions remain. For example, what is the distribution for the types of subjectsâe.g., experts versus students? Further, what we can learn from the papers that do and do not validate their claims with human studies, but still claim interpretability? What else do these papers have in common, and how else do they differ? Are there shared outcomes we can learn in testing explainability without human subjects? Moreover, do papers that validate with human studies tend to be published at a human-centered conference, e.g., ACM CHI, and not at an AI-centered conference, e.g., NeurIPS? The answers to these questions can help identify best practices in XAI testing, implications for AI design, and ultimately pave the way for rigorous human-centered evaluation frameworks. If other researchers would like to tackle these questions, we provide the exact keyword search used for our literature search in Table 1, which can be employed to replicate our findings.
5.2. Call to Action
Like claims of AI accuracy, claims of AI explainability must rest on empirical evidence. Contemporary artificial intelligence is an empirical discipline that rests on evidence of accuracy. However, our results show that work in contemporary explainable artificial intelligence almost never provides empirical evidence of its own.
While general heuristics such as conciseness or the use of domain-specific features are a reasonable place to start, we must not only rely on these heuristics. Empirical human testing is necessarily the gold standard for explainability claims. A pharmaceutical company would never distribute a newly developed drug without extensive clinical trials, even if they truly believed in its efficacy from biological principles, evidence collected in simulation, mouse trials, and so on. Why can XAI models, which may impact high-risk decision-making, be released untested with their human users?
Common objections to empirical evaluations are that humans are biased, or it is difficult to assemble a representative sample of humans. We note that the same can be said of every commonly-used machine learning benchmark, from ImageNet (Deng et al., 2009), to IMBD movie reviews (Maas et al., 2011), to medical question-and-answer (Jin et al., 2021). All contain human biases from their generation, curation, cultural context, or other factors. They may or may not match the intended application of the system being tested and took extensive time to collect. Nonetheless, they are used. The issue of limitations in the human testing population in interpretability testing is analogous to limitations in the dataset being used in accuracy testing and is not an unknown to the AI community.
The methods to test XAI clearly exist (Doshi-Velez and Kim, 2017) âour literature review shows that there are researchers using them and publishing about them. Undoubtedly there are challenges in moving from testing AI solely with static datasets into testing with humans, but they are not insurmountable. It is crucial that the XAI community take a critical look at the state of the field and determine whether the current trajectory and (lack of) standards is acceptable or not. As researchers, it seems unacceptable to produce copious volumes of âexplainableâ methodologies that have never been tested with the intended end-users. As reviewers, we are responsible for pushing back on research that claims explainability but provides little to no evidence. We hope the findings presented in this work challenge each otherâs viewpoints.
6. Conclusion
This late-breaking work highlights a critical gap in explainable AI (XAI) research: the overwhelming majority of papers that claim explainability fail to provide empirical evidence through human studies. By conducting a comprehensive literature search of 18,254 papers in collaboration with a professional librarian, we identified 253 papers that explicitly referenced human-related terms, such as human studies, user experiments, or end-user feedback. Among these, only 128 included some form of a human evaluations, revealing that 0.7% of XAI literature empirically validates claims of explainability. These findings highlight an urgent need for greater rigor in XAI research, particularly in substantiating claims about human interpretability and usability. Without empirical evidence, it is impossible to assess whether explainability methods truly address the needs of end-users. To foster transparency and reproducibility, we share our methodology and keyword search criteria, encouraging researchers to build on our work and prioritize human-centered evaluation in future explainable AI work.
Acknowledgements. DISTRIBUTION STATEMENT A. Approved for public release. Distribution is unlimited. This material is based upon work supported by the Under Secretary of Defense for Research and Engineering under Air Force Contract No. FA8702-15-D-0001. Any opinions, findings, conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the Under Secretary of Defense for Research and Engineering.
References
- (1)
- Algorithmia (2021) Algorithmia. 2021. 2021 Enterprise Trends in Machine Learning. https://info.algorithmia.com/hubfs/2020/Reports/2021-Trends-in-ML/Algorithmia_2021_enterprise_ML_trends.pdf
- Board (2019) Defense Innovation Board. 2019. AI principles: recommendations on the ethical use of artificial intelligence by the department of defense: supporting document.
- Borenstein and Howard (2021) Jason Borenstein and Ayanna Howard. 2021. Emerging challenges in AI and the need for AI ethics education. AI and Ethics 1 (2021), 61â65.
- Buçinca et al. (2021) Zana Buçinca, Maja Barbara Malaya, and Krzysztof Z Gajos. 2021. To trust or to think: cognitive forcing functions can reduce overreliance on AI in AI-assisted decision-making. Proceedings of the ACM on Human-computer Interaction 5, CSCW1 (2021), 1â21.
- Burnham (2006) Judy F Burnham. 2006. Scopus database: a review. Biomedical digital libraries 3 (2006), 1â8.
- Cath (2018) Corinne Cath. 2018. Governing artificial intelligence: ethical, legal and technical opportunities and challenges. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 376, 2133 (2018), 20180080.
- Dash et al. (2019) Rupa Dash, Mark McMurtrey, Carl Rebman, and Upendra K. Kar. 2019. Application of Artificial Intelligence in Automation of Supply Chain Management. Journal of Strategic Innovation and Sustainability 14, 3 (Jul. 2019). doi: 10.33423/jsis.v14i3.2105
- Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. ImageNet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, âMiami, FL, USAâ, 248â255. doi: 10.1109/CVPR.2009.5206848
- Doshi-Velez and Kim (2017) Finale Doshi-Velez and Been Kim. 2017. Towards A Rigorous Science of Interpretable Machine Learning. http://arxiv.org/abs/1702.08608 cite arxiv:1702.08608.
- Edwards (2021) Lilian Edwards. 2021. The EU AI Act: a summary of its significance and scope.
- Holzinger et al. (2017) Andreas Holzinger, Chris Biemann, Constantinos S. Pattichis, and Douglas B. Kell. 2017. What do we need to build explainable AI systems for the medical domain? arXiv:1712.09923 [cs.AI] https://arxiv.org/abs/1712.09923
- Jin et al. (2021) Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2021. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences 11, 14 (2021), 6421.
- Koo et al. (2015) Jeamin Koo, Jungsuk Kwac, Wendy Ju, Martin Steinert, Larry Leifer, and Clifford Nass. 2015. Why did my car just do that? Explaining semi-autonomous driving actions to improve driver understanding, trust, and performance. International Journal on Interactive Design and Manufacturing (IJIDeM) 9 (2015), 269â275.
- Lipton (2018) Zachary C Lipton. 2018. The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery. Queue 16, 3 (2018), 31â57.
- Ly et al. (2017) Kien Hoa Ly, Ann-Marie Ly, and Gerhard Andersson. 2017. A fully automated conversational agent for promoting mental well-being: a pilot RCT using mixed methods. Internet interventions 10 (2017), 39â46.
- Maas et al. (2011) Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies - Volume 1 (Portland, Oregon) (HLT â11). Association for Computational Linguistics, USA, 142â150.
- Miller et al. (2017) Tim Miller, Piers Howe, and Liz Sonenberg. 2017. Explainable AI: Beware of Inmates Running the Asylum Or: How I Learnt to Stop Worrying and Love the Social and Behavioural Sciences. arXiv:1712.00547 [cs.AI] https://arxiv.org/abs/1712.00547
- Nauta et al. (2023) Meike Nauta, Jan Trienes, Shreyasi Pathak, Elisa Nguyen, Michelle Peters, Yasmin Schmitt, Jörg Schlötterer, Maurice Van Keulen, and Christin Seifert. 2023. From anecdotal evidence to quantitative evaluation methods: A systematic review on evaluating explainable ai. Comput. Surveys 55, 13s (2023), 1â42.
- Rolnick et al. (2022) David Rolnick, Priya L Donti, Lynn H Kaack, Kelly Kochanski, Alexandre Lacoste, Kris Sankaran, Andrew Slavin Ross, Nikola Milojevic-Dupont, Natasha Jaques, Anna Waldman-Brown, et al. 2022. Tackling climate change with machine learning. ACM Computing Surveys (CSUR) 55, 2 (2022), 1â96.
- Rudin (2019) Cynthia Rudin. 2019. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nat Mach Intell 1 (5): 206â215.
- Rudin and Radin (2019) Cynthia Rudin and Joanna Radin. 2019. Why are we using black box models in AI when we donât need to? A lesson from an explainable AI competition. Harvard Data Science Review 1, 2 (2019), 1â9.
- Saeed and Omlin (2023) Waddah Saeed and Christian Omlin. 2023. Explainable AI (XAI): A systematic meta-survey of current challenges and future opportunities. Knowledge-Based Systems 263 (2023), 110273.
- Siu et al. (2023) Ho Chit Siu, Kevin Leahy, and Makai Mann. 2023. STL: Surprisingly Tricky Logic (for System Validation). In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, âDetroit, MI, USAâ, 8613â8620. doi: 10.1109/IROS55552.2023.10342290
- Siu et al. (2021) Ho Chit Siu, Jaime Peña, Edenna Chen, Yutai Zhou, Victor Lopez, Kyle Palko, Kimberlee Chang, and Ross Allen. 2021. Evaluation of human-ai teams for learned and rule-based agents in hanabi. Advances in Neural Information Processing Systems 34 (2021), 16183â16195.
- Suh et al. (2024) Ashley Suh, Gabriel Appleby, Erik W. Anderson, Luca Finelli, Remco Chang, and Dylan Cashman. 2024. Are Metrics Enough? Guidelines for Communicating and Visualizing Predictive Models to Subject Matter Experts. IEEE Transactions on Visualization and Computer Graphics 30, 7 (2024), 4137â4153. doi: 10.1109/TVCG.2023.3259341
- Wells and Bednarz (2021) Lindsay Wells and Tomasz Bednarz. 2021. Explainable ai and reinforcement learningâa systematic review of current approaches and trends. Frontiers in artificial intelligence 4 (2021), 550030.