# Fewer Than 1% of Explainable AI Papers Validate Explainability with Humans
**Authors**: Ashley Suh, Isabelle Hurley, Nora Smith, Ho Chit Siu
> 0000-0001-6513-8447 MIT Lincoln Laboratory Lexington MA USA
> 0009-0006-4031-9018 MIT Lincoln Laboratory Lexington MA USA
> 0000-0001-5255-9226 MIT Lincoln Laboratory Lexington MA USA
> 0000-0003-3451-8046 MIT Lincoln Laboratory Lexington MA USA
(2025)
## Abstract
This late-breaking work presents a large-scale analysis of explainable AI (XAI) literature to evaluate claims of human explainability. We collaborated with a professional librarian to identify 18,254 papers containing keywords related to explainability and interpretability. Of these, we find that only 253 papers included terms suggesting human involvement in evaluating an XAI technique, and just 128 of those conducted some form of a human study. In other words, fewer than 1% of XAI papers (0.7%) provide empirical evidence of human explainability when compared to the broader body of XAI literature. Our findings underscore a critical gap between claims of human explainability and evidence-based validation, raising concerns about the rigor of XAI research. We call for increased emphasis on human evaluations in XAI studies and provide our literature search methodology to enable both reproducibility and further investigation into this widespread issue.
Explainable AI, Trustworthy AI, Interpretability, Human-Centered AI journalyear: 2025 copyright: rightsretained conference: Extended Abstracts of the CHI Conference on Human Factors in Computing Systems; April 26-May 1, 2025; Yokohama, Japan booktitle: Extended Abstracts of the CHI Conference on Human Factors in Computing Systems (CHI EA â25), April 26-May 1, 2025, Yokohama, Japan doi: 10.1145/3706599.3719964 isbn: 979-8-4007-1395-8/2025/04 ccs: Human-centered computing Empirical studies in HCI ccs: Computing methodologies Cognitive science ccs: Computing methodologies Philosophical/theoretical foundations of artificial intelligence
## 1. Introduction
The growing accessibility of artificial intelligence (AI) promises significant benefits for our society (Dash et al., 2019; Rolnick et al., 2022; Ly et al., 2017), as well as a number of critical challenges (Board, 2019; Cath, 2018; Borenstein and Howard, 2021). A pressing obstacle in the field is the lack of explainability, trustworthiness, and transparency in AI technologies (Rudin, 2019; Saeed and Omlin, 2023; Rudin and Radin, 2019; Holzinger et al., 2017). With AI systems being deployed to diagnose medical conditions, navigate vehicles on the road, and forecast natural disasters, understanding a modelâs behavior, recommendations, and reasoning has become imperative (Edwards, 2021). Yet, habitually used yet nondescript metrics like âaccuracyâ and âconfidenceâ fail to capture the whole story of an AI model nor how it will be interpreted by humans in real-world contexts (Suh et al., 2024).
The potential for skepticism and hesitation in trusting AI has driven the community to develop techniques in interpretable and explainable AI (XAI), concepts that are generally interchangeably used (Lipton, 2018), along with transparent and trustworthy AI While a precise definition for interpretability is not universal, Doshi-Velez and Kim define it as the ability to explain or present AI/ML systems in understandable terms to a human (Doshi-Velez and Kim, 2017). We refer to these methods collectively as research in explainable AI (XAI).. Common XAI approaches include rule-based reasoning, visualizations of deep learning processes, text-based explanations, and example-driven counterfactuals (Saeed and Omlin, 2023). Altogether, the developers of these explainability methods argue that AI models are now more usable, interpretable, and effective for human end-users than their non-explainable âblack boxâ counterparts.
However, research has repeatedly shown that explainability methods are often never tested with actual end-users (Wells and Bednarz, 2021; Nauta et al., 2023; Miller et al., 2017) â leading to mismatched user expectations, unused models, and diminished trust in AI (Koo et al., 2015; Algorithmia, 2021; Siu et al., 2021; Buçinca et al., 2021). A small survey conducted by Miller et al. (Miller et al., 2017) revealed that, from 23 papers published at the 2017 Explainable AI workshop at the International Joint Conference on Artificial Intelligence, only 4 articles referenced relevant social science research, only 1 of them built a model on this research, and none of the 23 included any empirical testing. Wells and Bednarzâs (Wells and Bednarz, 2021) found that 17 out of the 25 papers they surveyed on explainable AI for reinforcement learning did not include any user testing. Similarly, Nauta et al. (Nauta et al., 2023) surveyed 300 papers published at top AI/ML conferences and found that 1 in 3 papers evaluated their methodology exclusively with anecdotal evidence, and only 1 in 5 papers evaluated with actual users.
Our recurrent observation of explainability claims that lack human evidence motivated this late-breaking work. We sought to understand precisely how many papers across all XAI literature are promoting explainable, trustworthy, transparent, etc. methods without empirical testing. While there are many surveys of the XAI literature (Saeed and Omlin, 2023; Wells and Bednarz, 2021), what they tend to focus on is questions about what methods the field is using for explainability, what metrics or quantitative experiments are being carried out, or in some cases, what the state of the empirical testing is like. Furthermore, Miller et al.âs 2017 survey (Miller et al., 2017) â while distinct from these surveys â was self-described as a âlight scan of the literature,â only reviewing 23 papers. Similarly, Wells and Bednarzâs (Wells and Bednarz, 2021) reviewed 25 papers, while Nauta et al. (Nauta et al., 2023) reviewed 300.
We build on these related works by significantly expanding the scope of reviewed papers to include all relevant XAI literature and preprints in the Scopus database (Burnham, 2006). We collaborated with a professional librarian who helped us gather over 18,000 XAI papers and filter them based on claims about human explainability. The research question we aimed to answer was, âOf all the XAI papers that claim human explainability, how many validate those claims with empirical evidence?â Shockingly, we found that only 0.7% of XAI papers in the last decade test their methods with humans evaluating the explanations. Based on our literature review, the field is nearly entirely devoid of empirical evidence about its central claims of human explainability. For the remainder of the paper, we outline our methodology (which includes our exact Scopus database search for others to reproduce our work), the results of our literature survey, and a call to action for the broader community.
## 2. Methods
Table 1. Keyword search for gathering relevant literature using the Scopus database. The left column describes the high-level search strategy in natural language. The middle column shows the keyword search for scraping all XAI-related papers (e.g., papers that include terms related to âexplainabilityâ, âinterpretabilityâ, âXAIâ, etc.). The last column scrapes the same set of XAI papers but adds constraints related to human end-user testing (e.g., papers including âhuman-subjectâ, âend-userâ, âexperimentâ, âfeedbackâ, âinterviewâ, and so on).
| XAI papers AND Human end-user testing AND English | TITLE-ABS(((explainab* OR understandab* OR âinterpretab*â OR transparen*) W/1 (âartificial intelligenceâ OR âAIâ OR âmachine learningâ OR âMLâ)) OR XAI) AND LANGUAGE (english) | TITLE-ABS(((explainab* OR understandab* OR âinterpretab*â OR transparen*) W/1 (âartificial intelligenceâ OR âAIâ OR âmachine learningâ OR âMLâ)) OR XAI) AND TITLE-ABS((human OR âhuman-subjectâ OR user OR âend-userâ) Pre/1 (experiment* OR test OR trial OR evaluat* OR validat* OR feedback OR interview)) AND LANGUAGE (english) |
| --- | --- | --- |
| NOT Review papers | AND NOT (TITLE(review OR survey OR overview) OR DOCTYPE(cr OR re)) | AND NOT (TITLE(review OR survey OR overview) OR DOCTYPE(cr OR re)) |
We worked with a professional librarian over the course of three months to refine and generate advanced searches that could identify all existing XAI literature, as well as XAI literature with human-subject testing. This process involved iterating over relevant XAI search criteria (Miller et al., 2017), reviewing the results of the keyword search, and refining the keywords to include or exclude relevant or irrelevant papers. This process was repeated multiple times.
For the literature search, we selected the Scopus database for its broad coverage and ability to include the published and preprint literature. We wanted to include preprints in our search as much of the AI field publishes their results on arXiv, OSF, or another preprint platform due to the speed at which the field develops. However, we did not include review articles, workshop papers, or non-English papers. We discuss this limitation further in Section 5.
A total of 18,254 XAI papers were identified (13,839 published papers, 4,394 preprints), we refer to this set of papers as All XAI papers. We then filtered to a smaller subset of 253 papers which contained terms consistent with human subjects participating in testing, feedback, validation, interviews, and so on. This subset is referred to as papers with Claims about humans for the remainder of this work. It is important to note that papers that fall into the âclaims about humansâ category are papers that contain specific terms like âhuman feedback,â âend-user evaluation,â âuser interview,â and so on â see the right column in Table 1. We provide the full keyword search for anyone that would like to reproduce or refine our same literature review.
### 2.1. Scoring Criteria
The final subset of papers was manually reviewed by the authors. We use the validation criteria from Miller et al. (Miller et al., 2017) and give each paper in review (i.e., papers in the âClaims about humansâ category) one of two scores: 1 when the paper itself validates explainability claims with humans, and 0 otherwise. In the future we plan to code the broader set of XAI papers using Miller et al.âs data-driven criteria, which scores a paper based on whether it cites or implements work from the social sciences. However, we note that Miller et al.âs data-driven criteria â which was determined in 2017 â may be outdated. We discuss further in Section 5. For now, we limit our scope to only the use of humans in a behavioral study to evaluate the XAI method (i.e., the validation criteria) for simplicity.
Each article in the final review set was scored by 3 people in total with XAI backgrounds. To assign a validation score, we checked if any human experiments were conducted to test the proposed XAI method. Disagreements in scoring were broken collaboratively.
We observed several reasons to assign a 0 in our validation scoring. They include:
- The paper claims it is explainable-by-design, so it does not require human experiments.
- The paper contributes a framework or evaluation method that claims to be explainable but does not test the framework or evaluation method with humans.
- The paper claims human explainability because it used data derived from humans in it its training of the AI model or XAI method but does not evaluate with humans. We discuss this further in Section 4.2.
- The paper conducted a quantitative study comparing to human-annotated data or human labels (e.g., as a baseline), but does not provide human validation.
Finally, we excluded 16 papers from the final set that were agreed upon as off topic. Off topic papers did not present an XAI method but rather were meta-reviews of the field of XAI, philosophical arguments about explainability, or were summaries of several papers by particular graduate students.
## 3. Results
Table 2. Results of scoring 253 explainable AI papers that contained keywords consistent with humans participating in testing, providing feedback for, etc. an XAI technique or system (see Table 1). Our validation criteria follows Miller et al.âs (Miller et al., 2017), where papers receive a â0â if they did not validate with humans and a â1â if they did.
| Validation | 109 | 128 | 16 |
| --- | --- | --- | --- |
In total we scored 237 XAI papers that were both on topic and had terms consistent with claims about human explainability. The breakdown of our scores is shown in Table 2. From the 237 papers, the difference between those that actually conduct a human evaluation and those that do not is very small (54% of papers provide an evaluation, 46% do not). If you consider this number against the larger body of explainable AI work, it implies only 0.7% of XAI papers provide evidence from a human evaluation. Not only does this mean that nearly half of the XAI papers that directly claim human explainability never test this claim with humans, but that for every 1,000 XAI papers that do not explicitly mention humans, fewer than 8 of them provide explainability validation with empirical evidence.
The distribution of our down-selection results are shown in Figure 1a, with a zoomed in version of the last three bars shown in Figure 1b â note the change in the x-axes. âAll XAI Papersâ is the entirety of XAI literature that came back from our Scopus search (Table 1, middle column), âClaims about humansâ is the subset of those XAI papers that use keywords related to human-subjects, end-users experiments, tests, trials, etc. (Table 1, right column), âOn topicâ is a smaller subset of those papers that we deemed on topic (were not meta-reviews of the literature, survey papers, etc.), and âValidatedâ is the smallest subset of those papers that actually provided empirical evidence of human explainability. The same data is presented in Table 3, where each row is a subset of the one above, corresponding to the previously described search and review refinements.
<details>
<summary>extracted/6278551/figures/lit_review_percent.png Details</summary>
### Visual Description
## Horizontal Bar Chart: XAI Paper Filtering Funnel
### Overview
The image displays a horizontal bar chart combined with a funnel diagram on the left side. It illustrates a multi-stage filtering process applied to a corpus of "XAI papers," showing the dramatic reduction in the number of papers at each successive stage of evaluation. The chart visually emphasizes the scarcity of papers that meet rigorous criteria for human explainability.
### Components/Axes
* **Chart Type:** Horizontal bar chart with an integrated funnel/process diagram.
* **Y-Axis (Left Side - Funnel Labels):** The vertical axis lists the stages of the filtering process. From top to bottom:
1. **All XAI papers**
2. **Claims about humans**
3. **On topic**
4. **Validated**
* **Filter Descriptions (Between Labels):** Text annotations with downward-pointing arrows describe the filtering criteria applied between stages:
* Between "All XAI papers" and "Claims about humans": **"Filter for claims of human explainability"**
* Between "Claims about humans" and "On topic": **"Filter for papers we deem on topic"**
* Between "On topic" and "Validated": **"Filter for evidence of explainability"**
* **X-Axis (Bottom):** A percentage scale running from **0%** to **100%**, with major grid lines and labels at every 10% increment (0%, 10%, 20%, ..., 100%).
* **Data Series:** A single data series represented by dark teal horizontal bars. There is no legend, as all bars represent the same metric (percentage of the initial corpus).
* **Spatial Layout:** The funnel labels and filter text are positioned to the left of the main chart area. The bars originate from the left edge of the chart area and extend rightward. The x-axis labels are centered below the chart.
### Detailed Analysis
The chart quantifies the attrition of papers through the defined filtering pipeline. The length of each bar represents the percentage of the original "All XAI papers" corpus that remains at that stage.
1. **All XAI papers:** The bar extends fully to the **100%** mark. This is the baseline corpus.
2. **Claims about humans:** After filtering for papers that make claims about human explainability, the bar length drops precipitously. Visually, it extends to approximately **5%** of the original width (±1% uncertainty).
3. **On topic:** After a further filter for papers deemed "on topic," the bar length appears very similar to the previous stage, estimated at **~5%** (±1%). This suggests the "on topic" filter removed very few additional papers from the "Claims about humans" subset.
4. **Validated:** After the final filter for "evidence of explainability," the bar shrinks further. It is the shortest bar, estimated at approximately **1-2%** of the original corpus (±0.5% uncertainty).
**Trend Verification:** The visual trend is one of severe, stepwise reduction. The first filter causes a massive drop (from 100% to ~5%). The second filter shows minimal change. The third filter causes another noticeable, though smaller, relative reduction.
### Key Observations
* **Massive Initial Attrition:** The most significant finding is that only a tiny fraction (â5%) of all XAI papers even make claims about human explainability.
* **Stability in the Middle Stage:** The "On topic" filter does not substantially reduce the number of papers further, indicating that most papers making claims about human explainability are considered relevant by the evaluators.
* **Final Validation is Rare:** The process of finding papers with validated evidence of explainability is highly selective, yielding only about 1-2% of the original literature.
* **Visual Emphasis:** The funnel diagram on the left, with its bold labels and arrows, explicitly narrates the logical flow of the filtering process, which the bar chart then quantifies.
### Interpretation
This visualization presents a critical, investigative look at the state of research in Explainable AI (XAI). It suggests a significant gap between the volume of published work and the subset that provides robust, validated evidence for its claims about human explainability.
The data implies that the field may be saturated with papers that propose methods or make claims, but a vanishingly small proportion undergo the rigorous evaluation necessary to validate those claims. The near-plateau between "Claims about humans" and "On topic" indicates that relevance is not the primary bottleneck; rather, the core challenges are (1) the initial formulation of human-centric explainability claims and, more critically, (2) the subsequent generation of empirical evidence to support them.
This chart likely serves as a call to action for the research community to prioritize methodological rigor, empirical validation, and a focus on human-centric outcomes over the mere proliferation of new techniques. It highlights a potential credibility gap in the XAI literature.
</details>
(a) Comparison of all XAI literature we reviewed.
<details>
<summary>extracted/6278551/figures/lit_review_zoomed.png Details</summary>
### Visual Description
## Horizontal Bar Chart: Research Paper Filtering Funnel
### Overview
The image displays a horizontal bar chart illustrating a three-stage filtering process for research papers, likely within the field of AI or machine learning explainability. The chart quantifies the percentage of papers retained at each stage, showing a significant reduction from initial claims to validated evidence.
### Components/Axes
* **Chart Type:** Horizontal bar chart.
* **Y-Axis (Categories):** Three categorical stages are listed vertically on the left side of the chart. From top to bottom:
1. **Claims about humans**
2. **On topic**
3. **Validated**
* **X-Axis (Scale):** A percentage scale running horizontally along the bottom. The axis is labeled with markers at:
* `0.0%`
* `0.5%`
* `1.0%`
* `1.5%`
* `2.0%`
* **Data Series:** A single data series represented by dark teal-colored bars. There is no separate legend, as all bars belong to the same series.
* **Annotations:** Text and arrows are positioned to the left of the bars, explaining the filtering criteria between stages.
* Between "Claims about humans" and "On topic": An arrow points downward with the text: **"Filter for papers we deem on topic"**.
* Between "On topic" and "Validated": An arrow points downward with the text: **"Filter for evidence of explainability"**.
### Detailed Analysis
* **Stage 1: Claims about humans**
* **Bar Length:** The bar extends to approximately **1.4%** on the x-axis.
* **Trend:** This is the longest bar, representing the initial pool of papers containing claims about humans.
* **Stage 2: On topic**
* **Bar Length:** The bar extends to approximately **1.3%** on the x-axis.
* **Trend:** This bar is slightly shorter than the first, indicating a small reduction after applying the "on topic" filter.
* **Stage 3: Validated**
* **Bar Length:** The bar extends to approximately **0.7%** on the x-axis.
* **Trend:** This bar is significantly shorter, showing a substantial drop after applying the "evidence of explainability" filter.
### Key Observations
1. **Funnel Effect:** The chart clearly depicts a funnel or filtering process where each subsequent stage retains a smaller percentage of the original set.
2. **Major Reduction Point:** The most significant reduction in percentage occurs between the "On topic" and "Validated" stages (from ~1.3% to ~0.7%), suggesting that finding papers with concrete evidence of explainability is a much stricter filter than simply determining topical relevance.
3. **Low Initial Percentage:** Even the starting point, "Claims about humans," represents only about 1.4% of some larger, unstated corpus (e.g., all surveyed papers). This implies the initial topic itself is niche within the broader field.
4. **Visual Design:** The use of a single color for all bars emphasizes the sequential nature of the process. The annotations are spatially aligned with the gaps between bars, clearly indicating the action taken to move from one stage to the next.
### Interpretation
This chart visualizes the attrition rate of research papers through a validation pipeline focused on human-centric claims and explainability. The data suggests that while a small fraction of papers (~1.4%) make claims about humans, an even smaller subset (~0.7%, or half of the initial claims) can be validated with evidence of explainability.
The process highlights a potential gap in the research landscape: the challenge of moving from making claims to providing validated, explainable evidence for those claims. The steep drop at the final filter indicates that "evidence of explainability" is a high bar that many papers claiming to be on-topic do not meet. This could point to issues with reproducibility, methodological rigor, or the inherent difficulty of establishing explainability in this domain. The chart serves as a quantitative argument for the scarcity of validated, explainable research on human-related claims within the analyzed corpus.
</details>
(b) Zoomed in comparison of filtered XAI papers.
Figure 1. Distribution comparing all XAI literature from our Scopus search (Table 1), including those we scored. âAll XAI papersâ is the superset that encompasses all papers with keywords related to explainability, interpretability, etc. âClaims about humansâ papers are a subset of those that were filtered based on keywords related to human explainability. âOn topicâ papers are a subset of those that we filtered to exclude meta reviews, surveys, etc. âValidatedâ papers are a subset of those that provided empirical evidence of explainability. On the right a zoomed in version of the last three bars from the left figure are shown, note the change in x-axes.
Two subfigures depicting bar charts side by side. The first subfigure on the left shows a bar that represents 100% of the x-axis, where the x-axis ranges from 0% and 100%. The bottom three bars below the first bar are extremely small in comparison. The right subfigure shows the same bottom three bars but with a smaller x-axis from 0% to 2%.
Table 3. Table representation of our down-selection results. Our down-selection criteria is described in Section 3 and partially illustrated by keyword search in Table 1.
| Has explainability-related keywords | 18,254 | 100% |
| --- | --- | --- |
| Has claims about human explanability | 253 | 1.39% |
| Were on-topic (not surveys, etc.) | 237 | 1.30% |
| Were validated on human subjects | 128 | 0.70% |
If we consider the inverse of the percentages above, we see that over 98% of XAI papers do not appear with human evaluation keywords, and over 99% of XAI papers do not contain empirical human evaluation. There is naturally some difficulty in being precise with inclusion or exclusion criteria, but the results are nonetheless striking. Even if our search criteria were dramatically incorrect, say, an order of magnitude off, a ten-fold increase in validation rates in the literature would only result in 7% of works being tested with humans. We discuss refining our search criteria in the future in Section 5.
Since our work is inspired by Miller et al.âs (Miller et al., 2017), we provide a few relevant insights. Of the 253 papers in our search, only 11 cited Miller et al. Of those 11, 6 provided a human evaluation and 5 did not, roughly matching the overall rates of human evaluation in the literature. Thus, even the papers that point to a need for such evaluations do not perform them. That said, in general, the number of citations for Miller et al. has increased since publication (roughly 15 additional citations between 2018-2019 and 100 additional citations between 2023-2024).
## 4. Discussion of Results
### 4.1. Studies with Humans
<details>
<summary>x1.png Details</summary>
### Visual Description
## Bar Chart: Distribution of Human Subjects in Validated XAI Papers
### Overview
The image is a bar chart titled "Distribution of Human Subjects in Validated XAI Papers." It displays the frequency distribution of papers based on the number of human subjects used in their validation studies. The chart shows a heavily right-skewed distribution, with the vast majority of papers using a small number of subjects.
### Components/Axes
* **Chart Title:** "Distribution of Human Subjects in Validated XAI Papers" (centered at the top).
* **Y-Axis (Vertical):** Represents the count of papers. The axis is labeled with numerical markers from 0 to 100 in increments of 10 (0, 10, 20, 30, 40, 50, 60, 70, 80, 90, 100).
* **X-Axis (Horizontal):** Represents ranges (bins) for the number of human subjects. The axis is labeled with specific interval notation for each bin. The bins are, from left to right:
* `[2, 152]`
* `(152, 302]`
* `(302, 452]`
* `(452, 602]`
* `(602, 752]`
* `(752, 902]`
* `(902, 1052]`
* `(1052, 1202]`
* `(1202, 1352]`
* `(1352, 1502]`
* `(1502, 1652]`
* `(1652, 1802]`
* `(1802, 1952]`
* `(1952, 2102]`
* `(2102, 2252]`
* `(2252, 2402]`
* `(2402, 2552]`
* `(2552, 2702]`
* `(2702, 2852]`
* `(2852, 3002]`
* `(3002, 3152]`
* `(3152, 3302]`
* **Data Series:** A single series represented by blue bars. The value for each bar is annotated directly above it.
### Detailed Analysis
The chart presents the following exact data points, corresponding to the bins listed above:
1. **[2, 152]:** 91 papers
2. **(152, 302]:** 16 papers
3. **(302, 452]:** 3 papers
4. **(452, 602]:** 1 paper
5. **(602, 752]:** 0 papers
6. **(752, 902]:** 1 paper
7. **(902, 1052]:** 0 papers
8. **(1052, 1202]:** 0 papers
9. **(1202, 1352]:** 0 papers
10. **(1352, 1502]:** 0 papers
11. **(1502, 1652]:** 0 papers
12. **(1652, 1802]:** 0 papers
13. **(1802, 1952]:** 0 papers
14. **(1952, 2102]:** 1 paper
15. **(2102, 2252]:** 0 papers
16. **(2252, 2402]:** 0 papers
17. **(2402, 2552]:** 0 papers
18. **(2552, 2702]:** 0 papers
19. **(2702, 2852]:** 0 papers
20. **(2852, 3002]:** 0 papers
21. **(3002, 3152]:** 0 papers
22. **(3152, 3302]:** 1 paper
**Trend Verification:** The visual trend is a sharp, exponential-like decay. The first bar is overwhelmingly the tallest, followed by a steep drop to the second bar. The third and fourth bars are very small. The remaining bars are almost entirely flat at zero, with three isolated, single-count bars appearing at much higher ranges.
### Key Observations
1. **Dominant Bin:** The bin `[2, 152]` contains 91 out of the total 113 papers represented (approximately 80.5%).
2. **Rapid Drop-off:** There is a dramatic decrease in frequency after the first bin. Only 16 papers (â14.2%) fall into the next bin `(152, 302]`.
3. **Long Tail with Gaps:** The distribution has an extremely long tail. After the first four bins, the counts are almost exclusively zero, with three notable exceptions: one paper in `(752, 902]`, one in `(1952, 2102]`, and one in `(3152, 3302]`.
4. **Sparse High-End Data:** The presence of single papers in bins far to the right indicates a few outlier studies with exceptionally large human subject pools, but these are very rare.
### Interpretation
This distribution suggests a strong methodological pattern in the field of Explainable AI (XAI) validation research. The data indicates that the vast majority of studies rely on relatively small-scale human subject evaluations, typically involving fewer than 152 participants. This could be due to common constraints such as cost, time, and the logistical complexity of recruiting and running human-subject experiments.
The sharp decline after the first bin implies that scaling up validation studies beyond a few hundred subjects is uncommon. The three isolated papers with very high subject counts (752-902, 1952-2102, and 3152-3302) are significant outliers. These likely represent large-scale user studies, crowdsourced experiments, or meta-analyses that aggregate data from multiple smaller studies. Their rarity highlights them as exceptional cases rather than the norm.
Overall, the chart paints a picture of a research area where validation is predominantly conducted with small to moderate sample sizes, with a few landmark studies employing much larger scales. This has implications for the generalizability and statistical power of the conclusions drawn from the majority of XAI validation papers.
</details>
(a) Distribution of subjects across XAI papers that validated with a human evaluation.
<details>
<summary>x2.png Details</summary>
### Visual Description
## Histogram: Distribution of Human Subjects in Validated XAI Papers
### Overview
The image displays a histogram titled "Distribution of Human Subjects in Validated XAI Papers (N=[2,152])". It visualizes the frequency distribution of the number of human participants used across a set of 2,152 validated papers in the field of Explainable Artificial Intelligence (XAI). The chart shows a right-skewed distribution, indicating that most studies use a small number of human subjects, with a long tail of studies using progressively larger sample sizes.
### Components/Axes
* **Chart Title:** "Distribution of Human Subjects in Validated XAI Papers (N=[2,152])" (Top Center).
* **Y-Axis (Vertical):** Represents the count or frequency of papers. The axis is labeled with numerical markers: 0, 2, 4, 6, 8, 10, 12.
* **X-Axis (Horizontal):** Represents the range (bin) of human subjects per paper. The bins are presented as intervals, using a mix of closed `[a, b]` and half-open `(a, b]` notation. The bins are, from left to right:
`[2, 7]`, `(7, 12]`, `(12, 17]`, `(17, 22]`, `(22, 27]`, `(27, 32]`, `(32, 37]`, `(37, 42]`, `(42, 47]`, `(47, 52]`, `(52, 57]`, `(57, 62]`, `(62, 67]`, `(67, 72]`, `(72, 77]`, `(77, 82]`, `(82, 87]`, `(87, 92]`, `(92, 97]`, `(97, 102]`, `(102, 107]`, `(107, 112]`, `(112, 117]`, `(117, 122]`, `(122, 127]`, `(127, 132]`, `(132, 137]`, `(137, 142]`, `(142, 147]`, `(147, 152]`.
* **Data Series:** A single series represented by blue vertical bars. The exact count for each bin is annotated directly above its corresponding bar.
### Detailed Analysis
The following table lists each bin (range of human subjects) and the corresponding number of papers (frequency) as labeled on the chart:
| Bin (Range of Human Subjects) | Number of Papers |
| :--- | :--- |
| [2, 7] | 11 |
| (7, 12] | 4 |
| (12, 17] | 7 |
| (17, 22] | 8 |
| (22, 27] | 8 |
| (27, 32] | 5 |
| (32, 37] | 4 |
| (37, 42] | 4 |
| (42, 47] | 1 |
| (47, 52] | 6 |
| (52, 57] | 3 |
| (57, 62] | 5 |
| (62, 67] | 2 |
| (67, 72] | 3 |
| (72, 77] | 1 |
| (77, 82] | 0 |
| (82, 87] | 2 |
| (87, 92] | 0 |
| (92, 97] | 0 |
| (97, 102] | 2 |
| (102, 107] | 1 |
| (107, 112] | 0 |
| (112, 117] | 2 |
| (117, 122] | 5 |
| (122, 127] | 0 |
| (127, 132] | 0 |
| (132, 137] | 2 |
| (137, 142] | 0 |
| (142, 147] | 3 |
| (147, 152] | 2 |
**Trend Verification:** The visual trend is a strong right skew. The tallest bar is the first bin `[2, 7]` with 11 papers. The frequency generally decreases as the number of subjects increases, but with notable fluctuations and a long tail extending to the maximum bin of `(147, 152]`.
### Key Observations
1. **Dominant Small Sample Sizes:** The most frequent category is studies with 2 to 7 human subjects (11 papers). The next most common ranges are 12-27 subjects, with several bins containing 7-8 papers each.
2. **Right-Skewed Distribution:** The histogram has a long tail to the right, indicating a small number of studies use very large sample sizes (over 100 subjects), while the vast majority use fewer than 50.
3. **Gaps in Distribution:** There are several bins with a frequency of 0, particularly in the higher ranges (e.g., 77-82, 87-97, 107-112, 122-132, 137-142). This suggests a non-uniform, potentially clustered approach to sample size selection in the literature.
4. **Secondary Peaks:** There are minor secondary peaks in frequency around the `(47, 52]` bin (6 papers) and the `(117, 122]` bin (5 papers), which may represent common sample size targets for certain types of user studies.
5. **Total Count Discrepancy:** The title states N=[2,152], which likely refers to the total number of papers analyzed. Summing the frequencies from all bars yields a total of 105 papers represented in this histogram. This indicates the histogram is showing the distribution for a subset of the total papersâspecifically, those that included human subject validation.
### Interpretation
This histogram provides a quantitative snapshot of experimental practice in validated XAI research. The data suggests that **human-subject validation in XAI is predominantly conducted with small-scale studies**, often involving fewer than 30 participants. This is consistent with common practices in HCI and user study research, where in-depth qualitative insights or pilot studies are valued, and large-N studies are resource-intensive.
The right-skewed distribution and the presence of gaps imply that **there is no standard or widely adopted sample size** for XAI validation studies. The secondary peaks at specific ranges (like ~50 and ~120 subjects) might correspond to conventions from related fields (e.g., psychology) or practical constraints of online participant recruitment platforms.
The significant difference between the total papers in the dataset (2,152) and those with human-subject data shown here (105) is a critical finding. It implies that **the majority of "validated" XAI papers may not involve direct human evaluation**, relying instead on other forms of validation (e.g., technical metrics, case studies). This highlights a potential gap in the field between algorithmic development and human-centered evaluation. The chart effectively argues for a need to understand and potentially improve the rigor and scale of human-subject research in XAI.
</details>
(b) Subset of distribution counts, up to N=152 and bin size=5.
Figure 2. Distribution of counts for human subjects involved in an evaluation, experiment, interview, etc. for validated XAI papers. It is important to note that 13 papers did not report their human subject count and 3 papers approximated their count. In the bottom figure, a subset of distribution counts up to N=152 and a bin size of 5.
Two subfigures depicting histograms, one on the top and one on the bottom. The first histogram shows the distribution of participant counts for the entire range of data, while the bottom histogram represents a zoomed in view from participant count 0 to 152.
We find that out of the 237 papers that claimed some form of human explainability, only 56% of them provided an evaluation with humans. Of course, this number drops down to 0.7% when compared to the larger body of XAI work in which empirical evidence is not explicitly mentioned as a keyword in the paper, but for which explainability is a central topic and often a central claim of a new AI technique. Even if we extrapolate the fraction of off topic papers in the review set to the entirety of the XAI set, and exclude the same fraction from the overall number of papers under consideration, the percentage of papers with human evaluations as a part of this new âon-topicâ set is still approximately 0.7%.
One point worth discussing is the quality of the empirical evaluations in those 128 human-validated papers. In some cases, the participants in the evaluation were never directly discussed. Seen in Figure 2b, 13 of the papers that involved a validation did not report their human subject count and 3 papers reported an approximate count. Furthermore, in many of the papers, the authors do not disclose the subjectsâ expertise, workplace, education, or relationship to the authors. We found ourselves asking: who are the evaluation subjects? If we know nothing about these usersâ backgrounds, or potential motivations to call an XAI technique âgood,â one might argue this introduces an unfair bias into the validation.
Furthermore, many of the papers recruited college students as evaluation participants. Prior research has questioned the validity of such experiments (Wells and Bednarz, 2021) when the end-users of these XAI techniques are intended to be cybersecurity experts, pilots, doctors, and so on. Of course, it can be difficult to assemble domain experts to test methods and tools, especially domain experts who you are not working on the tool with. Perhaps that is why we observed a number of papers only having two or three domain experts as part of the evaluation (Figure 2b). Regardless, it is difficult to trust generalizable claims about explainability when the participant pool is very small or dissimilar to the target users.
Another observation was the type of evaluation conducted. Many of the papersâ evaluation criteria consisted of: âDo you think this is explainable â yes or no?â or âWhich is more explainable: our explainable method, or this other baseline explanation that is clearly worse (e.g., contains grammatical errors or has logical flaws)?â It is important that evaluation scenarios are crafted realistically to their intended target use case. Many questions related to the risks, potential failures, or edge-cases remain unanswered when explainable AI techniques are not properly studied with the correct end-user group nor represent a realistic evaluation scenario.
### 4.2. Studies without Humans
The studies that propose new XAI techniques but lack human testing almost universally show results of algorithmic performance, and at times a few exemplar outputs. While these elements are important and may provide circumstantial evidence, they are insufficient if a central claim is that a method is âexplainable.â If we think about the production of a possibly-explainable artifact as a signal being sent, it is straightforward for us to realize that it takes confirmation that the signal was received and understood correctly to confirm a claim of explainability. Otherwise, there is no difference between âexplainableâ AI and any other kind of AI, and the many examples of possibly-explainable methods are simply sending signals into a void.
We previously noted that the field still tends to use terms such as âexplainable,â âinterpretable,â âtransparent,â and so on, interchangeably (Lipton, 2018). Rudin (Rudin, 2019) makes the distinction between âexplainableâ as a post-hoc treatment of an otherwise black-box model, âinterpretableâ stemming from model design that is grounded in expert domain understanding. A small number of the papers that received a validation score of 0 did develop their models with the help of domain experts, thereby meeting Rudinâs âinterpretableâ definition, but did not perform any formal evaluation with such experts â neither with the group that helped develop them or with a broader group. Even in these cases, it remains an important step that the evaluation of human understandingâbeyond the typical one or two domain experts who helped develop these modelsâis included. While it is more likely that signals crafted with the help of domain experts are meaningful to their intended audience, a validation that such signals were received and understood is still necessary, as previous work has shown that it is not always the case. For example, domain experts can be vastly overconfident in their understanding of XAI methods that they helped craft (Siu et al., 2023).
For the many other papers with a validation score of 0, there is an often implied or at times explicit claim that an explanatory method is âintuitively obviousâ due to the inclusion of some property such as the use of natural language or conciseness of explanation. While these properties may seem self-evident, they are neither inherently clear nor universally effective, as even brief texts in oneâs native language can often be confusing or indecipherable. These properties are perhaps useful, but not inherently sufficient or necessarily effective (Buçinca et al., 2021).
## 5. Future Work
### 5.1. Beyond this Late-Breaking Work
This late-breaking work intends to shed light on the lackluster empirical evidence in the field of explainable AI. However, we consider this research to be a âprequelâ to future work that will require refining our literature search, Miller et al.âs scoring criteria (Miller et al., 2017), and our analysis of the resultant literature.
Generalizability. There are multiple opportunities to build upon our XAI literature review to improve the generalizability of our findings. Future work should examine identifying and analyzing literature not included in our search criteria: review articles, workshop papers, and Non-English papers. Including these works in an analysis would not only help the community gain a deeper understanding of empirical testing in XAI, but would also mitigate the inherent bias of studying English-only works. Our literature search could also be refined to distinguish different types of XAI research, resulting in a potentially smaller set of âAll XAI papersâ that are deemed more on-topic. For example, there are cases of XAI work that do not formally evaluate with humans due to their focus on the use of explanations for scientific discovery, or to understand the mechanisms by which some system works. These are typically works in the sciences, and it is true that broader evaluation of human-interpretability is likely unnecessary here if they directly present what principles have been discovered, or in many cases rediscovered, since those principles can be checked directly by the relevant domain experts. We posit these cases form a small minority of the work in the field; however, we would like to consider additional cases that should not be included in our broadest literature search.
Scoring criteria. Another important direction of research would be to rework Miller et al.âs scoring criteria (Miller et al., 2017) from 2017, specifically the data-driven criteria. Miller et al. rewarded XAI papers with the highest score of a 2 if the paper: (1) explicitly referenced a work in the social sciences or in a computer science venue, so long as the paper was not centrally about AI; and (2) incorporated a method from the social sciences in the paperâs contributed XAI method. We find that the data-driven criteria is somewhat unclear, and it is difficult to understand which computer science papers âcountâ versus âdo not count.â Should XAI papers that cite other XAI papersâwhich do include empirical evidenceâreceive a higher score? What if a new XAI method incorporates the results of a prior XAI paper but does not test this new method with human subjects? The validation score we used may also need to be reworked, as the sole presence of an evaluation may not be equivalent to a paper that presents a âgoodâ evaluation (as discussed in Section 3, some evaluations did not report a participant count or participant expertise).
Comprehensive analysis. It is imperative that a deeper analysis is conducted of our final set of âon-topicâ XAI papers. We briefly touched on the number of human subjects across all validated papers (Figure 2b), but some questions remain. For example, what is the distribution for the types of subjectsâe.g., experts versus students? Further, what we can learn from the papers that do and do not validate their claims with human studies, but still claim interpretability? What else do these papers have in common, and how else do they differ? Are there shared outcomes we can learn in testing explainability without human subjects? Moreover, do papers that validate with human studies tend to be published at a human-centered conference, e.g., ACM CHI, and not at an AI-centered conference, e.g., NeurIPS? The answers to these questions can help identify best practices in XAI testing, implications for AI design, and ultimately pave the way for rigorous human-centered evaluation frameworks. If other researchers would like to tackle these questions, we provide the exact keyword search used for our literature search in Table 1, which can be employed to replicate our findings.
### 5.2. Call to Action
Like claims of AI accuracy, claims of AI explainability must rest on empirical evidence. Contemporary artificial intelligence is an empirical discipline that rests on evidence of accuracy. However, our results show that work in contemporary explainable artificial intelligence almost never provides empirical evidence of its own.
While general heuristics such as conciseness or the use of domain-specific features are a reasonable place to start, we must not only rely on these heuristics. Empirical human testing is necessarily the gold standard for explainability claims. A pharmaceutical company would never distribute a newly developed drug without extensive clinical trials, even if they truly believed in its efficacy from biological principles, evidence collected in simulation, mouse trials, and so on. Why can XAI models, which may impact high-risk decision-making, be released untested with their human users?
Common objections to empirical evaluations are that humans are biased, or it is difficult to assemble a representative sample of humans. We note that the same can be said of every commonly-used machine learning benchmark, from ImageNet (Deng et al., 2009), to IMBD movie reviews (Maas et al., 2011), to medical question-and-answer (Jin et al., 2021). All contain human biases from their generation, curation, cultural context, or other factors. They may or may not match the intended application of the system being tested and took extensive time to collect. Nonetheless, they are used. The issue of limitations in the human testing population in interpretability testing is analogous to limitations in the dataset being used in accuracy testing and is not an unknown to the AI community.
The methods to test XAI clearly exist (Doshi-Velez and Kim, 2017) âour literature review shows that there are researchers using them and publishing about them. Undoubtedly there are challenges in moving from testing AI solely with static datasets into testing with humans, but they are not insurmountable. It is crucial that the XAI community take a critical look at the state of the field and determine whether the current trajectory and (lack of) standards is acceptable or not. As researchers, it seems unacceptable to produce copious volumes of âexplainableâ methodologies that have never been tested with the intended end-users. As reviewers, we are responsible for pushing back on research that claims explainability but provides little to no evidence. We hope the findings presented in this work challenge each otherâs viewpoints.
## 6. Conclusion
This late-breaking work highlights a critical gap in explainable AI (XAI) research: the overwhelming majority of papers that claim explainability fail to provide empirical evidence through human studies. By conducting a comprehensive literature search of 18,254 papers in collaboration with a professional librarian, we identified 253 papers that explicitly referenced human-related terms, such as human studies, user experiments, or end-user feedback. Among these, only 128 included some form of a human evaluations, revealing that 0.7% of XAI literature empirically validates claims of explainability. These findings highlight an urgent need for greater rigor in XAI research, particularly in substantiating claims about human interpretability and usability. Without empirical evidence, it is impossible to assess whether explainability methods truly address the needs of end-users. To foster transparency and reproducibility, we share our methodology and keyword search criteria, encouraging researchers to build on our work and prioritize human-centered evaluation in future explainable AI work.
Acknowledgements. DISTRIBUTION STATEMENT A. Approved for public release. Distribution is unlimited. This material is based upon work supported by the Under Secretary of Defense for Research and Engineering under Air Force Contract No. FA8702-15-D-0001. Any opinions, findings, conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the Under Secretary of Defense for Research and Engineering.
## References
- (1)
- Algorithmia (2021) Algorithmia. 2021. 2021 Enterprise Trends in Machine Learning. https://info.algorithmia.com/hubfs/2020/Reports/2021-Trends-in-ML/Algorithmia_2021_enterprise_ML_trends.pdf
- Board (2019) Defense Innovation Board. 2019. AI principles: recommendations on the ethical use of artificial intelligence by the department of defense: supporting document.
- Borenstein and Howard (2021) Jason Borenstein and Ayanna Howard. 2021. Emerging challenges in AI and the need for AI ethics education. AI and Ethics 1 (2021), 61â65.
- Buçinca et al. (2021) Zana Buçinca, Maja Barbara Malaya, and Krzysztof Z Gajos. 2021. To trust or to think: cognitive forcing functions can reduce overreliance on AI in AI-assisted decision-making. Proceedings of the ACM on Human-computer Interaction 5, CSCW1 (2021), 1â21.
- Burnham (2006) Judy F Burnham. 2006. Scopus database: a review. Biomedical digital libraries 3 (2006), 1â8.
- Cath (2018) Corinne Cath. 2018. Governing artificial intelligence: ethical, legal and technical opportunities and challenges. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 376, 2133 (2018), 20180080.
- Dash et al. (2019) Rupa Dash, Mark McMurtrey, Carl Rebman, and Upendra K. Kar. 2019. Application of Artificial Intelligence in Automation of Supply Chain Management. Journal of Strategic Innovation and Sustainability 14, 3 (Jul. 2019). doi: 10.33423/jsis.v14i3.2105
- Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. ImageNet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, âMiami, FL, USAâ, 248â255. doi: 10.1109/CVPR.2009.5206848
- Doshi-Velez and Kim (2017) Finale Doshi-Velez and Been Kim. 2017. Towards A Rigorous Science of Interpretable Machine Learning. http://arxiv.org/abs/1702.08608 cite arxiv:1702.08608.
- Edwards (2021) Lilian Edwards. 2021. The EU AI Act: a summary of its significance and scope.
- Holzinger et al. (2017) Andreas Holzinger, Chris Biemann, Constantinos S. Pattichis, and Douglas B. Kell. 2017. What do we need to build explainable AI systems for the medical domain? arXiv:1712.09923 [cs.AI] https://arxiv.org/abs/1712.09923
- Jin et al. (2021) Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2021. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences 11, 14 (2021), 6421.
- Koo et al. (2015) Jeamin Koo, Jungsuk Kwac, Wendy Ju, Martin Steinert, Larry Leifer, and Clifford Nass. 2015. Why did my car just do that? Explaining semi-autonomous driving actions to improve driver understanding, trust, and performance. International Journal on Interactive Design and Manufacturing (IJIDeM) 9 (2015), 269â275.
- Lipton (2018) Zachary C Lipton. 2018. The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery. Queue 16, 3 (2018), 31â57.
- Ly et al. (2017) Kien Hoa Ly, Ann-Marie Ly, and Gerhard Andersson. 2017. A fully automated conversational agent for promoting mental well-being: a pilot RCT using mixed methods. Internet interventions 10 (2017), 39â46.
- Maas et al. (2011) Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies - Volume 1 (Portland, Oregon) (HLT â11). Association for Computational Linguistics, USA, 142â150.
- Miller et al. (2017) Tim Miller, Piers Howe, and Liz Sonenberg. 2017. Explainable AI: Beware of Inmates Running the Asylum Or: How I Learnt to Stop Worrying and Love the Social and Behavioural Sciences. arXiv:1712.00547 [cs.AI] https://arxiv.org/abs/1712.00547
- Nauta et al. (2023) Meike Nauta, Jan Trienes, Shreyasi Pathak, Elisa Nguyen, Michelle Peters, Yasmin Schmitt, Jörg Schlötterer, Maurice Van Keulen, and Christin Seifert. 2023. From anecdotal evidence to quantitative evaluation methods: A systematic review on evaluating explainable ai. Comput. Surveys 55, 13s (2023), 1â42.
- Rolnick et al. (2022) David Rolnick, Priya L Donti, Lynn H Kaack, Kelly Kochanski, Alexandre Lacoste, Kris Sankaran, Andrew Slavin Ross, Nikola Milojevic-Dupont, Natasha Jaques, Anna Waldman-Brown, et al. 2022. Tackling climate change with machine learning. ACM Computing Surveys (CSUR) 55, 2 (2022), 1â96.
- Rudin (2019) Cynthia Rudin. 2019. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nat Mach Intell 1 (5): 206â215.
- Rudin and Radin (2019) Cynthia Rudin and Joanna Radin. 2019. Why are we using black box models in AI when we donât need to? A lesson from an explainable AI competition. Harvard Data Science Review 1, 2 (2019), 1â9.
- Saeed and Omlin (2023) Waddah Saeed and Christian Omlin. 2023. Explainable AI (XAI): A systematic meta-survey of current challenges and future opportunities. Knowledge-Based Systems 263 (2023), 110273.
- Siu et al. (2023) Ho Chit Siu, Kevin Leahy, and Makai Mann. 2023. STL: Surprisingly Tricky Logic (for System Validation). In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, âDetroit, MI, USAâ, 8613â8620. doi: 10.1109/IROS55552.2023.10342290
- Siu et al. (2021) Ho Chit Siu, Jaime Peña, Edenna Chen, Yutai Zhou, Victor Lopez, Kyle Palko, Kimberlee Chang, and Ross Allen. 2021. Evaluation of human-ai teams for learned and rule-based agents in hanabi. Advances in Neural Information Processing Systems 34 (2021), 16183â16195.
- Suh et al. (2024) Ashley Suh, Gabriel Appleby, Erik W. Anderson, Luca Finelli, Remco Chang, and Dylan Cashman. 2024. Are Metrics Enough? Guidelines for Communicating and Visualizing Predictive Models to Subject Matter Experts. IEEE Transactions on Visualization and Computer Graphics 30, 7 (2024), 4137â4153. doi: 10.1109/TVCG.2023.3259341
- Wells and Bednarz (2021) Lindsay Wells and Tomasz Bednarz. 2021. Explainable ai and reinforcement learningâa systematic review of current approaches and trends. Frontiers in artificial intelligence 4 (2021), 550030.