# Fewer Than 1% of Explainable AI Papers Validate Explainability with Humans
**Authors**: Ashley Suh, Isabelle Hurley, Nora Smith, Ho Chit Siu
> 0000-0001-6513-8447 MIT Lincoln Laboratory Lexington MA USA
> 0009-0006-4031-9018 MIT Lincoln Laboratory Lexington MA USA
> 0000-0001-5255-9226 MIT Lincoln Laboratory Lexington MA USA
> 0000-0003-3451-8046 MIT Lincoln Laboratory Lexington MA USA
(2025)
## Abstract
This late-breaking work presents a large-scale analysis of explainable AI (XAI) literature to evaluate claims of human explainability. We collaborated with a professional librarian to identify 18,254 papers containing keywords related to explainability and interpretability. Of these, we find that only 253 papers included terms suggesting human involvement in evaluating an XAI technique, and just 128 of those conducted some form of a human study. In other words, fewer than 1% of XAI papers (0.7%) provide empirical evidence of human explainability when compared to the broader body of XAI literature. Our findings underscore a critical gap between claims of human explainability and evidence-based validation, raising concerns about the rigor of XAI research. We call for increased emphasis on human evaluations in XAI studies and provide our literature search methodology to enable both reproducibility and further investigation into this widespread issue.
Explainable AI, Trustworthy AI, Interpretability, Human-Centered AI journalyear: 2025 copyright: rightsretained conference: Extended Abstracts of the CHI Conference on Human Factors in Computing Systems; April 26-May 1, 2025; Yokohama, Japan booktitle: Extended Abstracts of the CHI Conference on Human Factors in Computing Systems (CHI EA â25), April 26-May 1, 2025, Yokohama, Japan doi: 10.1145/3706599.3719964 isbn: 979-8-4007-1395-8/2025/04 ccs: Human-centered computing Empirical studies in HCI ccs: Computing methodologies Cognitive science ccs: Computing methodologies Philosophical/theoretical foundations of artificial intelligence
## 1. Introduction
The growing accessibility of artificial intelligence (AI) promises significant benefits for our society (Dash et al., 2019; Rolnick et al., 2022; Ly et al., 2017), as well as a number of critical challenges (Board, 2019; Cath, 2018; Borenstein and Howard, 2021). A pressing obstacle in the field is the lack of explainability, trustworthiness, and transparency in AI technologies (Rudin, 2019; Saeed and Omlin, 2023; Rudin and Radin, 2019; Holzinger et al., 2017). With AI systems being deployed to diagnose medical conditions, navigate vehicles on the road, and forecast natural disasters, understanding a modelâs behavior, recommendations, and reasoning has become imperative (Edwards, 2021). Yet, habitually used yet nondescript metrics like âaccuracyâ and âconfidenceâ fail to capture the whole story of an AI model nor how it will be interpreted by humans in real-world contexts (Suh et al., 2024).
The potential for skepticism and hesitation in trusting AI has driven the community to develop techniques in interpretable and explainable AI (XAI), concepts that are generally interchangeably used (Lipton, 2018), along with transparent and trustworthy AI While a precise definition for interpretability is not universal, Doshi-Velez and Kim define it as the ability to explain or present AI/ML systems in understandable terms to a human (Doshi-Velez and Kim, 2017). We refer to these methods collectively as research in explainable AI (XAI).. Common XAI approaches include rule-based reasoning, visualizations of deep learning processes, text-based explanations, and example-driven counterfactuals (Saeed and Omlin, 2023). Altogether, the developers of these explainability methods argue that AI models are now more usable, interpretable, and effective for human end-users than their non-explainable âblack boxâ counterparts.
However, research has repeatedly shown that explainability methods are often never tested with actual end-users (Wells and Bednarz, 2021; Nauta et al., 2023; Miller et al., 2017) â leading to mismatched user expectations, unused models, and diminished trust in AI (Koo et al., 2015; Algorithmia, 2021; Siu et al., 2021; Buçinca et al., 2021). A small survey conducted by Miller et al. (Miller et al., 2017) revealed that, from 23 papers published at the 2017 Explainable AI workshop at the International Joint Conference on Artificial Intelligence, only 4 articles referenced relevant social science research, only 1 of them built a model on this research, and none of the 23 included any empirical testing. Wells and Bednarzâs (Wells and Bednarz, 2021) found that 17 out of the 25 papers they surveyed on explainable AI for reinforcement learning did not include any user testing. Similarly, Nauta et al. (Nauta et al., 2023) surveyed 300 papers published at top AI/ML conferences and found that 1 in 3 papers evaluated their methodology exclusively with anecdotal evidence, and only 1 in 5 papers evaluated with actual users.
Our recurrent observation of explainability claims that lack human evidence motivated this late-breaking work. We sought to understand precisely how many papers across all XAI literature are promoting explainable, trustworthy, transparent, etc. methods without empirical testing. While there are many surveys of the XAI literature (Saeed and Omlin, 2023; Wells and Bednarz, 2021), what they tend to focus on is questions about what methods the field is using for explainability, what metrics or quantitative experiments are being carried out, or in some cases, what the state of the empirical testing is like. Furthermore, Miller et al.âs 2017 survey (Miller et al., 2017) â while distinct from these surveys â was self-described as a âlight scan of the literature,â only reviewing 23 papers. Similarly, Wells and Bednarzâs (Wells and Bednarz, 2021) reviewed 25 papers, while Nauta et al. (Nauta et al., 2023) reviewed 300.
We build on these related works by significantly expanding the scope of reviewed papers to include all relevant XAI literature and preprints in the Scopus database (Burnham, 2006). We collaborated with a professional librarian who helped us gather over 18,000 XAI papers and filter them based on claims about human explainability. The research question we aimed to answer was, âOf all the XAI papers that claim human explainability, how many validate those claims with empirical evidence?â Shockingly, we found that only 0.7% of XAI papers in the last decade test their methods with humans evaluating the explanations. Based on our literature review, the field is nearly entirely devoid of empirical evidence about its central claims of human explainability. For the remainder of the paper, we outline our methodology (which includes our exact Scopus database search for others to reproduce our work), the results of our literature survey, and a call to action for the broader community.
## 2. Methods
Table 1. Keyword search for gathering relevant literature using the Scopus database. The left column describes the high-level search strategy in natural language. The middle column shows the keyword search for scraping all XAI-related papers (e.g., papers that include terms related to âexplainabilityâ, âinterpretabilityâ, âXAIâ, etc.). The last column scrapes the same set of XAI papers but adds constraints related to human end-user testing (e.g., papers including âhuman-subjectâ, âend-userâ, âexperimentâ, âfeedbackâ, âinterviewâ, and so on).
| XAI papers AND Human end-user testing AND English | TITLE-ABS(((explainab* OR understandab* OR âinterpretab*â OR transparen*) W/1 (âartificial intelligenceâ OR âAIâ OR âmachine learningâ OR âMLâ)) OR XAI) AND LANGUAGE (english) | TITLE-ABS(((explainab* OR understandab* OR âinterpretab*â OR transparen*) W/1 (âartificial intelligenceâ OR âAIâ OR âmachine learningâ OR âMLâ)) OR XAI) AND TITLE-ABS((human OR âhuman-subjectâ OR user OR âend-userâ) Pre/1 (experiment* OR test OR trial OR evaluat* OR validat* OR feedback OR interview)) AND LANGUAGE (english) |
| --- | --- | --- |
| NOT Review papers | AND NOT (TITLE(review OR survey OR overview) OR DOCTYPE(cr OR re)) | AND NOT (TITLE(review OR survey OR overview) OR DOCTYPE(cr OR re)) |
We worked with a professional librarian over the course of three months to refine and generate advanced searches that could identify all existing XAI literature, as well as XAI literature with human-subject testing. This process involved iterating over relevant XAI search criteria (Miller et al., 2017), reviewing the results of the keyword search, and refining the keywords to include or exclude relevant or irrelevant papers. This process was repeated multiple times.
For the literature search, we selected the Scopus database for its broad coverage and ability to include the published and preprint literature. We wanted to include preprints in our search as much of the AI field publishes their results on arXiv, OSF, or another preprint platform due to the speed at which the field develops. However, we did not include review articles, workshop papers, or non-English papers. We discuss this limitation further in Section 5.
A total of 18,254 XAI papers were identified (13,839 published papers, 4,394 preprints), we refer to this set of papers as All XAI papers. We then filtered to a smaller subset of 253 papers which contained terms consistent with human subjects participating in testing, feedback, validation, interviews, and so on. This subset is referred to as papers with Claims about humans for the remainder of this work. It is important to note that papers that fall into the âclaims about humansâ category are papers that contain specific terms like âhuman feedback,â âend-user evaluation,â âuser interview,â and so on â see the right column in Table 1. We provide the full keyword search for anyone that would like to reproduce or refine our same literature review.
### 2.1. Scoring Criteria
The final subset of papers was manually reviewed by the authors. We use the validation criteria from Miller et al. (Miller et al., 2017) and give each paper in review (i.e., papers in the âClaims about humansâ category) one of two scores: 1 when the paper itself validates explainability claims with humans, and 0 otherwise. In the future we plan to code the broader set of XAI papers using Miller et al.âs data-driven criteria, which scores a paper based on whether it cites or implements work from the social sciences. However, we note that Miller et al.âs data-driven criteria â which was determined in 2017 â may be outdated. We discuss further in Section 5. For now, we limit our scope to only the use of humans in a behavioral study to evaluate the XAI method (i.e., the validation criteria) for simplicity.
Each article in the final review set was scored by 3 people in total with XAI backgrounds. To assign a validation score, we checked if any human experiments were conducted to test the proposed XAI method. Disagreements in scoring were broken collaboratively.
We observed several reasons to assign a 0 in our validation scoring. They include:
- The paper claims it is explainable-by-design, so it does not require human experiments.
- The paper contributes a framework or evaluation method that claims to be explainable but does not test the framework or evaluation method with humans.
- The paper claims human explainability because it used data derived from humans in it its training of the AI model or XAI method but does not evaluate with humans. We discuss this further in Section 4.2.
- The paper conducted a quantitative study comparing to human-annotated data or human labels (e.g., as a baseline), but does not provide human validation.
Finally, we excluded 16 papers from the final set that were agreed upon as off topic. Off topic papers did not present an XAI method but rather were meta-reviews of the field of XAI, philosophical arguments about explainability, or were summaries of several papers by particular graduate students.
## 3. Results
Table 2. Results of scoring 253 explainable AI papers that contained keywords consistent with humans participating in testing, providing feedback for, etc. an XAI technique or system (see Table 1). Our validation criteria follows Miller et al.âs (Miller et al., 2017), where papers receive a â0â if they did not validate with humans and a â1â if they did.
| Validation | 109 | 128 | 16 |
| --- | --- | --- | --- |
In total we scored 237 XAI papers that were both on topic and had terms consistent with claims about human explainability. The breakdown of our scores is shown in Table 2. From the 237 papers, the difference between those that actually conduct a human evaluation and those that do not is very small (54% of papers provide an evaluation, 46% do not). If you consider this number against the larger body of explainable AI work, it implies only 0.7% of XAI papers provide evidence from a human evaluation. Not only does this mean that nearly half of the XAI papers that directly claim human explainability never test this claim with humans, but that for every 1,000 XAI papers that do not explicitly mention humans, fewer than 8 of them provide explainability validation with empirical evidence.
The distribution of our down-selection results are shown in Figure 1a, with a zoomed in version of the last three bars shown in Figure 1b â note the change in the x-axes. âAll XAI Papersâ is the entirety of XAI literature that came back from our Scopus search (Table 1, middle column), âClaims about humansâ is the subset of those XAI papers that use keywords related to human-subjects, end-users experiments, tests, trials, etc. (Table 1, right column), âOn topicâ is a smaller subset of those papers that we deemed on topic (were not meta-reviews of the literature, survey papers, etc.), and âValidatedâ is the smallest subset of those papers that actually provided empirical evidence of human explainability. The same data is presented in Table 3, where each row is a subset of the one above, corresponding to the previously described search and review refinements.
<details>
<summary>extracted/6278551/figures/lit_review_percent.png Details</summary>
### Visual Description
# Technical Document Analysis: XAI Paper Filtering Process
## Chart Type
Horizontal bar chart with sequential filtering stages.
## Axes
- **X-axis**: Percentage scale (0% to 100%) with grid lines at 10% intervals.
- **Y-axis**: Categorical labels representing filtering stages (top to bottom):
1. All XAI papers
2. Claims about humans
3. On topic
4. Validated
## Key Components
1. **Bars**:
- All bars colored dark blue (#003f5c).
- Lengths proportional to remaining papers after each filter.
2. **Annotations**:
- Arrows connecting stages with explanatory text:
- "Filter for claims of human explainability" (between All XAI papers â Claims about humans)
- "Filter for papers we deem on topic" (between Claims about humans â On topic)
- "Filter for evidence of explainability" (between On topic â Validated)
## Data Points (Estimated Percentages)
| Stage | Estimated % | Notes |
|------------------------|-------------|--------------------------------|
| All XAI papers | 100% | Full dataset |
| Claims about humans | ~2-3% | Post-human explainability filter |
| On topic | ~1-2% | Post-topic relevance filter |
| Validated | ~0.5-1% | Post-evidence filter |
## Process Flow
1. **Initial Scope**: All XAI papers (100%) represent the total dataset.
2. **First Filter**: Papers claiming human explainability reduce the pool to ~2-3%.
3. **Second Filter**: Topic relevance further narrows to ~1-2%.
4. **Final Filter**: Evidence validation results in ~0.5-1% validated papers.
## Observations
- Each filter reduces the dataset by ~90%+.
- Final validated papers represent <1% of the original corpus.
- No legend present; single color used for all bars.
## Language
English only. No non-English text detected.
## Spatial Grounding
- Y-axis labels positioned left of bars.
- X-axis markers at bottom edge.
- Arrows connect stages vertically with explanatory text.
## Trend Verification
- Bars decrease monotonically from top to bottom (100% â ~1%).
- Visual confirmation matches expected filtering pattern.
## Component Isolation
1. **Header**: No explicit title present.
2. **Main Chart**: Horizontal bars with percentage scale.
3. **Footer**: No additional elements.
## Critical Notes
- Exact percentages not provided; estimates based on bar length relative to x-axis markers.
- No data table present; process described via sequential filtering stages.
</details>
(a) Comparison of all XAI literature we reviewed.
<details>
<summary>extracted/6278551/figures/lit_review_zoomed.png Details</summary>
### Visual Description
# Technical Document Analysis of Chart
## Chart Type
Horizontal bar chart with three data series and explanatory arrows.
## Labels and Axis Titles
- **X-axis**: Labeled with percentage values from **0.0%** to **2.0%** in increments of 0.5%.
- **Y-axis**: Categories listed as:
1. **Claims about humans**
2. **On topic**
3. **Validated**
## Data Points and Trends
1. **Claims about humans**
- **Value**: 1.4%
- **Position**: Topmost bar, spanning from 0.0% to 1.4% on the x-axis.
- **Trend**: Highest value in the dataset.
2. **On topic**
- **Value**: 1.3%
- **Position**: Middle bar, spanning from 0.0% to 1.3% on the x-axis.
- **Trend**: Slight decrease from the previous category (1.4% â 1.3%).
3. **Validated**
- **Value**: 0.7%
- **Position**: Bottom bar, spanning from 0.0% to 0.7% on the x-axis.
- **Trend**: Significant drop from the previous category (1.3% â 0.7%).
## Explanatory Arrows and Text
- **Arrow 1**:
- **From**: "Claims about humans" (1.4%)
- **To**: "On topic" (1.3%)
- **Text**: *"Filter for papers we deem on topic"*
- **Arrow 2**:
- **From**: "On topic" (1.3%)
- **To**: "Validated" (0.7%)
- **Text**: *"Filter for evidence of explainability"*
## Color and Legend
- **Bar Color**: All bars are **dark blue**.
- **Legend**: No explicit legend is present in the chart.
## Spatial Grounding
- **X-axis**: Positioned at the bottom of the chart, with percentage labels.
- **Y-axis**: Categories are vertically aligned on the left side.
- **Arrows**: Connect the end of each bar to the next category, positioned to the right of the bars.
## Component Isolation
1. **Header**: No explicit header title is visible.
2. **Main Chart**:
- Three horizontal bars with percentage values.
- Two explanatory arrows with descriptive text.
3. **Footer**: No footer elements are present.
## Transcribed Text
- **Y-axis Labels**:
- "Claims about humans"
- "On topic"
- "Validated"
- **Arrow Text**:
- "Filter for papers we deem on topic"
- "Filter for evidence of explainability"
- **X-axis Labels**:
- 0.0%, 0.5%, 1.0%, 1.5%, 2.0%
## Additional Notes
- The chart illustrates a **filtering process** where the initial set of claims ("Claims about humans") is reduced through two sequential filters:
1. **Topic relevance** (1.4% â 1.3%).
2. **Explainability evidence** (1.3% â 0.7%).
- No other languages or non-English text are present in the image.
## Conclusion
The chart visually represents a stepwise reduction in the number of papers through two filtering criteria, with the final validated set being 50% smaller than the initial claims. All data points and explanatory text are explicitly labeled.
</details>
(b) Zoomed in comparison of filtered XAI papers.
Figure 1. Distribution comparing all XAI literature from our Scopus search (Table 1), including those we scored. âAll XAI papersâ is the superset that encompasses all papers with keywords related to explainability, interpretability, etc. âClaims about humansâ papers are a subset of those that were filtered based on keywords related to human explainability. âOn topicâ papers are a subset of those that we filtered to exclude meta reviews, surveys, etc. âValidatedâ papers are a subset of those that provided empirical evidence of explainability. On the right a zoomed in version of the last three bars from the left figure are shown, note the change in x-axes.
Two subfigures depicting bar charts side by side. The first subfigure on the left shows a bar that represents 100% of the x-axis, where the x-axis ranges from 0% and 100%. The bottom three bars below the first bar are extremely small in comparison. The right subfigure shows the same bottom three bars but with a smaller x-axis from 0% to 2%.
Table 3. Table representation of our down-selection results. Our down-selection criteria is described in Section 3 and partially illustrated by keyword search in Table 1.
| Has explainability-related keywords | 18,254 | 100% |
| --- | --- | --- |
| Has claims about human explanability | 253 | 1.39% |
| Were on-topic (not surveys, etc.) | 237 | 1.30% |
| Were validated on human subjects | 128 | 0.70% |
If we consider the inverse of the percentages above, we see that over 98% of XAI papers do not appear with human evaluation keywords, and over 99% of XAI papers do not contain empirical human evaluation. There is naturally some difficulty in being precise with inclusion or exclusion criteria, but the results are nonetheless striking. Even if our search criteria were dramatically incorrect, say, an order of magnitude off, a ten-fold increase in validation rates in the literature would only result in 7% of works being tested with humans. We discuss refining our search criteria in the future in Section 5.
Since our work is inspired by Miller et al.âs (Miller et al., 2017), we provide a few relevant insights. Of the 253 papers in our search, only 11 cited Miller et al. Of those 11, 6 provided a human evaluation and 5 did not, roughly matching the overall rates of human evaluation in the literature. Thus, even the papers that point to a need for such evaluations do not perform them. That said, in general, the number of citations for Miller et al. has increased since publication (roughly 15 additional citations between 2018-2019 and 100 additional citations between 2023-2024).
## 4. Discussion of Results
### 4.1. Studies with Humans
<details>
<summary>x1.png Details</summary>
### Visual Description
# Distribution of Human Subjects in Validated XAI Papers
## Chart Description
This bar chart illustrates the distribution of human subjects across validated Explainable Artificial Intelligence (XAI) papers, categorized by publication intervals. The x-axis represents time intervals (in years), while the y-axis represents the percentage of papers within each interval.
### Key Components
1. **Title**: "Distribution of Human Subjects in Validated XAI Papers"
2. **X-Axis**: Time intervals (e.g., [2, 152], (152, 302], etc.)
3. **Y-Axis**: Percentage of papers (0â100)
4. **Bars**: Blue-colored bars representing the count of papers per interval
### Data Points
| Interval | Count |
|------------------|-------|
| [2, 152] | 91 |
| (152, 302] | 16 |
| (302, 452] | 3 |
| (452, 602] | 1 |
| (602, 752] | 0 |
| (752, 902] | 1 |
| (902, 1052] | 0 |
| (1052, 1202] | 1 |
| (1202, 1352] | 0 |
| (1352, 1502] | 1 |
| (1502, 1652] | 0 |
| (1652, 1802] | 1 |
| (1802, 1952] | 0 |
| (1952, 2102] | 1 |
| (2102, 2252] | 0 |
| (2252, 2402] | 1 |
| (2402, 2552] | 0 |
| (2552, 2702] | 1 |
| (2702, 2852] | 0 |
| (2852, 3002] | 1 |
| (3002, 3152] | 0 |
| (3152, 3302] | 1 |
### Trends and Observations
1. **Dominance of Early Intervals**: The interval [2, 152] accounts for **91%** of all papers, indicating a significant concentration of research in this period.
2. **Rapid Decline**: The next interval (152, 302] shows a sharp drop to **16%**, followed by minimal activity in subsequent intervals.
3. **Sparse Distribution**: Only **5 intervals** contain non-zero values (16, 3, 1, 1, 1), with the majority of intervals (12/17) having **0 papers**.
4. **Late-Period Activity**: The final interval (3152, 3302] shows a single paper, suggesting sporadic research in recent years.
### Legend and Color Analysis
- **Legend**: No explicit legend is present in the chart. All bars are uniformly colored in **blue**, consistent with standard bar chart conventions.
### Spatial Grounding
- **Legend Placement**: Not applicable (no legend exists).
- **Bar Colors**: All bars are blue, matching the default color scheme for bar charts.
### Trend Verification
- **Primary Trend**: A steep decline from 91% to 16% between the first two intervals, followed by near-zero activity until sporadic single-paper intervals appear.
- **Secondary Trend**: A pattern of alternating zeros and ones in later intervals, indicating minimal but persistent research activity.
### Component Isolation
1. **Header**: Title and axis labels.
2. **Main Chart**: Bars representing data points.
3. **Footer**: No additional text or annotations.
### Conclusion
The chart highlights a historical concentration of XAI research in early intervals, with minimal activity in later periods. This suggests a potential gap in recent advancements or a shift in research focus.
</details>
(a) Distribution of subjects across XAI papers that validated with a human evaluation.
<details>
<summary>x2.png Details</summary>
### Visual Description
# Technical Document: Distribution of Human Subjects in Validated XAI Papers (N=2,152)
## Chart Type
Bar chart
## Axes
- **X-axis**: Age ranges (intervals)
Labels: `[2, 7]`, `[7, 12]`, `[12, 17]`, `[17, 22]`, `[22, 27]`, `[27, 32]`, `[32, 37]`, `[37, 42]`, `[42, 47]`, `[47, 52]`, `[52, 57]`, `[57, 62]`, `[62, 67]`, `[67, 72]`, `[72, 77]`, `[77, 82]`, `[82, 87]`, `[87, 92]`, `[92, 97]`, `[97, 102]`, `[102, 107]`, `[107, 112]`, `[112, 117]`, `[117, 122]`, `[122, 127]`, `[127, 132]`, `[132, 137]`, `[137, 142]`, `[142, 147]`, `[147, 152]`
- **Y-axis**: Count of human subjects (N)
Scale: 0 to 12 (increments of 2)
## Title
"Distribution of Human Subjects in Validated XAI Papers (N=2,152)"
## Data Points
| Age Range | Count (N) |
|-------------|-----------|
| [2, 7] | 11 |
| [7, 12] | 4 |
| [12, 17] | 7 |
| [17, 22] | 8 |
| [22, 27] | 8 |
| [27, 32] | 5 |
| [32, 37] | 4 |
| [37, 42] | 4 |
| [42, 47] | 1 |
| [47, 52] | 6 |
| [52, 57] | 3 |
| [57, 62] | 5 |
| [62, 67] | 2 |
| [67, 72] | 3 |
| [72, 77] | 1 |
| [77, 82] | 0 |
| [82, 87] | 2 |
| [87, 92] | 0 |
| [92, 97] | 0 |
| [97, 102] | 2 |
| [102, 107] | 1 |
| [107, 112] | 0 |
| [112, 117] | 2 |
| [117, 122] | 5 |
| [122, 127] | 0 |
| [127, 132] | 2 |
| [132, 137] | 0 |
| [137, 142] | 3 |
| [142, 147] | 2 |
| [147, 152] | 2 |
## Key Trends
1. **Peaks**:
- Highest concentration in `[2, 7]` (11 subjects).
- Secondary peaks in `[17, 22]` and `[22, 27]` (8 subjects each).
- Another notable peak in `[47, 52]` (6 subjects).
2. **Declines**:
- Sharp drop from `[22, 27]` to `[42, 47]` (8 â 1).
- Gradual decline from `[57, 62]` (5) to `[67, 72]` (3).
3. **Zeros**:
- No subjects in `[77, 82]`, `[87, 92]`, `[92, 97]`, and `[107, 112]`.
4. **Late-Age Distribution**:
- Sparse but non-zero counts in `[117, 122]` (5) and `[137, 142]` (3).
## Observations
- **Majority of subjects** are concentrated in early age ranges (`[2, 7]` to `[27, 32]`).
- **Middle-age ranges** (`[42, 47]` to `[77, 82]`) show minimal representation.
- **Late-age ranges** (`[82, 87]` to `[147, 152]`) exhibit sporadic but increasing counts, peaking at `[117, 122]`.
## Legend
No legend present in the chart.
## Spatial Grounding
- **X-axis labels**: Positioned below each bar, centered.
- **Y-axis labels**: Positioned left of the axis, aligned with tick marks.
- **Bars**: Blue, uniform width, height proportional to count.
## Component Isolation
- **Header**: Title centered at the top.
- **Main Chart**: Bars with x-axis labels and y-axis scale.
- **Footer**: No additional text or components.
## Data Validation
- Total subjects: Sum of all counts = 2,152 (matches title).
- No overlapping or misaligned bars.
## Language
English (no non-English text detected).
</details>
(b) Subset of distribution counts, up to N=152 and bin size=5.
Figure 2. Distribution of counts for human subjects involved in an evaluation, experiment, interview, etc. for validated XAI papers. It is important to note that 13 papers did not report their human subject count and 3 papers approximated their count. In the bottom figure, a subset of distribution counts up to N=152 and a bin size of 5.
Two subfigures depicting histograms, one on the top and one on the bottom. The first histogram shows the distribution of participant counts for the entire range of data, while the bottom histogram represents a zoomed in view from participant count 0 to 152.
We find that out of the 237 papers that claimed some form of human explainability, only 56% of them provided an evaluation with humans. Of course, this number drops down to 0.7% when compared to the larger body of XAI work in which empirical evidence is not explicitly mentioned as a keyword in the paper, but for which explainability is a central topic and often a central claim of a new AI technique. Even if we extrapolate the fraction of off topic papers in the review set to the entirety of the XAI set, and exclude the same fraction from the overall number of papers under consideration, the percentage of papers with human evaluations as a part of this new âon-topicâ set is still approximately 0.7%.
One point worth discussing is the quality of the empirical evaluations in those 128 human-validated papers. In some cases, the participants in the evaluation were never directly discussed. Seen in Figure 2b, 13 of the papers that involved a validation did not report their human subject count and 3 papers reported an approximate count. Furthermore, in many of the papers, the authors do not disclose the subjectsâ expertise, workplace, education, or relationship to the authors. We found ourselves asking: who are the evaluation subjects? If we know nothing about these usersâ backgrounds, or potential motivations to call an XAI technique âgood,â one might argue this introduces an unfair bias into the validation.
Furthermore, many of the papers recruited college students as evaluation participants. Prior research has questioned the validity of such experiments (Wells and Bednarz, 2021) when the end-users of these XAI techniques are intended to be cybersecurity experts, pilots, doctors, and so on. Of course, it can be difficult to assemble domain experts to test methods and tools, especially domain experts who you are not working on the tool with. Perhaps that is why we observed a number of papers only having two or three domain experts as part of the evaluation (Figure 2b). Regardless, it is difficult to trust generalizable claims about explainability when the participant pool is very small or dissimilar to the target users.
Another observation was the type of evaluation conducted. Many of the papersâ evaluation criteria consisted of: âDo you think this is explainable â yes or no?â or âWhich is more explainable: our explainable method, or this other baseline explanation that is clearly worse (e.g., contains grammatical errors or has logical flaws)?â It is important that evaluation scenarios are crafted realistically to their intended target use case. Many questions related to the risks, potential failures, or edge-cases remain unanswered when explainable AI techniques are not properly studied with the correct end-user group nor represent a realistic evaluation scenario.
### 4.2. Studies without Humans
The studies that propose new XAI techniques but lack human testing almost universally show results of algorithmic performance, and at times a few exemplar outputs. While these elements are important and may provide circumstantial evidence, they are insufficient if a central claim is that a method is âexplainable.â If we think about the production of a possibly-explainable artifact as a signal being sent, it is straightforward for us to realize that it takes confirmation that the signal was received and understood correctly to confirm a claim of explainability. Otherwise, there is no difference between âexplainableâ AI and any other kind of AI, and the many examples of possibly-explainable methods are simply sending signals into a void.
We previously noted that the field still tends to use terms such as âexplainable,â âinterpretable,â âtransparent,â and so on, interchangeably (Lipton, 2018). Rudin (Rudin, 2019) makes the distinction between âexplainableâ as a post-hoc treatment of an otherwise black-box model, âinterpretableâ stemming from model design that is grounded in expert domain understanding. A small number of the papers that received a validation score of 0 did develop their models with the help of domain experts, thereby meeting Rudinâs âinterpretableâ definition, but did not perform any formal evaluation with such experts â neither with the group that helped develop them or with a broader group. Even in these cases, it remains an important step that the evaluation of human understandingâbeyond the typical one or two domain experts who helped develop these modelsâis included. While it is more likely that signals crafted with the help of domain experts are meaningful to their intended audience, a validation that such signals were received and understood is still necessary, as previous work has shown that it is not always the case. For example, domain experts can be vastly overconfident in their understanding of XAI methods that they helped craft (Siu et al., 2023).
For the many other papers with a validation score of 0, there is an often implied or at times explicit claim that an explanatory method is âintuitively obviousâ due to the inclusion of some property such as the use of natural language or conciseness of explanation. While these properties may seem self-evident, they are neither inherently clear nor universally effective, as even brief texts in oneâs native language can often be confusing or indecipherable. These properties are perhaps useful, but not inherently sufficient or necessarily effective (Buçinca et al., 2021).
## 5. Future Work
### 5.1. Beyond this Late-Breaking Work
This late-breaking work intends to shed light on the lackluster empirical evidence in the field of explainable AI. However, we consider this research to be a âprequelâ to future work that will require refining our literature search, Miller et al.âs scoring criteria (Miller et al., 2017), and our analysis of the resultant literature.
Generalizability. There are multiple opportunities to build upon our XAI literature review to improve the generalizability of our findings. Future work should examine identifying and analyzing literature not included in our search criteria: review articles, workshop papers, and Non-English papers. Including these works in an analysis would not only help the community gain a deeper understanding of empirical testing in XAI, but would also mitigate the inherent bias of studying English-only works. Our literature search could also be refined to distinguish different types of XAI research, resulting in a potentially smaller set of âAll XAI papersâ that are deemed more on-topic. For example, there are cases of XAI work that do not formally evaluate with humans due to their focus on the use of explanations for scientific discovery, or to understand the mechanisms by which some system works. These are typically works in the sciences, and it is true that broader evaluation of human-interpretability is likely unnecessary here if they directly present what principles have been discovered, or in many cases rediscovered, since those principles can be checked directly by the relevant domain experts. We posit these cases form a small minority of the work in the field; however, we would like to consider additional cases that should not be included in our broadest literature search.
Scoring criteria. Another important direction of research would be to rework Miller et al.âs scoring criteria (Miller et al., 2017) from 2017, specifically the data-driven criteria. Miller et al. rewarded XAI papers with the highest score of a 2 if the paper: (1) explicitly referenced a work in the social sciences or in a computer science venue, so long as the paper was not centrally about AI; and (2) incorporated a method from the social sciences in the paperâs contributed XAI method. We find that the data-driven criteria is somewhat unclear, and it is difficult to understand which computer science papers âcountâ versus âdo not count.â Should XAI papers that cite other XAI papersâwhich do include empirical evidenceâreceive a higher score? What if a new XAI method incorporates the results of a prior XAI paper but does not test this new method with human subjects? The validation score we used may also need to be reworked, as the sole presence of an evaluation may not be equivalent to a paper that presents a âgoodâ evaluation (as discussed in Section 3, some evaluations did not report a participant count or participant expertise).
Comprehensive analysis. It is imperative that a deeper analysis is conducted of our final set of âon-topicâ XAI papers. We briefly touched on the number of human subjects across all validated papers (Figure 2b), but some questions remain. For example, what is the distribution for the types of subjectsâe.g., experts versus students? Further, what we can learn from the papers that do and do not validate their claims with human studies, but still claim interpretability? What else do these papers have in common, and how else do they differ? Are there shared outcomes we can learn in testing explainability without human subjects? Moreover, do papers that validate with human studies tend to be published at a human-centered conference, e.g., ACM CHI, and not at an AI-centered conference, e.g., NeurIPS? The answers to these questions can help identify best practices in XAI testing, implications for AI design, and ultimately pave the way for rigorous human-centered evaluation frameworks. If other researchers would like to tackle these questions, we provide the exact keyword search used for our literature search in Table 1, which can be employed to replicate our findings.
### 5.2. Call to Action
Like claims of AI accuracy, claims of AI explainability must rest on empirical evidence. Contemporary artificial intelligence is an empirical discipline that rests on evidence of accuracy. However, our results show that work in contemporary explainable artificial intelligence almost never provides empirical evidence of its own.
While general heuristics such as conciseness or the use of domain-specific features are a reasonable place to start, we must not only rely on these heuristics. Empirical human testing is necessarily the gold standard for explainability claims. A pharmaceutical company would never distribute a newly developed drug without extensive clinical trials, even if they truly believed in its efficacy from biological principles, evidence collected in simulation, mouse trials, and so on. Why can XAI models, which may impact high-risk decision-making, be released untested with their human users?
Common objections to empirical evaluations are that humans are biased, or it is difficult to assemble a representative sample of humans. We note that the same can be said of every commonly-used machine learning benchmark, from ImageNet (Deng et al., 2009), to IMBD movie reviews (Maas et al., 2011), to medical question-and-answer (Jin et al., 2021). All contain human biases from their generation, curation, cultural context, or other factors. They may or may not match the intended application of the system being tested and took extensive time to collect. Nonetheless, they are used. The issue of limitations in the human testing population in interpretability testing is analogous to limitations in the dataset being used in accuracy testing and is not an unknown to the AI community.
The methods to test XAI clearly exist (Doshi-Velez and Kim, 2017) âour literature review shows that there are researchers using them and publishing about them. Undoubtedly there are challenges in moving from testing AI solely with static datasets into testing with humans, but they are not insurmountable. It is crucial that the XAI community take a critical look at the state of the field and determine whether the current trajectory and (lack of) standards is acceptable or not. As researchers, it seems unacceptable to produce copious volumes of âexplainableâ methodologies that have never been tested with the intended end-users. As reviewers, we are responsible for pushing back on research that claims explainability but provides little to no evidence. We hope the findings presented in this work challenge each otherâs viewpoints.
## 6. Conclusion
This late-breaking work highlights a critical gap in explainable AI (XAI) research: the overwhelming majority of papers that claim explainability fail to provide empirical evidence through human studies. By conducting a comprehensive literature search of 18,254 papers in collaboration with a professional librarian, we identified 253 papers that explicitly referenced human-related terms, such as human studies, user experiments, or end-user feedback. Among these, only 128 included some form of a human evaluations, revealing that 0.7% of XAI literature empirically validates claims of explainability. These findings highlight an urgent need for greater rigor in XAI research, particularly in substantiating claims about human interpretability and usability. Without empirical evidence, it is impossible to assess whether explainability methods truly address the needs of end-users. To foster transparency and reproducibility, we share our methodology and keyword search criteria, encouraging researchers to build on our work and prioritize human-centered evaluation in future explainable AI work.
Acknowledgements. DISTRIBUTION STATEMENT A. Approved for public release. Distribution is unlimited. This material is based upon work supported by the Under Secretary of Defense for Research and Engineering under Air Force Contract No. FA8702-15-D-0001. Any opinions, findings, conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the Under Secretary of Defense for Research and Engineering.
## References
- (1)
- Algorithmia (2021) Algorithmia. 2021. 2021 Enterprise Trends in Machine Learning. https://info.algorithmia.com/hubfs/2020/Reports/2021-Trends-in-ML/Algorithmia_2021_enterprise_ML_trends.pdf
- Board (2019) Defense Innovation Board. 2019. AI principles: recommendations on the ethical use of artificial intelligence by the department of defense: supporting document.
- Borenstein and Howard (2021) Jason Borenstein and Ayanna Howard. 2021. Emerging challenges in AI and the need for AI ethics education. AI and Ethics 1 (2021), 61â65.
- Buçinca et al. (2021) Zana Buçinca, Maja Barbara Malaya, and Krzysztof Z Gajos. 2021. To trust or to think: cognitive forcing functions can reduce overreliance on AI in AI-assisted decision-making. Proceedings of the ACM on Human-computer Interaction 5, CSCW1 (2021), 1â21.
- Burnham (2006) Judy F Burnham. 2006. Scopus database: a review. Biomedical digital libraries 3 (2006), 1â8.
- Cath (2018) Corinne Cath. 2018. Governing artificial intelligence: ethical, legal and technical opportunities and challenges. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 376, 2133 (2018), 20180080.
- Dash et al. (2019) Rupa Dash, Mark McMurtrey, Carl Rebman, and Upendra K. Kar. 2019. Application of Artificial Intelligence in Automation of Supply Chain Management. Journal of Strategic Innovation and Sustainability 14, 3 (Jul. 2019). doi: 10.33423/jsis.v14i3.2105
- Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. ImageNet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, âMiami, FL, USAâ, 248â255. doi: 10.1109/CVPR.2009.5206848
- Doshi-Velez and Kim (2017) Finale Doshi-Velez and Been Kim. 2017. Towards A Rigorous Science of Interpretable Machine Learning. http://arxiv.org/abs/1702.08608 cite arxiv:1702.08608.
- Edwards (2021) Lilian Edwards. 2021. The EU AI Act: a summary of its significance and scope.
- Holzinger et al. (2017) Andreas Holzinger, Chris Biemann, Constantinos S. Pattichis, and Douglas B. Kell. 2017. What do we need to build explainable AI systems for the medical domain? arXiv:1712.09923 [cs.AI] https://arxiv.org/abs/1712.09923
- Jin et al. (2021) Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2021. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences 11, 14 (2021), 6421.
- Koo et al. (2015) Jeamin Koo, Jungsuk Kwac, Wendy Ju, Martin Steinert, Larry Leifer, and Clifford Nass. 2015. Why did my car just do that? Explaining semi-autonomous driving actions to improve driver understanding, trust, and performance. International Journal on Interactive Design and Manufacturing (IJIDeM) 9 (2015), 269â275.
- Lipton (2018) Zachary C Lipton. 2018. The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery. Queue 16, 3 (2018), 31â57.
- Ly et al. (2017) Kien Hoa Ly, Ann-Marie Ly, and Gerhard Andersson. 2017. A fully automated conversational agent for promoting mental well-being: a pilot RCT using mixed methods. Internet interventions 10 (2017), 39â46.
- Maas et al. (2011) Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies - Volume 1 (Portland, Oregon) (HLT â11). Association for Computational Linguistics, USA, 142â150.
- Miller et al. (2017) Tim Miller, Piers Howe, and Liz Sonenberg. 2017. Explainable AI: Beware of Inmates Running the Asylum Or: How I Learnt to Stop Worrying and Love the Social and Behavioural Sciences. arXiv:1712.00547 [cs.AI] https://arxiv.org/abs/1712.00547
- Nauta et al. (2023) Meike Nauta, Jan Trienes, Shreyasi Pathak, Elisa Nguyen, Michelle Peters, Yasmin Schmitt, Jörg Schlötterer, Maurice Van Keulen, and Christin Seifert. 2023. From anecdotal evidence to quantitative evaluation methods: A systematic review on evaluating explainable ai. Comput. Surveys 55, 13s (2023), 1â42.
- Rolnick et al. (2022) David Rolnick, Priya L Donti, Lynn H Kaack, Kelly Kochanski, Alexandre Lacoste, Kris Sankaran, Andrew Slavin Ross, Nikola Milojevic-Dupont, Natasha Jaques, Anna Waldman-Brown, et al. 2022. Tackling climate change with machine learning. ACM Computing Surveys (CSUR) 55, 2 (2022), 1â96.
- Rudin (2019) Cynthia Rudin. 2019. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nat Mach Intell 1 (5): 206â215.
- Rudin and Radin (2019) Cynthia Rudin and Joanna Radin. 2019. Why are we using black box models in AI when we donât need to? A lesson from an explainable AI competition. Harvard Data Science Review 1, 2 (2019), 1â9.
- Saeed and Omlin (2023) Waddah Saeed and Christian Omlin. 2023. Explainable AI (XAI): A systematic meta-survey of current challenges and future opportunities. Knowledge-Based Systems 263 (2023), 110273.
- Siu et al. (2023) Ho Chit Siu, Kevin Leahy, and Makai Mann. 2023. STL: Surprisingly Tricky Logic (for System Validation). In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, âDetroit, MI, USAâ, 8613â8620. doi: 10.1109/IROS55552.2023.10342290
- Siu et al. (2021) Ho Chit Siu, Jaime Peña, Edenna Chen, Yutai Zhou, Victor Lopez, Kyle Palko, Kimberlee Chang, and Ross Allen. 2021. Evaluation of human-ai teams for learned and rule-based agents in hanabi. Advances in Neural Information Processing Systems 34 (2021), 16183â16195.
- Suh et al. (2024) Ashley Suh, Gabriel Appleby, Erik W. Anderson, Luca Finelli, Remco Chang, and Dylan Cashman. 2024. Are Metrics Enough? Guidelines for Communicating and Visualizing Predictive Models to Subject Matter Experts. IEEE Transactions on Visualization and Computer Graphics 30, 7 (2024), 4137â4153. doi: 10.1109/TVCG.2023.3259341
- Wells and Bednarz (2021) Lindsay Wells and Tomasz Bednarz. 2021. Explainable ai and reinforcement learningâa systematic review of current approaches and trends. Frontiers in artificial intelligence 4 (2021), 550030.