# AI Transparency Atlas: Framework, Scoring, and Real-Time Model Card Evaluation Pipeline
**Authors**: Akhmadillo Mamirov1, Faiaz Azmain1, Hanyu Wang
## Abstract
AI model documentation is fragmented across platforms and inconsistent in structure, preventing policymakers, auditors, and users from reliably assessing safety claims, data provenance, and version changes. We analyzed documentation from five frontier models (Gemini 3, Grok 4.1, Llama 4, GPT-5, Claude 4.5) and 100 Hugging Face model cards, identifying 947 unique section names with extreme naming variation—usage information alone appeared under 97 different labels. Using the EU AI Act Annex IV and Stanford Transparency Index as baselines, we developed a weighted transparency framework with 8 sections and 23 subsections that prioritizes safety-critical disclosures (Safety Evaluation: 25%, Critical Risk: 20%) over technical specifications. We implemented an automated multi-agent pipeline that extracts documentation from public sources and scores completeness through LLM consensus. Evaluating 50 models across vision, multimodal, open-source, and closed-source systems cost less than $3 total and revealed systematic gaps: frontier labs (xAI, Microsoft, Anthropic) achieve 80% compliance, while most providers fall below 60%. Safety-critical categories show the largest deficits—deception behaviors, hallucinations, and child safety evaluations account for 148, 124, and 116 aggregate points lost respectively across all evaluated models.
## I Introduction
AI model documentation today is fragmented across whitepapers, GitHub READMEs, Hugging Face model cards, system cards, and blog posts. This fragmentation raises a core question: what practical steps can move the ecosystem from documentation inconsistency toward something standardized enough to be useful?
Documentation gaps affect every stakeholder. Regulators cannot reliably assess governance or safety without consistent reporting. Downstream institutions such as hospitals, schools, and public agencies lack visibility into model risks, evaluation protocols, and version-level changes. Even within the same platform, model cards differ dramatically in length, scope, and granularity. Some include details about architecture, training data, and evaluation settings, while others provide only brief paragraphs. Critically, documentation often does not evolve as models evolve. Major capability updates, training adjustments, and safety interventions rarely trigger corresponding updates. Versioning is ad hoc or entirely absent, causing transparency to degrade over time.
System cards attempt to address transparency at the deployment level, but they introduce their own challenges. Many system cards are high-level but not actionable; others remain closed, incomplete, or not understandable to external auditors. When modern AI systems depend on chains of interconnected models, datasets, and processes, opacity at the system layer becomes a structural barrier to accountability. When something goes wrong, responding is difficult because relevant information is scattered across multiple documents and repositories [winecoff2024improvinggovernanceoutcomesai].
### I-A Why Does This Inconsistency Persist?
The persistence of fragmented AI documentation reflects structural challenges in the AI development ecosystem. Unlike regulated industries where documentation is mandatory and enforcement mechanisms are well established, AI transparency remains largely voluntary, resulting in misaligned incentives.
Economic and competitive pressures. Comprehensive documentation is resource-intensive, requiring dedicated teams for safety evaluations, data provenance tracking, and version management. Organizations prioritizing rapid deployment often treat documentation as secondary to product development. Closed-source developers face additional tension: detailed transparency can expose competitive advantages related to training methods, data sources, or architectural choices. As a result, disclosure decisions frequently involve trade-offs between transparency commitments and intellectual property protection.
Organizational fragmentation. High-quality documentation requires coordination across teams that typically operate independently. Engineers prioritize model performance, safety teams focus on risk assessment, and communications teams manage external messaging. Without integrated workflows that treat documentation as a natural byproduct of development, information remains siloed across internal wikis, isolated reports, and fragmented public communications.
Competing standards. Developers encounter overlapping documentation proposals, including Model Cards, Datasheets for Datasets, System Cards, and emerging regulatory frameworks, each emphasizing different priorities and formats. In the absence of a unified standard specifying required content, level of detail, and structure, developers make inconsistent choices. Some release brief model cards, while others publish extensive technical reports. Neither approach consistently satisfies stakeholder needs because no authoritative standard exists.
Limited accountability. Documentation quality improves when it can be credibly evaluated. Currently, users lack practical means to assess completeness or accuracy, regulators lack scalable audit tools, and civil society organizations can identify only the most visible gaps. Without mechanisms to systematically measure and publicly compare documentation quality, providers face limited accountability for incomplete or uneven disclosures.
### I-B Our Approach Builds on Existing Regulatory and Academic Frameworks.
We address these challenges through a structured transparency framework grounded in existing regulatory and academic standards. The EU AI Act Annex IV provides concrete documentation requirements that we adopt as a regulatory baseline [eu_ai_act_annex_iv]. We use the Stanford Transparency Index as an academic reference point for evaluating disclosure completeness across AI providers [bommasani2024fmti].
To populate this framework at scale, we built an automated pipeline that extracts documentation from dispersed public sources, evaluates completeness using multi-agent LLM consensus, and generates transparency scores. This approach enables systematic, continuous assessment of documentation quality across hundreds of models without requiring direct developer cooperation.
## II Background and Related Work
The Foundation Model Transparency Index (May 2024) by Stanford University provides a comprehensive assessment of foundation model developer transparency [bommasani2024fmti]. The index evaluates disclosure practices across major AI labs using data provided directly by developers. While this approach offers depth and accuracy for covered models, it relies on developer cooperation and self-reported information. Our work takes a complementary approach: we evaluate documentation completeness using only publicly available information online—the same information accessible to regulators, auditors, and downstream users in practice. This distinction is critical because real-world transparency depends on what stakeholders can actually access and verify, not just what developers are willing to share upon request.
<details>
<summary>sections.png Details</summary>
### Visual Description
## Horizontal Bar Chart: Information Category Presence Across Model Cards
### Overview
The chart visualizes the presence of 11 information categories across model cards, categorized by their completeness (Present, Partially Present, Not Present). A 50% threshold is marked with a vertical dashed line, indicating a baseline for "adequate" inclusion.
### Components/Axes
- **X-axis**: "Percentage of Model Cards" (0–80%, increments of 10%).
- **Y-axis**: Information categories (listed left-to-right):
1. Model Architecture
2. Evaluation Metrics
3. Compute Requirements
4. Intended Use
5. License
6. Limitations
7. Training Data
8. Bias Fairness
9. Safety Evaluation
10. Out Of Scope
11. Interpretability
- **Legend** (bottom-right):
- Green: "Present"
- Orange: "Partially Present"
- Red: "Not Present"
### Detailed Analysis
1. **Green Bars (Present, 80%)**:
- Model Architecture
- Evaluation Metrics
- Compute Requirements
- All exceed the 50% threshold by 30 percentage points.
2. **Orange Bars (Partially Present, 50–65%)**:
- Intended Use (65%)
- License (65%)
- Limitations (60%)
- Training Data (60%)
- Bias Fairness (55%)
- Safety Evaluation (50%)
- Out Of Scope (50%)
3. **Red Bar (Not Present, 20%)**:
- Interpretability
### Key Observations
- **Dominance of Technical Categories**: 100% of technical categories (Model Architecture, Evaluation Metrics, Compute Requirements) are fully present.
- **Ethical/Governance Categories**: Only 50% of categories like Bias Fairness, Safety Evaluation, and Out Of Scope meet the 50% threshold.
- **Interpretability Gap**: Interpretability is the only category marked "Not Present" (20%), highlighting a critical omission.
### Interpretation
The chart reveals a stark imbalance in model card completeness:
- **Technical Focus**: Core technical details (architecture, evaluation, compute) are consistently prioritized, suggesting a focus on operational transparency.
- **Ethical Oversights**: Categories like Bias Fairness and Safety Evaluation are only partially addressed, indicating potential gaps in addressing societal impacts.
- **Interpretability Crisis**: The absence of Interpretability (20%) is alarming, as it undermines trust and accountability in AI systems. This omission may reflect a lack of tools or prioritization for explainability in model development workflows.
The data underscores a need for standardized frameworks to ensure ethical and interpretability requirements are met alongside technical specifications.
</details>
Figure 1: Category presence rates across 100 model cards. Technical categories exceed 90% presence while safety-critical categories often fall below 50%
## III CURRENT STATE OF AI MODEL DOCUMENTATION
### III-A Frontier Model Documentation Review
To ground our framework in existing practice, we conducted a structured review of recent documentation artifacts from leading AI developers, including Google (Gemini 3), xAI (Grok 4.1), Meta (Llama 4), OpenAI (GPT–5), and Anthropic (Claude 4.5). We also compared these against the original Model Cards proposal by Mitchell et al. (2019), which remains the conceptual baseline [DBLP:journals/corr/abs-1810-03993].
Across these models, we observed major differences in scope, depth, structure, and emphasis:
- Google’s Gemini 3 includes detailed sections on training data, architecture, sustainability, safety evaluations, and frontier-safety considerations [google_gemini3_modelcard].
- xAI’s Grok 4.1 concentrates heavily on refusal behavior, adversarial robustness, dual-use risk, and transparency around data and training [xai_grok4_1_modelcard].
- Meta’s Llama 4 emphasizes environmental footprint, quantization, safeguards, community governance, critical risks, and detailed fine-tuning behavior [meta_llama4_modelcard].
- OpenAI’s GPT-5 provides the most extensive safety-evaluation sections, including red-teaming by multiple external organizations, biological risk assessments, cybersecurity stress tests, deception and sycophancy evaluations, and system-level protections [openai_gpt5_system_card].
- Anthropic’s Claude 4.5 offers the most granular system-level reporting, covering agentic risks, alignment attempts, cyber ranges, interpretability studies, reward-hacking investigations, welfare assessments, and Responsibly Scaling Policy (RSP) aligned evaluations [anthropic_claude_sonnet45_system_card].
### III-B Systematic Analysis of Hugging Face Model Cards
To assess whether these documentation patterns extend beyond frontier models, we analyzed 100 model cards from Hugging Face, spanning diverse model types, parameter scales, and deployment contexts [AI_documentation_analysis_2025]. This approach follows Liang et al. [liang2024whatsdocumentedaisystematic], who conducted a systematic analysis of 32,000 AI model cards and demonstrated that Hugging Face provides a representative sample of documentation practices across the broader AI ecosystem. From this broader sample, we identified 11 recurring documentation categories that appeared with varying frequency and depth: Model Architecture, Compute Requirements, Evaluation Metrics, License, Intended Use, Training Data, Limitations, Bias and Fairness, Safety Evaluation, Out-of-Scope Use, and Interpretability.
Each model card was evaluated according to three levels of completeness:
- Detailed: The category includes substantive information with specific details, metrics, or actionable content.
- Mentioned Only: The category is present but limited to high-level or superficial statements.
- Absent: The category is not addressed.
Figure 1 illustrates category presence rates across the dataset. While over 90% of model cards included Model Architecture, Evaluation Metrics, and Compute Requirements, several critical transparency indicators fell below the 50% threshold. Interpretability appeared in only approximately 20% of cards, and Out-of-Scope use cases in roughly 60%. Even when present, Safety Evaluation (approximately 65%) and Bias and Fairness (approximately 60%) were often addressed superficially.
<details>
<summary>completeness.png Details</summary>
### Visual Description
## Horizontal Bar Chart: Documentation Depth by Category
### Overview
The chart visualizes the depth of documentation across 12 categories related to AI/ML model cards. Each category is represented by a horizontal bar segmented into three colors: blue ("Detailed"), orange ("Mentioned Only"), and red ("Absent"). The x-axis quantifies documentation coverage (0–100), while the y-axis lists categories in descending order of "Detailed" documentation.
### Components/Axes
- **X-axis**: "Number of Model Cards" (0–100, linear scale)
- **Y-axis**: Categories (Model Architecture, Compute Requirements, Evaluation Metrics, License, Intended Use, Training Data, Limitations, Bias Fairness, Safety Evaluation, Out Of Scope, Interpretability)
- **Legend**:
- Blue = Detailed
- Orange = Mentioned Only
- Red = Absent
- **Spatial Layout**:
- Legend positioned in the top-left corner
- Bars aligned horizontally, descending from top (Model Architecture) to bottom (Interpretability)
### Detailed Analysis
1. **Model Architecture**:
- Detailed: ~75 (blue)
- Mentioned Only: ~10 (orange)
- Absent: ~15 (red)
2. **Compute Requirements**:
- Detailed: ~75 (blue)
- Mentioned Only: ~10 (orange)
- Absent: ~15 (red)
3. **Evaluation Metrics**:
- Detailed: ~70 (blue)
- Mentioned Only: ~10 (orange)
- Absent: ~20 (red)
4. **License**:
- Detailed: ~65 (blue)
- Mentioned Only: ~5 (orange)
- Absent: ~30 (red)
5. **Intended Use**:
- Detailed: ~60 (blue)
- Mentioned Only: ~5 (orange)
- Absent: ~35 (red)
6. **Training Data**:
- Detailed: ~55 (blue)
- Mentioned Only: ~5 (orange)
- Absent: ~40 (red)
7. **Limitations**:
- Detailed: ~55 (blue)
- Mentioned Only: ~5 (orange)
- Absent: ~40 (red)
8. **Bias Fairness**:
- Detailed: ~50 (blue)
- Mentioned Only: ~5 (orange)
- Absent: ~45 (red)
9. **Safety Evaluation**:
- Detailed: ~45 (blue)
- Mentioned Only: ~5 (orange)
- Absent: ~50 (red)
10. **Out Of Scope**:
- Detailed: ~45 (blue)
- Mentioned Only: ~5 (orange)
- Absent: ~50 (red)
11. **Interpretability**:
- Detailed: ~20 (blue)
- Mentioned Only: ~0 (orange)
- Absent: ~80 (red)
### Key Observations
- **Trend Verification**:
- Blue bars ("Detailed") decrease monotonically from top to bottom (75 → 20).
- Red bars ("Absent") increase monotonically (15 → 80).
- Orange bars ("Mentioned Only") remain consistently low (~5–10) except for License (~5) and Intended Use (~5).
- **Notable Outliers**:
- Interpretability has the lowest "Detailed" documentation (~20) and highest "Absent" (~80).
- License and Intended Use show minimal "Mentioned Only" documentation.
### Interpretation
The data suggests a systematic decline in detailed documentation quality as categories progress from technical specifications (Model Architecture) to ethical considerations (Interpretability). Critical gaps appear in:
1. **Ethical/Ethical-Related Categories**:
- Bias Fairness, Safety Evaluation, and Interpretability show significant "Absent" documentation (>45).
2. **Implementation-Specific Categories**:
- Compute Requirements and Evaluation Metrics maintain high "Detailed" coverage but lack depth in "Mentioned Only" segments.
3. **Structural Gaps**:
- License and Interpretability categories exhibit the largest documentation voids, raising concerns about compliance and model transparency.
The chart underscores a prioritization of technical documentation over ethical and operational aspects, with Interpretability being the most neglected area. This pattern may reflect resource allocation biases or underestimation of interpretability's importance in model deployment.
</details>
Figure 2: Documentation completeness by category across 100 Hugging Face model cards. Technical details are well-documented while safety-critical information is frequently absent.
Figure 2 shows the distribution of documentation depth across all 100 model cards. Technical categories such as Model Architecture, Compute Requirements, and Evaluation Metrics were most consistently documented in detail. In contrast, safety-critical categories exhibited significant gaps: Interpretability was detailed in fewer than 20% of model cards and absent in over 80%. Safety Evaluation, Bias and Fairness, and Limitations were frequently mentioned but rarely described in depth.
## IV Proposed Approach
### IV-A Defining a Minimal Core Documentation Schema
We propose establishing a minimal consensus core schema that layers on top of existing approaches such as model cards, data sheets, and system cards. This schema would specify 20–30 essential fields that all high-impact models should report, including intended use, primary benchmarks (with exact benchmark names), high-level training data types, and known risk domains.
<details>
<summary>variation.png Details</summary>
### Visual Description
## Bar Chart: Naming Variation by Concept (Inconsistency Measure)
### Overview
The chart visualizes the inconsistency in naming across eight technical documentation concepts. Bars represent the number of different section names used for each concept, with "usage" showing the highest inconsistency (~98) and "license" the lowest (~8). All bars are purple, and the legend is positioned on the right.
### Components/Axes
- **X-axis (Categories)**:
`model_info`, `evaluation`, `usage`, `license`, `citation`, `limitations`, `safety`, `training`
- **Y-axis (Values)**:
Labeled "Number of Different Section Names" with a scale from 0 to 100.
- **Legend**:
Single entry for "Number of Different Section Names" (purple bars).
### Detailed Analysis
- **model_info**: ~12 different names (low inconsistency).
- **evaluation**: ~52 different names (moderate-high inconsistency).
- **usage**: ~98 different names (highest inconsistency).
- **license**: ~8 different names (lowest inconsistency).
- **citation**: ~15 different names (low inconsistency).
- **limitations**: ~21 different names (moderate inconsistency).
- **safety**: ~10 different names (low inconsistency).
- **training**: ~38 different names (moderate inconsistency).
### Key Observations
1. **Usage** dominates with ~98 different names, suggesting extreme variability in terminology.
2. **Evaluation** and **training** follow as the next most inconsistent concepts (~52 and ~38, respectively).
3. **License**, **safety**, and **citation** show the least inconsistency (~8–15 names).
4. **Limitations** (~21) and **model_info** (~12) fall in the mid-to-low range.
### Interpretation
The data highlights significant naming inconsistency in **usage** sections, potentially reflecting ad-hoc or context-dependent terminology. **Evaluation** and **training** also show notable variability, possibly due to iterative or audience-specific naming conventions. In contrast, **license** and **safety** sections exhibit standardized naming, likely due to regulatory or compliance requirements. The stark contrast between **usage** (~98) and **license** (~8) underscores the need for clearer naming guidelines in high-variability areas. The uniform purple color across bars simplifies interpretation but limits differentiation between concepts.
## Peircean Investigative Insight
The chart invites further inquiry into why **usage** sections exhibit such high inconsistency. Are these sections more prone to subjective interpretation, or do they lack centralized oversight? The low inconsistency in **license** and **safety** suggests these areas benefit from formalized standards, offering a model for reducing variability elsewhere. The absence of a secondary color scheme (e.g., for severity levels) leaves room for deeper analysis of root causes.
</details>
Figure 3: Naming variation across core documentation concepts. Usage information appears under 97 different section names, demonstrating severe inconsistency in model card structure
Our analysis identified 947 unique section names across all cards. We used fuzzy matching to group semantically similar section names by core concept (e.g., all variations describing model usage, evaluation, or training). Figure 3 shows the number of different section names used for each concept. Usage-related information showed the highest variation with 97 different section names, followed by evaluation (52 variations) and training (38 variations). Even fundamental concepts like license information appeared under 8 different names. This extreme variability makes it nearly impossible to systematically extract and compare information across models, reinforcing the need for a stable and shared documentation schema.
We also found that:
- Safety-critical fields (e.g., hallucination behavior, jailbreak resistance, cyber risk) show the largest deficits (Figure 4).
- Providers differ widely in average compliance (Figure 5).
- Weighted scoring exposes high-impact gaps that unweighted scoring fails to surface.
<details>
<summary>gap-comply.png Details</summary>
### Visual Description
## Bar Chart: 15 Highest-Impact Compliance Gaps
### Overview
This horizontal bar chart visualizes the 15 highest-impact compliance gaps across AI models, ranked by total points lost. The x-axis represents "Total Points Lost Across All Models," while the y-axis lists compliance gaps with associated point values and severity percentages. Colors transition from dark red (highest severity) to green (lowest severity).
### Components/Axes
- **X-Axis**: "Total Points Lost Across All Models" (linear scale, 0–160).
- **Y-Axis**: 15 compliance gaps, ordered by points lost (highest to lowest).
- **Legend**: Implicit color gradient (dark red = critical, green = lower priority).
### Detailed Analysis
1. **Highest Severity (Dark Red/Orange)**:
- **Deception Behaviors (4.0 pts)**: 148 pts (23%)
- **Hallucinations (4.0 pts)**: 124 pts (35%)
- **Child Safety Evaluations (4.0 pts)**: 116 pts (40%)
- **Jailbreak (4.0 pts)**: 104 pts (46%)
- **Cyber Risk (5.0 pts)**: 100 pts (58%)
2. **Mid-Range Severity (Yellow/Green)**:
- **Sycophancy (2.0 pts)**: 90 pts (6%)
- **Knowledge Count (2.0 pts)**: 68 pts (29%)
- **Out-of-scope use cases (3.0 pts)**: 48 pts (67%)
- **Training Data Processing (6.0 pts)**: 48 pts (83%)
- **Privacy Risks (2.0 pts)**: 46 pts (52%)
- **Fairness & Bias Evaluations (incl. BBQ) (3.0 pts)**: 45 pts (69%)
3. **Lower Severity (Light Green)**:
- **Disallowed Content Handling (4.0 pts)**: 44 pts (77%)
- **Harmful Manipulation (4.0 pts)**: 44 pts (77%)
- **Adversarial Robustness (2.0 pts)**: 40 pts (58%)
- **Risk Mitigations (4.0 pts)**: 40 pts (79%)
### Key Observations
- **Deception Behaviors** is the most critical gap, losing the most points (148) despite a relatively low point value (4.0).
- **Cyber Risk** (5.0 pts) has the highest percentage of total points lost (58%), indicating disproportionate impact relative to its point value.
- **Training Data Processing** (6.0 pts) and **Out-of-scope use cases** (3.0 pts) show high percentages (83% and 67%, respectively), suggesting systemic issues in data handling.
- **Risk Mitigations** and **Adversarial Robustness** (both 40 pts) are the least severe but still represent notable gaps.
### Interpretation
The chart reveals that **higher point-value gaps** (e.g., Cyber Risk at 5.0 pts) often correlate with **higher percentages of total points lost**, highlighting their outsized impact. Conversely, gaps like **Training Data Processing** (6.0 pts) and **Out-of-scope use cases** (3.0 pts) demonstrate that even lower-point categories can represent significant systemic failures. The color gradient underscores prioritization: dark red/orange gaps require immediate attention, while green gaps may warrant long-term monitoring. The data suggests compliance frameworks should focus on high-impact, high-severity areas first, while addressing lower-severity gaps through iterative improvements.
</details>
Figure 4: Aggregate point loss by subsection across all models. Higher point loss indicates categories where documentation gaps have the greatest impact on overall transparency scores due to their weight and prevalence of absence.
### IV-B Introduce a Documentation Benchmarking Scorecard
This core schema is implemented as a framework that consists of 8 main sections and 23 subsections (see Table I). The framework synthesizes common documentation categories we identified across existing models, aligns them with EU AI Act Annex IV requirements and the Stanford Transparency Index, and prioritizes fields based on their importance for safety and governance [eu_ai_act_annex_iv, bommasani2024fmti]. Our framework assigns weighted scores to each subsection, with weights reflecting the relative importance of different disclosure types. Safety-critical information receives substantially higher weights: Safety Evaluation (25%), Critical Risk (20%), and Model Data (15%) together account for 60% of the total score, while technical specifications like Model Implementation and Sustainability (5%) and Risk Mitigations (4%) receive lower weights. By emphasizing safety-critical disclosures, the scorecard highlights governance-relevant gaps that unweighted scoring obscures.
<details>
<summary>compliance.jpg Details</summary>
### Visual Description
## Bar Chart: Compliance by Provider (Weighted)
### Overview
The chart visualizes the average weighted compliance percentages of various AI providers, ranked from highest to lowest compliance. Compliance is represented by horizontal bars transitioning from green (high compliance) to orange (low compliance), with numerical values displayed at the end of each bar.
### Components/Axes
- **X-Axis**: "Average Weighted Compliance (%)" (0–100 scale)
- **Y-Axis**: List of AI providers (ordered by compliance, top to bottom)
- **Legend**: Implied color gradient (green = high compliance, orange = low compliance)
- **Key Labels**: Provider names and compliance percentages (e.g., "xAI: 80.5%")
### Detailed Analysis
1. **Top Performers (Green Bars)**:
- xAI: 80.5%
- Microsoft: 79.5%
- Anthropic: 79.5%
- Meta AI: 79.0%
- DeepSeek-AI: 78.0%
- OpenAI: 76.6%
- Google: 76.2%
2. **Mid-Tier Providers (Yellow-Green Bars)**:
- Stability AI: 64.0%
- MoonshotAI: 64.0%
- Alibaba: 61.0%
- HuggingFaceH4: 58.5%
- BRIA AI: 58.0%
- Mistral AI: 56.8%
- Qwen: 56.0%
- Iliyasviel: 54.0%
- Tencent: 52.5%
3. **Lower Compliance (Yellow-Orange Bars)**:
- 2Noise: 49.5%
- Lighttricks: 49.0%
- TII UAE: 48.5%
- Stepfun AI: 48.0%
- ByteDance: 47.5%
- Docling Project: 45.0%
- Black Forest Labs: 40.3%
### Key Observations
- **Dominance of Major Providers**: The top 7 providers (xAI, Microsoft, Anthropic, Meta AI, DeepSeek-AI, OpenAI, Google) all exceed 76% compliance, with xAI leading at 80.5%.
- **Significant Drop-Off**: Compliance drops sharply after the top 7, with Stability AI and MoonshotAI at 64%.
- **Lowest Compliance**: Black Forest Labs (40.3%) and Docling Project (45.0%) are the only providers below 50%.
- **Color Gradient**: Green dominates the top 7, yellow-green covers mid-tier providers, and orange-yellow represents the lowest performers.
### Interpretation
The chart highlights a clear hierarchy in compliance among AI providers, with major players like xAI and Microsoft demonstrating significantly higher adherence to standards compared to smaller or newer entities. The weighted average suggests compliance may be influenced by factors such as organizational maturity, regulatory pressure, or resource allocation. The stark contrast between the top performers and lower-tier providers (e.g., Black Forest Labs at 40.3%) raises questions about industry-wide challenges in meeting compliance benchmarks. The color gradient effectively communicates performance tiers, though the absence of an explicit legend leaves room for ambiguity in interpreting the exact meaning of the color transitions.
</details>
Figure 5: Weighted compliance scores by provider. Major frontier labs (xAI, Microsoft, Anthropic, Meta) achieve 80% compliance, while smaller providers range from 40-65%.
| Model Details (15%) | Model overview | 3 |
| --- | --- | --- |
| Organization developing the model | 1 | |
| Model Version | 2 | |
| Model Release Date | 0.5 | |
| Model Version Progression | 1 | |
| Model Architecture | 4 | |
| Model Dependencies | 1 | |
| Paper and relevant links | 0.5 | |
| Model Distribution Forms | 2 | |
| Model Inputs & Outputs (6%) | Inputs | 2 |
| Outputs | 2 | |
| Token Count | 2 | |
| Model Data (15%) | Training Dataset | 7 |
| Training Data Processing | 6 | |
| Knowledge Count | 2 | |
| Model Implementation and Sustainability (5%) | Hardware Used During Training & Inference | 2 |
| Software Frameworks & Tooling | 2 | |
| Energy Use / Sustainability Metrics | 1 | |
| Intended Use (10%) | Primary intended uses | 5 |
| Primary intended users | 2 | |
| Out-of-scope use cases | 3 | |
| Critical Risk (20%) | CBRN (Chemical, Biological, Radiological or Nuclear) | 5 |
| Cyber Risk | 5 | |
| Harmful Manipulation | 4 | |
| Child Safety Evaluations | 4 | |
| Privacy Risks | 2 | |
| Safety Evaluation (25%) | Refusals | 1 |
| Disallowed Content Handling | 4 | |
| Sycophancy | 2 | |
| Jailbreak | 4 | |
| Hallucinations | 4 | |
| Deception Behaviors | 4 | |
| Fairness & Bias Evaluations (incl. BBQ) | 3 | |
| Adversarial Robustness | 2 | |
| Red Teaming Results | 1 | |
| Risk Mitigations (4%) | Risk Mitigation | 4 |
TABLE I: Proposed Documentation Transparency Framework: Sections, Subsections, and Scores
### IV-C Framework Objectives and Weighting Approach
Our framework targets the two largest deficiencies in current AI documentation ecosystems: consistency and evaluability. Rather than adding yet another lengthy template, it distills the essential fields that matter for oversight and aligns them with existing regulatory expectations, making adoption practical for developers and meaningful for policymakers.
Our weighted scoring method also addresses the problem of treating all documentation fields as equally important. By weighting safety-critical disclosures more heavily, the scorecard highlights the gaps that matter most for governance. In our evaluation, this method surfaced high-risk blind spots that were invisible under unweighted scoring, producing a clearer and more policy-relevant picture of model transparency.
We recommend requiring AI developers to publish model and system documentation not only as PDFs or web pages but also as machine-readable JSON following an open, extensible schema. This schema would encode core fields such as model name and version, training data summary, evaluation benchmarks, known limitations, and implemented safety mitigations. Machine-readable documentation enables researchers, regulators, and civil-society organizations to programmatically ingest, compare, and audit models at scale.
In developing our methodology, we also examined several plausible approaches for improving AI documentation consistency and transparency. Each alternative offers conceptual advantages but also reveals structural limitations that constrain its practical effectiveness.
- Developing a comprehensive new documentation framework. One approach is to design a fully new documentation standard that supersedes existing model cards, data sheets, and system cards. While appealing in terms of conceptual clarity, this strategy adds another framework to an already crowded landscape. Creating a standalone standard risks poor adoption, limited interoperability with existing tools, and slow industry uptake. It also overlooks areas where partial convergence already exists, such as shared fields across model cards and system cards.
- Strengthening voluntary narrative guidelines. Another approach is to rely on high-level best-practice recommendations encouraging developers to “write better model cards.” Although lightweight and easy to disseminate, purely narrative guidance does not correct the structural issues observed in our empirical analysis: inconsistent field definitions, lack of machine-readable formats, and wide variability in documentation depth. Moreover, voluntary guidelines provide weak incentives and no mechanism for automated evaluation, limiting their ability to drive meaningful improvement.
- Establishing an immediate, globally harmonized regulatory standard. A third approach is to push for a fully harmonized international standard for AI documentation. While this could theoretically provide a unified foundation, it is politically and technically difficult to implement in the near term. Regulatory ecosystems differ across jurisdictions, and AI systems evolve faster than global regulatory consensus can form. Such an approach risks regulatory stagnation or adoption of a lowest-common-denominator standard. As a result, it provides limited near-term utility for practitioners or policymakers.
<details>
<summary>system-design.jpg Details</summary>
### Visual Description
## Flowchart: Model Card Framework and Query Generation Process
### Overview
This flowchart illustrates a technical workflow for generating and validating queries using a model card framework, integrating external data sources, and leveraging multiple language models (LLMs) for consensus-based decision-making. The process is cyclical, with feedback loops between components.
### Components/Axes
1. **Start** (Yellow Box): Initiates the workflow.
2. **Source of Truth** (Green Box): Contains the "Our New Model Card Framework."
3. **Query Generation** (Orange Box): Generates queries based on selected model sections (e.g., safety).
4. **Perplexity Search API** (Purple Box): Searches the internet for safety data and subsections of models like Claude Sonnet 4, returning results in chunks with sources.
5. **Populate the Framework** (Cyan Box): Updates the model card framework using consensus outputs.
6. **Consensus: Majority Vote** (Pink Box): Aggregates outputs from three LLMs (LLM1, LLM2, LLM3) to determine the final result.
7. **LLMs** (Colored Circles):
- LLM1 (Green)
- LLM2 (Yellow)
- LLM3 (Gray)
- Outputs are scored based on the new model card.
### Detailed Analysis
- **Flow Direction**:
- Starts at "Start" → "Source of Truth" → "Our New Model Card Framework."
- Branches to "Query Generation," which connects to the "Perplexity Search API."
- API results feed back into "Query Generation," which then routes outputs to all three LLMs.
- LLM outputs are aggregated via "Consensus: Majority Vote" to "Populate the Framework," which loops back to the "Source of Truth."
- **Key Connections**:
- Arrows labeled "Select a section e.g. Safety" link the model card to query generation.
- Arrows labeled "Score based on our new model card" connect LLMs to query generation.
- Feedback loop from "Populate the Framework" to "Source of Truth" ensures iterative refinement.
### Key Observations
- **Cyclical Process**: The workflow is designed for continuous improvement, with outputs from the consensus step directly updating the foundational model card.
- **External Data Integration**: The Perplexity Search API introduces real-time, external data (e.g., safety information) to inform query generation.
- **Multi-LLM Consensus**: Three distinct LLMs contribute to decision-making, with majority voting ensuring robustness.
### Interpretation
This diagram represents a hybrid AI system that combines:
1. **Internal Knowledge** (model card framework) with **External Data** (Perplexity API) to generate context-aware queries.
2. **Model Diversity** (three LLMs) to mitigate individual model biases, using consensus to enhance reliability.
3. **Iterative Refinement**, where outputs from the consensus step directly improve the foundational model card, creating a self-improving loop.
The use of color-coding (e.g., green for safety, purple for external search) suggests a modular design where each component has a distinct role. The absence of numerical values implies this is a conceptual workflow rather than a data-driven analysis. The process emphasizes transparency (via source attribution in API results) and adaptability (through iterative framework updates).
</details>
Figure 6: Overview of our automated data extraction and scoring pipeline. Starting from our model card framework (source of truth), the system selects a model, generates targeted queries for each section, retrieves evidence from the Perplexity Search API, evaluates the results using multiple LLMs through majority-vote consensus, and populates the standardized model card schema.
## V Automated Documentation Extraction Pipeline
Manual evaluation of model documentation against our framework would be time-consuming and not scalable across hundreds of models. To address this limitation, we developed an automated agentic pipeline that extracts, evaluates, and scores documentation using a multi-agent LLM system with web search capabilities [AI_transparency_atlas_2025]. This approach builds on recent work by Liu et al. [liu2024automaticgenerationmodeldata], who demonstrated the feasibility of using LLMs to automatically generate model and data cards. While their work focuses on creating documentation artifacts, our pipeline evaluates and scores existing documentation by extracting information from dispersed public sources.
### V-A Pipeline Architecture
Figure 6 illustrates the overall pipeline architecture. The process begins by selecting a model from our dataset (e.g., Claude Sonnet 4, GPT-5, Llama 4). For each selected model, the pipeline iterates through the eight main sections and 23 subsections defined in our framework (see Table I).
### V-B Query Generation and Information Retrieval
For each subsection (e.g., Safety Evaluation $\rightarrow$ Jailbreak), the pipeline generates targeted search queries designed to retrieve relevant documentation from publicly available sources. We use the Perplexity Search API to collect evidence from model cards, system cards, technical reports, blog posts, and GitHub repositories.
This design reflects real-world auditing conditions, as it evaluates only information accessible to external auditors, policymakers, and users without direct cooperation from model developers. Search results are returned in structured chunks with source citations, preserving traceability and transparency.
### V-C Multi-Agent Consensus Scoring
Retrieved documentation is independently evaluated by three LLM agents (LLM 1, LLM 2, LLM 3). Each agent assesses documentation completeness for a given subsection according to our framework criteria and assigns one of the following labels:
- Detailed: Substantive, specific, and actionable information
- Mentioned: Present but superficial or vague information
- Absent: No relevant information found
The final score for each subsection is determined using majority-vote consensus across the three agents.
### V-D Framework Population
After all subsections are evaluated, the pipeline populates the framework with the extracted evidence and consensus scores. Subsection scores are then aggregated using the weighting scheme defined in Table I to produce an overall transparency rating for each model.
### V-E Scalability and Cost
The pipeline is fully automated and cost-efficient. Evaluating all 50 models across 23 subsections cost less than $3 total (under $0.06 per model), enabling large-scale analysis across hundreds of models. We validated the pipeline on models spanning vision, multimodal, open-source, and closed-source systems, demonstrating robustness across diverse documentation formats and styles.
## VI Discussion, Limitations, and Future Work
Our framework enables model providers to assess their documentation transparency in real-time. Providers can access a live dashboard showing their overall transparency score and subsection-level breakdowns, immediately identifying gaps in their documentation. For example, a provider scoring high on technical specifications (Model Architecture, Compute Requirements) but low on safety-critical categories (Jailbreak Evaluations, Child Safety) can prioritize improvements in those areas.
Our pipeline offers a pragmatic middle ground: instead of requiring providers to restructure their documentation, we automatically aggregate information from dispersed sources and present it through a standardized evaluation framework. This enables consistent cross-model comparison even when underlying documentation practices vary, lowering adoption barriers while improving transparency assessment for regulators, researchers, and users.
A potential limitation of any weighted scoring system is that providers may optimize for score maximization rather than genuine transparency. Since our framework assigns higher weights to safety-critical categories (Safety Evaluation: 25%, Critical Risk: 20%), providers could strategically prioritize these sections to boost scores while neglecting lower-weighted but still important disclosures.
However, several design features mitigate this risk. First, our framework requires substantive, detailed information—not just the presence of a section. Surface-level statements that mention safety evaluations without specific methodologies, results, or limitations receive lower scores than comprehensive disclosures. Second, the multi-agent consensus mechanism evaluates documentation quality, not just existence, making it harder to game through minimal compliance. Third, transparency scores are most valuable when used comparatively and longitudinally; providers who inflate scores through selective disclosure will be evident when their documentation is examined alongside peers or tracked over time.
Beyond point-in-time evaluation, our structured representation of publicly available documentation enables systematic tracking of documentation changes across model versions. By normalizing dispersed online disclosures into a consistent schema, the framework creates a stable reference that can be compared longitudinally as models evolve. While the current study focuses on static snapshots, we plan to extend this pipeline to support version-aware analysis, allowing transparency scores and subsection-level disclosures to be tracked across releases. This would enable stakeholders to monitor how safety claims, evaluation practices, and risk disclosures change over time, and to identify regressions or improvements in transparency as models are updated.
## VII Conclusion
AI model documentation today is fragmented, inconsistent, and insufficient for meaningful oversight. Information is scattered across platforms, section naming varies wildly across providers, and safety-critical disclosures are frequently absent or superficial. These gaps undermine the ability of regulators, researchers, and downstream users to assess model risks, compare alternatives, and make informed decisions. This paper presented a structured response to these challenges. Through empirical analysis of frontier model documentation and 100 Hugging Face model cards, we identified systematic inconsistencies in how models are documented. We developed a transparency framework with 8 sections and 23 subsections, grounded in the EU AI Act Annex IV and the Stanford Transparency Index, with weighted scoring that prioritizes safety-critical information over technical specifications.To operationalize this framework at scale, we built an automated agentic pipeline that extracts documentation from public sources, evaluates completeness using multi-agent consensus, and generates transparency scores. The pipeline cost less than $3 to evaluate 50 diverse models, demonstrating economic feasibility for continuous, large-scale monitoring. Our evaluation reveals significant transparency gaps across the ecosystem. While frontier labs like xAI, Microsoft, and Anthropic achieve 80% compliance, many smaller providers fall below 50%. Categories like Interpretability and Safety Evaluation—critical for governance—remain poorly documented across most models. The framework offers practical value for multiple stakeholders. Providers can use live dashboards to identify documentation gaps and track improvements over time. Regulators gain evidence-based tools for compliance assessment without manual audits. Researchers obtain standardized metrics for cross-model comparison. Users gain visibility into which models have comprehensive safety documentation. Moving forward, transparency in AI cannot rely solely on voluntary adoption of documentation standards. Our approach demonstrates that automated extraction and standardized evaluation can bridge the gap between current fragmented practices and the structured information needed for accountability. As AI systems become more capable and widely deployed, robust documentation transparency is not optional—it is foundational to responsible governance.