# AI Transparency Atlas: Framework, Scoring, and Real-Time Model Card Evaluation Pipeline
**Authors**: Akhmadillo Mamirov1, Faiaz Azmain1, Hanyu Wang
Abstract
AI model documentation is fragmented across platforms and inconsistent in structure, preventing policymakers, auditors, and users from reliably assessing safety claims, data provenance, and version changes. We analyzed documentation from five frontier models (Gemini 3, Grok 4.1, Llama 4, GPT-5, Claude 4.5) and 100 Hugging Face model cards, identifying 947 unique section names with extreme naming variation—usage information alone appeared under 97 different labels. Using the EU AI Act Annex IV and Stanford Transparency Index as baselines, we developed a weighted transparency framework with 8 sections and 23 subsections that prioritizes safety-critical disclosures (Safety Evaluation: 25%, Critical Risk: 20%) over technical specifications. We implemented an automated multi-agent pipeline that extracts documentation from public sources and scores completeness through LLM consensus. Evaluating 50 models across vision, multimodal, open-source, and closed-source systems cost less than $3 total and revealed systematic gaps: frontier labs (xAI, Microsoft, Anthropic) achieve 80% compliance, while most providers fall below 60%. Safety-critical categories show the largest deficits—deception behaviors, hallucinations, and child safety evaluations account for 148, 124, and 116 aggregate points lost respectively across all evaluated models.
I Introduction
AI model documentation today is fragmented across whitepapers, GitHub READMEs, Hugging Face model cards, system cards, and blog posts. This fragmentation raises a core question: what practical steps can move the ecosystem from documentation inconsistency toward something standardized enough to be useful?
Documentation gaps affect every stakeholder. Regulators cannot reliably assess governance or safety without consistent reporting. Downstream institutions such as hospitals, schools, and public agencies lack visibility into model risks, evaluation protocols, and version-level changes. Even within the same platform, model cards differ dramatically in length, scope, and granularity. Some include details about architecture, training data, and evaluation settings, while others provide only brief paragraphs. Critically, documentation often does not evolve as models evolve. Major capability updates, training adjustments, and safety interventions rarely trigger corresponding updates. Versioning is ad hoc or entirely absent, causing transparency to degrade over time.
System cards attempt to address transparency at the deployment level, but they introduce their own challenges. Many system cards are high-level but not actionable; others remain closed, incomplete, or not understandable to external auditors. When modern AI systems depend on chains of interconnected models, datasets, and processes, opacity at the system layer becomes a structural barrier to accountability. When something goes wrong, responding is difficult because relevant information is scattered across multiple documents and repositories [winecoff2024improvinggovernanceoutcomesai].
I-A Why Does This Inconsistency Persist?
The persistence of fragmented AI documentation reflects structural challenges in the AI development ecosystem. Unlike regulated industries where documentation is mandatory and enforcement mechanisms are well established, AI transparency remains largely voluntary, resulting in misaligned incentives.
Economic and competitive pressures. Comprehensive documentation is resource-intensive, requiring dedicated teams for safety evaluations, data provenance tracking, and version management. Organizations prioritizing rapid deployment often treat documentation as secondary to product development. Closed-source developers face additional tension: detailed transparency can expose competitive advantages related to training methods, data sources, or architectural choices. As a result, disclosure decisions frequently involve trade-offs between transparency commitments and intellectual property protection.
Organizational fragmentation. High-quality documentation requires coordination across teams that typically operate independently. Engineers prioritize model performance, safety teams focus on risk assessment, and communications teams manage external messaging. Without integrated workflows that treat documentation as a natural byproduct of development, information remains siloed across internal wikis, isolated reports, and fragmented public communications.
Competing standards. Developers encounter overlapping documentation proposals, including Model Cards, Datasheets for Datasets, System Cards, and emerging regulatory frameworks, each emphasizing different priorities and formats. In the absence of a unified standard specifying required content, level of detail, and structure, developers make inconsistent choices. Some release brief model cards, while others publish extensive technical reports. Neither approach consistently satisfies stakeholder needs because no authoritative standard exists.
Limited accountability. Documentation quality improves when it can be credibly evaluated. Currently, users lack practical means to assess completeness or accuracy, regulators lack scalable audit tools, and civil society organizations can identify only the most visible gaps. Without mechanisms to systematically measure and publicly compare documentation quality, providers face limited accountability for incomplete or uneven disclosures.
I-B Our Approach Builds on Existing Regulatory and Academic Frameworks.
We address these challenges through a structured transparency framework grounded in existing regulatory and academic standards. The EU AI Act Annex IV provides concrete documentation requirements that we adopt as a regulatory baseline [eu_ai_act_annex_iv]. We use the Stanford Transparency Index as an academic reference point for evaluating disclosure completeness across AI providers [bommasani2024fmti].
To populate this framework at scale, we built an automated pipeline that extracts documentation from dispersed public sources, evaluates completeness using multi-agent LLM consensus, and generates transparency scores. This approach enables systematic, continuous assessment of documentation quality across hundreds of models without requiring direct developer cooperation.
II Background and Related Work
The Foundation Model Transparency Index (May 2024) by Stanford University provides a comprehensive assessment of foundation model developer transparency [bommasani2024fmti]. The index evaluates disclosure practices across major AI labs using data provided directly by developers. While this approach offers depth and accuracy for covered models, it relies on developer cooperation and self-reported information. Our work takes a complementary approach: we evaluate documentation completeness using only publicly available information online—the same information accessible to regulators, auditors, and downstream users in practice. This distinction is critical because real-world transparency depends on what stakeholders can actually access and verify, not just what developers are willing to share upon request.
<details>
<summary>sections.png Details</summary>
### Visual Description
## Horizontal Bar Chart: Information Category Presence Across Model Cards
### Overview
The image is a horizontal bar chart displaying the percentage of model cards that include specific information categories. The categories are listed on the vertical axis, and the percentage of model cards is on the horizontal axis. A vertical dashed line indicates a 50% threshold. The bars are colored green, orange, and red, with green indicating high presence, orange indicating moderate presence, and red indicating low presence.
### Components/Axes
* **Title:** Information Category Presence Across Model Cards
* **Horizontal Axis:** Percentage of Model Cards, ranging from 0% to 80% in increments of 10%.
* **Vertical Axis:** Information Categories (Model Architecture, Evaluation Metrics, Compute Requirements, Intended Use, License, Limitations, Training Data, Bias Fairness, Safety Evaluation, Out Of Scope, Interpretability).
* **Legend:** "50% threshold" represented by a dashed gray line.
* **Bar Colors:**
* Green: Model Architecture, Evaluation Metrics, Compute Requirements
* Orange: Intended Use, License, Limitations, Training Data, Bias Fairness, Safety Evaluation
* Red: Out Of Scope, Interpretability
### Detailed Analysis
The chart presents the percentage of model cards that include each information category.
* **Model Architecture:** Approximately 83% (Green)
* **Evaluation Metrics:** Approximately 82% (Green)
* **Compute Requirements:** Approximately 80% (Green)
* **Intended Use:** Approximately 65% (Orange)
* **License:** Approximately 64% (Orange)
* **Limitations:** Approximately 60% (Orange)
* **Training Data:** Approximately 58% (Orange)
* **Bias Fairness:** Approximately 54% (Orange)
* **Safety Evaluation:** Approximately 51% (Orange)
* **Out Of Scope:** Approximately 45% (Red)
* **Interpretability:** Approximately 18% (Red)
### Key Observations
* Model Architecture, Evaluation Metrics, and Compute Requirements are present in over 80% of model cards.
* Intended Use, License, Limitations, Training Data, Bias Fairness, and Safety Evaluation are present in between 50% and 65% of model cards.
* Out Of Scope and Interpretability are present in less than 50% of model cards.
* Interpretability has the lowest presence, at approximately 18%.
### Interpretation
The data suggests that certain information categories, such as Model Architecture, Evaluation Metrics, and Compute Requirements, are considered standard and are consistently included in model cards. Other categories, like Intended Use, License, and Limitations, are included in a moderate percentage of model cards. Notably, information regarding "Out Of Scope" and "Interpretability" is less frequently included, with "Interpretability" being the least represented category. This could indicate a lack of emphasis on these aspects in model documentation or potential challenges in defining and communicating them effectively. The 50% threshold line provides a visual reference for quickly assessing which categories are included in a majority of model cards.
</details>
Figure 1: Category presence rates across 100 model cards. Technical categories exceed 90% presence while safety-critical categories often fall below 50%
III CURRENT STATE OF AI MODEL DOCUMENTATION
III-A Frontier Model Documentation Review
To ground our framework in existing practice, we conducted a structured review of recent documentation artifacts from leading AI developers, including Google (Gemini 3), xAI (Grok 4.1), Meta (Llama 4), OpenAI (GPT–5), and Anthropic (Claude 4.5). We also compared these against the original Model Cards proposal by Mitchell et al. (2019), which remains the conceptual baseline [DBLP:journals/corr/abs-1810-03993].
Across these models, we observed major differences in scope, depth, structure, and emphasis:
- Google’s Gemini 3 includes detailed sections on training data, architecture, sustainability, safety evaluations, and frontier-safety considerations [google_gemini3_modelcard].
- xAI’s Grok 4.1 concentrates heavily on refusal behavior, adversarial robustness, dual-use risk, and transparency around data and training [xai_grok4_1_modelcard].
- Meta’s Llama 4 emphasizes environmental footprint, quantization, safeguards, community governance, critical risks, and detailed fine-tuning behavior [meta_llama4_modelcard].
- OpenAI’s GPT-5 provides the most extensive safety-evaluation sections, including red-teaming by multiple external organizations, biological risk assessments, cybersecurity stress tests, deception and sycophancy evaluations, and system-level protections [openai_gpt5_system_card].
- Anthropic’s Claude 4.5 offers the most granular system-level reporting, covering agentic risks, alignment attempts, cyber ranges, interpretability studies, reward-hacking investigations, welfare assessments, and Responsibly Scaling Policy (RSP) aligned evaluations [anthropic_claude_sonnet45_system_card].
III-B Systematic Analysis of Hugging Face Model Cards
To assess whether these documentation patterns extend beyond frontier models, we analyzed 100 model cards from Hugging Face, spanning diverse model types, parameter scales, and deployment contexts [AI_documentation_analysis_2025]. This approach follows Liang et al. [liang2024whatsdocumentedaisystematic], who conducted a systematic analysis of 32,000 AI model cards and demonstrated that Hugging Face provides a representative sample of documentation practices across the broader AI ecosystem. From this broader sample, we identified 11 recurring documentation categories that appeared with varying frequency and depth: Model Architecture, Compute Requirements, Evaluation Metrics, License, Intended Use, Training Data, Limitations, Bias and Fairness, Safety Evaluation, Out-of-Scope Use, and Interpretability.
Each model card was evaluated according to three levels of completeness:
- Detailed: The category includes substantive information with specific details, metrics, or actionable content.
- Mentioned Only: The category is present but limited to high-level or superficial statements.
- Absent: The category is not addressed.
Figure 1 illustrates category presence rates across the dataset. While over 90% of model cards included Model Architecture, Evaluation Metrics, and Compute Requirements, several critical transparency indicators fell below the 50% threshold. Interpretability appeared in only approximately 20% of cards, and Out-of-Scope use cases in roughly 60%. Even when present, Safety Evaluation (approximately 65%) and Bias and Fairness (approximately 60%) were often addressed superficially.
<details>
<summary>completeness.png Details</summary>
### Visual Description
## Horizontal Bar Chart: Documentation Depth by Category
### Overview
The image is a horizontal bar chart illustrating the documentation depth for various categories related to model cards. The chart compares the number of model cards that have "Detailed," "Mentioned Only," or "Absent" documentation for each category. The categories are listed on the vertical axis, and the number of model cards is on the horizontal axis, ranging from 0 to 100.
### Components/Axes
* **Title:** Documentation Depth by Category
* **X-Axis:** Number of Model Cards (scale from 0 to 100, with increments of 20)
* **Y-Axis:** Categories (Model Architecture, Compute Requirements, Evaluation Metrics, License, Intended Use, Training Data, Limitations, Bias Fairness, Safety Evaluation, Out Of Scope, Interpretability)
* **Legend:** Located at the top-left of the chart.
* Blue: Detailed
* Yellow/Orange: Mentioned Only
* Red: Absent
### Detailed Analysis
Here's a breakdown of the documentation depth for each category:
* **Model Architecture:**
* Detailed (Blue): ~75
* Mentioned Only (Yellow/Orange): ~10
* Absent (Red): ~15
* **Compute Requirements:**
* Detailed (Blue): ~75
* Mentioned Only (Yellow/Orange): ~10
* Absent (Red): ~15
* **Evaluation Metrics:**
* Detailed (Blue): ~75
* Mentioned Only (Yellow/Orange): ~10
* Absent (Red): ~15
* **License:**
* Detailed (Blue): ~60
* Absent (Red): ~40
* **Intended Use:**
* Detailed (Blue): ~60
* Mentioned Only (Yellow/Orange): ~5
* Absent (Red): ~35
* **Training Data:**
* Detailed (Blue): ~60
* Mentioned Only (Yellow/Orange): ~5
* Absent (Red): ~35
* **Limitations:**
* Detailed (Blue): ~60
* Mentioned Only (Yellow/Orange): ~5
* Absent (Red): ~35
* **Bias Fairness:**
* Detailed (Blue): ~50
* Mentioned Only (Yellow/Orange): ~10
* Absent (Red): ~40
* **Safety Evaluation:**
* Detailed (Blue): ~50
* Mentioned Only (Yellow/Orange): ~10
* Absent (Red): ~40
* **Out Of Scope:**
* Detailed (Blue): ~45
* Mentioned Only (Yellow/Orange): ~5
* Absent (Red): ~50
* **Interpretability:**
* Detailed (Blue): ~15
* Absent (Red): ~85
### Key Observations
* "Model Architecture," "Compute Requirements," and "Evaluation Metrics" have the highest proportion of "Detailed" documentation.
* "Interpretability" has the lowest proportion of "Detailed" documentation and the highest proportion of "Absent" documentation.
* "License," "Intended Use," "Training Data," "Limitations," "Bias Fairness," and "Safety Evaluation" have a significant portion of "Absent" documentation.
* "Out of Scope" has nearly equal "Detailed" and "Absent" documentation.
### Interpretation
The chart reveals significant variations in documentation depth across different categories of model cards. Categories like "Model Architecture," "Compute Requirements," and "Evaluation Metrics" are generally well-documented, with a large proportion of model cards providing detailed information. In contrast, "Interpretability" is poorly documented, with most model cards lacking detailed information. This suggests that interpretability aspects of models are often overlooked or not thoroughly documented. The relatively high proportion of "Absent" documentation for categories like "License," "Intended Use," "Training Data," and "Limitations" indicates potential areas for improvement in model card documentation practices. The data highlights the need for a more consistent and comprehensive approach to documenting all relevant aspects of machine learning models.
</details>
Figure 2: Documentation completeness by category across 100 Hugging Face model cards. Technical details are well-documented while safety-critical information is frequently absent.
Figure 2 shows the distribution of documentation depth across all 100 model cards. Technical categories such as Model Architecture, Compute Requirements, and Evaluation Metrics were most consistently documented in detail. In contrast, safety-critical categories exhibited significant gaps: Interpretability was detailed in fewer than 20% of model cards and absent in over 80%. Safety Evaluation, Bias and Fairness, and Limitations were frequently mentioned but rarely described in depth.
IV Proposed Approach
IV-A Defining a Minimal Core Documentation Schema
We propose establishing a minimal consensus core schema that layers on top of existing approaches such as model cards, data sheets, and system cards. This schema would specify 20–30 essential fields that all high-impact models should report, including intended use, primary benchmarks (with exact benchmark names), high-level training data types, and known risk domains.
<details>
<summary>variation.png Details</summary>
### Visual Description
## Bar Chart: Naming Variation by Concept (Inconsistency Measure)
### Overview
The image is a bar chart illustrating the naming variation by concept, which serves as an inconsistency measure. The chart displays the number of different section names for various concepts, providing a visual representation of how consistently each concept is named across different sections.
### Components/Axes
* **Title:** Naming Variation by Concept (Inconsistency Measure)
* **X-axis:** Concepts (model\_info, evaluation, usage, license, citation, limitations, safety, training)
* **Y-axis:** Number of Different Section Names (scale from 0 to 100)
### Detailed Analysis
The bar chart presents the following data for each concept:
* **model\_info:** Approximately 12
* **evaluation:** Approximately 52
* **usage:** Approximately 97
* **license:** Approximately 8
* **citation:** Approximately 15
* **limitations:** Approximately 21
* **safety:** Approximately 10
* **training:** Approximately 38
### Key Observations
* The "usage" concept has the highest number of different section names, indicating the greatest inconsistency in naming.
* "license" and "safety" have the lowest number of different section names, suggesting higher consistency.
* There is a significant difference in naming variation between "usage" and other concepts.
### Interpretation
The chart suggests that the "usage" concept is named inconsistently across different sections, which could lead to confusion or difficulty in locating relevant information. Conversely, "license" and "safety" are named more consistently. This information can be used to improve the standardization of section names, particularly for concepts with high naming variation. The data highlights areas where documentation or naming conventions may need to be clarified or enforced to ensure consistency.
</details>
Figure 3: Naming variation across core documentation concepts. Usage information appears under 97 different section names, demonstrating severe inconsistency in model card structure
Our analysis identified 947 unique section names across all cards. We used fuzzy matching to group semantically similar section names by core concept (e.g., all variations describing model usage, evaluation, or training). Figure 3 shows the number of different section names used for each concept. Usage-related information showed the highest variation with 97 different section names, followed by evaluation (52 variations) and training (38 variations). Even fundamental concepts like license information appeared under 8 different names. This extreme variability makes it nearly impossible to systematically extract and compare information across models, reinforcing the need for a stable and shared documentation schema.
We also found that:
- Safety-critical fields (e.g., hallucination behavior, jailbreak resistance, cyber risk) show the largest deficits (Figure 4).
- Providers differ widely in average compliance (Figure 5).
- Weighted scoring exposes high-impact gaps that unweighted scoring fails to surface.
<details>
<summary>gap-comply.png Details</summary>
### Visual Description
## Horizontal Bar Chart: 15 Highest-Impact Compliance Gaps
### Overview
The image is a horizontal bar chart displaying the 15 highest-impact compliance gaps. The chart ranks these gaps based on the total points lost across all models, with the most points lost indicating critical priority areas. The bars are colored in a gradient from dark red (highest impact) to light green (lowest impact). Each bar is labeled with the compliance gap name, its associated point value, the total points lost, and the percentage of points lost.
### Components/Axes
* **Title:** 15 Highest-Impact Compliance Gaps (Most points lost = Critical priority areas)
* **X-axis:** Total Points Lost Across All Models. Scale ranges from 0 to 140, with tick marks at intervals of 20.
* **Y-axis:** Lists the compliance gaps. The gaps are:
* Deception Behaviors (4.0 pts)
* Hallucinations (4.0 pts)
* Child Safety Evaluations (4.0 pts)
* Jailbreak (4.0 pts)
* Cyber Risk (5.0 pts)
* Sycophancy (2.0 pts)
* Knowledge Count (2.0 pts)
* Out-of-scope use cases (3.0 pts)
* Training Data Processing (6.0 pts)
* Privacy Risks (2.0 pts)
* Fairness & Bias Evaluations (incl. BBQ) (3.0 pts)
* Disallowed Content Handling (4.0 pts)
* Harmful Manipulation (4.0 pts)
* Adversarial Robustness (2.0 pts)
* Risk Mitigations (4.0 pts)
### Detailed Analysis
The chart presents data for each compliance gap, including the points lost and the percentage of points lost. Here's a breakdown:
* **Deception Behaviors (4.0 pts):** 148 pts (23%) - Dark Red Bar
* **Hallucinations (4.0 pts):** 124 pts (35%) - Red-Orange Bar
* **Child Safety Evaluations (4.0 pts):** 116 pts (40%) - Orange Bar
* **Jailbreak (4.0 pts):** 104 pts (46%) - Orange-Yellow Bar
* **Cyber Risk (5.0 pts):** 100 pts (58%) - Yellow Bar
* **Sycophancy (2.0 pts):** 90 pts (6%) - Light Yellow Bar
* **Knowledge Count (2.0 pts):** 68 pts (29%) - Pale Yellow Bar
* **Out-of-scope use cases (3.0 pts):** 48 pts (67%) - Light Green Bar
* **Training Data Processing (6.0 pts):** 48 pts (83%) - Light Green Bar
* **Privacy Risks (2.0 pts):** 46 pts (52%) - Light Green Bar
* **Fairness & Bias Evaluations (incl. BBQ) (3.0 pts):** 45 pts (69%) - Light Green Bar
* **Disallowed Content Handling (4.0 pts):** 44 pts (77%) - Light Green Bar
* **Harmful Manipulation (4.0 pts):** 44 pts (77%) - Light Green Bar
* **Adversarial Robustness (2.0 pts):** 40 pts (58%) - Light Green Bar
* **Risk Mitigations (4.0 pts):** 40 pts (79%) - Light Green Bar
### Key Observations
* **Deception Behaviors** has the highest point loss (148 pts, 23%), making it the most critical priority area.
* The point loss decreases as you move down the chart, indicating a decreasing level of impact.
* The color gradient visually reinforces the ranking, with darker colors representing higher impact.
* Several compliance gaps have similar point losses (e.g., Out-of-scope use cases and Training Data Processing), suggesting they may require similar levels of attention.
### Interpretation
The chart effectively highlights the areas where compliance efforts should be focused. The "Deception Behaviors" category stands out as the most significant compliance gap, requiring immediate attention. The chart also allows for a comparative analysis of different compliance gaps, enabling prioritization based on the magnitude of point loss. The percentage of points lost does not appear to correlate strongly with the total points lost, suggesting that the total points lost is the primary driver of the ranking. The data suggests that focusing on the top few compliance gaps will likely yield the most significant improvements in overall compliance.
</details>
Figure 4: Aggregate point loss by subsection across all models. Higher point loss indicates categories where documentation gaps have the greatest impact on overall transparency scores due to their weight and prevalence of absence.
IV-B Introduce a Documentation Benchmarking Scorecard
This core schema is implemented as a framework that consists of 8 main sections and 23 subsections (see Table I). The framework synthesizes common documentation categories we identified across existing models, aligns them with EU AI Act Annex IV requirements and the Stanford Transparency Index, and prioritizes fields based on their importance for safety and governance [eu_ai_act_annex_iv, bommasani2024fmti]. Our framework assigns weighted scores to each subsection, with weights reflecting the relative importance of different disclosure types. Safety-critical information receives substantially higher weights: Safety Evaluation (25%), Critical Risk (20%), and Model Data (15%) together account for 60% of the total score, while technical specifications like Model Implementation and Sustainability (5%) and Risk Mitigations (4%) receive lower weights. By emphasizing safety-critical disclosures, the scorecard highlights governance-relevant gaps that unweighted scoring obscures.
<details>
<summary>compliance.jpg Details</summary>
### Visual Description
## Horizontal Bar Chart: Compliance by Provider (Weighted)
### Overview
The image is a horizontal bar chart displaying the average weighted compliance of various providers. The chart ranks providers from highest to lowest compliance, with the x-axis representing the average weighted compliance percentage and the y-axis listing the providers. The bars are colored in a gradient from yellow to green, with green indicating higher compliance.
### Components/Axes
* **Title:** Compliance by Provider (Weighted)
* **X-axis:** Average Weighted Compliance (%), with scale from 0 to 100, incrementing by 20.
* **Y-axis:** List of providers (xAI, Microsoft, Anthropic, Meta AI, DeepSeek-AI, OpenAI, Google, Stability AI, MoonshotAI, Alibaba, HuggingFaceH4, BRIA AI, Mistral AI, Qwen, Illyasviel, Tencent, 2Noise, Lightricks, TII UAE, Stepfun AI, ByteDance, Docling Project, Black Forest Labs).
* **Bars:** Horizontal bars representing the compliance percentage for each provider. The bars are colored in a gradient from yellow to green, with green indicating higher compliance.
### Detailed Analysis
The chart presents the following data points for each provider:
* **xAI:** 80.5% (Green)
* **Microsoft:** 79.5% (Green)
* **Anthropic:** 79.5% (Green)
* **Meta AI:** 79.0% (Green)
* **DeepSeek-AI:** 78.0% (Green)
* **OpenAI:** 76.6% (Green)
* **Google:** 76.2% (Green)
* **Stability AI:** 64.0% (Yellow-Green)
* **MoonshotAI:** 64.0% (Yellow-Green)
* **Alibaba:** 61.0% (Yellow-Green)
* **HuggingFaceH4:** 58.5% (Yellow-Green)
* **BRIA AI:** 58.0% (Yellow-Green)
* **Mistral AI:** 56.8% (Yellow-Green)
* **Qwen:** 56.0% (Yellow-Green)
* **Illyasviel:** 54.0% (Yellow-Green)
* **Tencent:** 52.5% (Yellow-Green)
* **2Noise:** 49.5% (Yellow)
* **Lightricks:** 49.0% (Yellow)
* **TII UAE:** 48.5% (Yellow)
* **Stepfun AI:** 48.0% (Yellow)
* **ByteDance:** 47.5% (Yellow)
* **Docling Project:** 45.0% (Yellow)
* **Black Forest Labs:** 40.3% (Yellow)
### Key Observations
* xAI has the highest average weighted compliance at 80.5%.
* Black Forest Labs has the lowest average weighted compliance at 40.3%.
* There is a clear distinction between the top 7 providers (xAI to Google) and the rest, with a significant drop in compliance percentage after Google.
* The compliance percentages range from 40.3% to 80.5%, indicating a wide variation in compliance among the listed providers.
### Interpretation
The chart illustrates the relative compliance of different AI providers based on a weighted average. The data suggests that some providers are significantly more compliant than others, with xAI leading the pack. The color gradient visually reinforces this, highlighting the higher compliance of the top providers. The wide range in compliance percentages could be attributed to various factors, such as differences in internal policies, implementation of compliance measures, or the nature of the AI services offered. The chart provides a snapshot of the compliance landscape among these providers, which could be useful for benchmarking, regulatory oversight, or informing consumer choices.
</details>
Figure 5: Weighted compliance scores by provider. Major frontier labs (xAI, Microsoft, Anthropic, Meta) achieve 80% compliance, while smaller providers range from 40-65%.
| Model Details (15%) | Model overview | 3 |
| --- | --- | --- |
| Organization developing the model | 1 | |
| Model Version | 2 | |
| Model Release Date | 0.5 | |
| Model Version Progression | 1 | |
| Model Architecture | 4 | |
| Model Dependencies | 1 | |
| Paper and relevant links | 0.5 | |
| Model Distribution Forms | 2 | |
| Model Inputs & Outputs (6%) | Inputs | 2 |
| Outputs | 2 | |
| Token Count | 2 | |
| Model Data (15%) | Training Dataset | 7 |
| Training Data Processing | 6 | |
| Knowledge Count | 2 | |
| Model Implementation and Sustainability (5%) | Hardware Used During Training & Inference | 2 |
| Software Frameworks & Tooling | 2 | |
| Energy Use / Sustainability Metrics | 1 | |
| Intended Use (10%) | Primary intended uses | 5 |
| Primary intended users | 2 | |
| Out-of-scope use cases | 3 | |
| Critical Risk (20%) | CBRN (Chemical, Biological, Radiological or Nuclear) | 5 |
| Cyber Risk | 5 | |
| Harmful Manipulation | 4 | |
| Child Safety Evaluations | 4 | |
| Privacy Risks | 2 | |
| Safety Evaluation (25%) | Refusals | 1 |
| Disallowed Content Handling | 4 | |
| Sycophancy | 2 | |
| Jailbreak | 4 | |
| Hallucinations | 4 | |
| Deception Behaviors | 4 | |
| Fairness & Bias Evaluations (incl. BBQ) | 3 | |
| Adversarial Robustness | 2 | |
| Red Teaming Results | 1 | |
| Risk Mitigations (4%) | Risk Mitigation | 4 |
TABLE I: Proposed Documentation Transparency Framework: Sections, Subsections, and Scores
IV-C Framework Objectives and Weighting Approach
Our framework targets the two largest deficiencies in current AI documentation ecosystems: consistency and evaluability. Rather than adding yet another lengthy template, it distills the essential fields that matter for oversight and aligns them with existing regulatory expectations, making adoption practical for developers and meaningful for policymakers.
Our weighted scoring method also addresses the problem of treating all documentation fields as equally important. By weighting safety-critical disclosures more heavily, the scorecard highlights the gaps that matter most for governance. In our evaluation, this method surfaced high-risk blind spots that were invisible under unweighted scoring, producing a clearer and more policy-relevant picture of model transparency.
We recommend requiring AI developers to publish model and system documentation not only as PDFs or web pages but also as machine-readable JSON following an open, extensible schema. This schema would encode core fields such as model name and version, training data summary, evaluation benchmarks, known limitations, and implemented safety mitigations. Machine-readable documentation enables researchers, regulators, and civil-society organizations to programmatically ingest, compare, and audit models at scale.
In developing our methodology, we also examined several plausible approaches for improving AI documentation consistency and transparency. Each alternative offers conceptual advantages but also reveals structural limitations that constrain its practical effectiveness.
- Developing a comprehensive new documentation framework. One approach is to design a fully new documentation standard that supersedes existing model cards, data sheets, and system cards. While appealing in terms of conceptual clarity, this strategy adds another framework to an already crowded landscape. Creating a standalone standard risks poor adoption, limited interoperability with existing tools, and slow industry uptake. It also overlooks areas where partial convergence already exists, such as shared fields across model cards and system cards.
- Strengthening voluntary narrative guidelines. Another approach is to rely on high-level best-practice recommendations encouraging developers to “write better model cards.” Although lightweight and easy to disseminate, purely narrative guidance does not correct the structural issues observed in our empirical analysis: inconsistent field definitions, lack of machine-readable formats, and wide variability in documentation depth. Moreover, voluntary guidelines provide weak incentives and no mechanism for automated evaluation, limiting their ability to drive meaningful improvement.
- Establishing an immediate, globally harmonized regulatory standard. A third approach is to push for a fully harmonized international standard for AI documentation. While this could theoretically provide a unified foundation, it is politically and technically difficult to implement in the near term. Regulatory ecosystems differ across jurisdictions, and AI systems evolve faster than global regulatory consensus can form. Such an approach risks regulatory stagnation or adoption of a lowest-common-denominator standard. As a result, it provides limited near-term utility for practitioners or policymakers.
<details>
<summary>system-design.jpg Details</summary>
### Visual Description
## Flowchart: Model Card Framework
### Overview
The image is a flowchart illustrating a process for evaluating and populating a model card framework. It involves data from various models, query generation, internet searches, and a consensus mechanism.
### Components/Axes
* **Nodes:** The flowchart consists of several nodes, each represented by a colored rectangle or cloud shape, indicating different stages or components of the process.
* **Arrows:** Arrows indicate the flow of information and the sequence of steps.
* **Colors:** Different colors are used to distinguish between different types of nodes.
* Yellow: Start
* Green: Source of Truth, LLM1
* Orange: Query Generation
* Blue: Data set of models, Populate the Framework, Perplexity Search API
* Purple: Consensus
* Gray: LLM3
* Yellow: LLM2
* **Text Labels:** Each node contains text describing its function or content.
### Detailed Analysis
1. **Start:** The process begins with a yellow node labeled "Start" located at the top-left.
2. **Source of Truth:** From "Start", a curved arrow leads to a green rectangle labeled "Source of Truth: Our New Model Card Framework".
3. **Data Set of Models:** A blue cloud shape at the top-center is labeled "Data set of models (Claude, GPT...etc)". Below this cloud, the text "Select a model e.g. Claude Sonnet 4" is present.
4. **Query Generation (1):** From "Source of Truth", an arrow labeled "Select a section e.g. Safety" leads to an orange rectangle labeled "Query Generation".
5. **Perplexity Search API:** From the first "Query Generation" node, an arrow labeled "Search the internet for safety and its subsections for Claude Sonnet 4" leads to a blue rectangle labeled "Perplexity Search API". The Perplexity Search API node also has a logo of a purple geometric shape. Below this node, the text "Results in Chunks with sources" is present.
6. **Query Generation (2):** An arrow from "Perplexity Search API" leads to another orange rectangle labeled "Query Generation".
7. **LLM Nodes:** Three circles, arranged vertically, are labeled "LLM1" (green), "LLM2" (yellow), and "LLM3" (gray). These are enclosed in a light gray box.
8. **Scoring:** Arrows from both "Query Generation" nodes point to the LLM nodes. The arrow pointing to the LLM nodes is labeled "Score based on our new model card".
9. **Consensus:** Arrows from the LLM nodes lead to a purple rectangle labeled "Consensus: Majority vote".
10. **Populate the Framework:** An arrow from "Consensus" leads to a blue rectangle labeled "Populate the Framework".
11. **Loop:** A curved arrow from "Populate the Framework" leads back to the "Start" node, indicating a loop in the process.
### Key Observations
* The flowchart describes an iterative process for populating a model card framework.
* The process involves selecting a model, generating queries, searching the internet, scoring using LLMs, and reaching a consensus.
* The loop from "Populate the Framework" back to "Start" suggests continuous refinement and updating of the model card.
### Interpretation
The flowchart outlines a systematic approach to evaluating and documenting AI models using a model card framework. The process leverages external data sources (Perplexity Search API) and multiple LLMs to generate scores and reach a consensus on model characteristics, specifically safety. The iterative nature of the process allows for continuous improvement and refinement of the model card, ensuring it remains up-to-date and accurate. The use of a consensus mechanism suggests an effort to reduce bias and ensure a more objective evaluation of the models.
</details>
Figure 6: Overview of our automated data extraction and scoring pipeline. Starting from our model card framework (source of truth), the system selects a model, generates targeted queries for each section, retrieves evidence from the Perplexity Search API, evaluates the results using multiple LLMs through majority-vote consensus, and populates the standardized model card schema.
V Automated Documentation Extraction Pipeline
Manual evaluation of model documentation against our framework would be time-consuming and not scalable across hundreds of models. To address this limitation, we developed an automated agentic pipeline that extracts, evaluates, and scores documentation using a multi-agent LLM system with web search capabilities [AI_transparency_atlas_2025]. This approach builds on recent work by Liu et al. [liu2024automaticgenerationmodeldata], who demonstrated the feasibility of using LLMs to automatically generate model and data cards. While their work focuses on creating documentation artifacts, our pipeline evaluates and scores existing documentation by extracting information from dispersed public sources.
V-A Pipeline Architecture
Figure 6 illustrates the overall pipeline architecture. The process begins by selecting a model from our dataset (e.g., Claude Sonnet 4, GPT-5, Llama 4). For each selected model, the pipeline iterates through the eight main sections and 23 subsections defined in our framework (see Table I).
V-B Query Generation and Information Retrieval
For each subsection (e.g., Safety Evaluation $→$ Jailbreak), the pipeline generates targeted search queries designed to retrieve relevant documentation from publicly available sources. We use the Perplexity Search API to collect evidence from model cards, system cards, technical reports, blog posts, and GitHub repositories.
This design reflects real-world auditing conditions, as it evaluates only information accessible to external auditors, policymakers, and users without direct cooperation from model developers. Search results are returned in structured chunks with source citations, preserving traceability and transparency.
V-C Multi-Agent Consensus Scoring
Retrieved documentation is independently evaluated by three LLM agents (LLM 1, LLM 2, LLM 3). Each agent assesses documentation completeness for a given subsection according to our framework criteria and assigns one of the following labels:
- Detailed: Substantive, specific, and actionable information
- Mentioned: Present but superficial or vague information
- Absent: No relevant information found
The final score for each subsection is determined using majority-vote consensus across the three agents.
V-D Framework Population
After all subsections are evaluated, the pipeline populates the framework with the extracted evidence and consensus scores. Subsection scores are then aggregated using the weighting scheme defined in Table I to produce an overall transparency rating for each model.
V-E Scalability and Cost
The pipeline is fully automated and cost-efficient. Evaluating all 50 models across 23 subsections cost less than $3 total (under $0.06 per model), enabling large-scale analysis across hundreds of models. We validated the pipeline on models spanning vision, multimodal, open-source, and closed-source systems, demonstrating robustness across diverse documentation formats and styles.
VI Discussion, Limitations, and Future Work
Our framework enables model providers to assess their documentation transparency in real-time. Providers can access a live dashboard showing their overall transparency score and subsection-level breakdowns, immediately identifying gaps in their documentation. For example, a provider scoring high on technical specifications (Model Architecture, Compute Requirements) but low on safety-critical categories (Jailbreak Evaluations, Child Safety) can prioritize improvements in those areas.
Our pipeline offers a pragmatic middle ground: instead of requiring providers to restructure their documentation, we automatically aggregate information from dispersed sources and present it through a standardized evaluation framework. This enables consistent cross-model comparison even when underlying documentation practices vary, lowering adoption barriers while improving transparency assessment for regulators, researchers, and users.
A potential limitation of any weighted scoring system is that providers may optimize for score maximization rather than genuine transparency. Since our framework assigns higher weights to safety-critical categories (Safety Evaluation: 25%, Critical Risk: 20%), providers could strategically prioritize these sections to boost scores while neglecting lower-weighted but still important disclosures.
However, several design features mitigate this risk. First, our framework requires substantive, detailed information—not just the presence of a section. Surface-level statements that mention safety evaluations without specific methodologies, results, or limitations receive lower scores than comprehensive disclosures. Second, the multi-agent consensus mechanism evaluates documentation quality, not just existence, making it harder to game through minimal compliance. Third, transparency scores are most valuable when used comparatively and longitudinally; providers who inflate scores through selective disclosure will be evident when their documentation is examined alongside peers or tracked over time.
Beyond point-in-time evaluation, our structured representation of publicly available documentation enables systematic tracking of documentation changes across model versions. By normalizing dispersed online disclosures into a consistent schema, the framework creates a stable reference that can be compared longitudinally as models evolve. While the current study focuses on static snapshots, we plan to extend this pipeline to support version-aware analysis, allowing transparency scores and subsection-level disclosures to be tracked across releases. This would enable stakeholders to monitor how safety claims, evaluation practices, and risk disclosures change over time, and to identify regressions or improvements in transparency as models are updated.
VII Conclusion
AI model documentation today is fragmented, inconsistent, and insufficient for meaningful oversight. Information is scattered across platforms, section naming varies wildly across providers, and safety-critical disclosures are frequently absent or superficial. These gaps undermine the ability of regulators, researchers, and downstream users to assess model risks, compare alternatives, and make informed decisions. This paper presented a structured response to these challenges. Through empirical analysis of frontier model documentation and 100 Hugging Face model cards, we identified systematic inconsistencies in how models are documented. We developed a transparency framework with 8 sections and 23 subsections, grounded in the EU AI Act Annex IV and the Stanford Transparency Index, with weighted scoring that prioritizes safety-critical information over technical specifications.To operationalize this framework at scale, we built an automated agentic pipeline that extracts documentation from public sources, evaluates completeness using multi-agent consensus, and generates transparency scores. The pipeline cost less than $3 to evaluate 50 diverse models, demonstrating economic feasibility for continuous, large-scale monitoring. Our evaluation reveals significant transparency gaps across the ecosystem. While frontier labs like xAI, Microsoft, and Anthropic achieve 80% compliance, many smaller providers fall below 50%. Categories like Interpretability and Safety Evaluation—critical for governance—remain poorly documented across most models. The framework offers practical value for multiple stakeholders. Providers can use live dashboards to identify documentation gaps and track improvements over time. Regulators gain evidence-based tools for compliance assessment without manual audits. Researchers obtain standardized metrics for cross-model comparison. Users gain visibility into which models have comprehensive safety documentation. Moving forward, transparency in AI cannot rely solely on voluntary adoption of documentation standards. Our approach demonstrates that automated extraction and standardized evaluation can bridge the gap between current fragmented practices and the structured information needed for accountability. As AI systems become more capable and widely deployed, robust documentation transparency is not optional—it is foundational to responsible governance.