## Flowchart: Model Card Framework and Query Generation Process ### Overview This flowchart illustrates a technical workflow for generating and validating queries using a model card framework, integrating external data sources, and leveraging multiple language models (LLMs) for consensus-based decision-making. The process is cyclical, with feedback loops between components. ### Components/Axes 1. **Start** (Yellow Box): Initiates the workflow. 2. **Source of Truth** (Green Box): Contains the "Our New Model Card Framework." 3. **Query Generation** (Orange Box): Generates queries based on selected model sections (e.g., safety). 4. **Perplexity Search API** (Purple Box): Searches the internet for safety data and subsections of models like Claude Sonnet 4, returning results in chunks with sources. 5. **Populate the Framework** (Cyan Box): Updates the model card framework using consensus outputs. 6. **Consensus: Majority Vote** (Pink Box): Aggregates outputs from three LLMs (LLM1, LLM2, LLM3) to determine the final result. 7. **LLMs** (Colored Circles): - LLM1 (Green) - LLM2 (Yellow) - LLM3 (Gray) - Outputs are scored based on the new model card. ### Detailed Analysis - **Flow Direction**: - Starts at "Start" → "Source of Truth" → "Our New Model Card Framework." - Branches to "Query Generation," which connects to the "Perplexity Search API." - API results feed back into "Query Generation," which then routes outputs to all three LLMs. - LLM outputs are aggregated via "Consensus: Majority Vote" to "Populate the Framework," which loops back to the "Source of Truth." - **Key Connections**: - Arrows labeled "Select a section e.g. Safety" link the model card to query generation. - Arrows labeled "Score based on our new model card" connect LLMs to query generation. - Feedback loop from "Populate the Framework" to "Source of Truth" ensures iterative refinement. ### Key Observations - **Cyclical Process**: The workflow is designed for continuous improvement, with outputs from the consensus step directly updating the foundational model card. - **External Data Integration**: The Perplexity Search API introduces real-time, external data (e.g., safety information) to inform query generation. - **Multi-LLM Consensus**: Three distinct LLMs contribute to decision-making, with majority voting ensuring robustness. ### Interpretation This diagram represents a hybrid AI system that combines: 1. **Internal Knowledge** (model card framework) with **External Data** (Perplexity API) to generate context-aware queries. 2. **Model Diversity** (three LLMs) to mitigate individual model biases, using consensus to enhance reliability. 3. **Iterative Refinement**, where outputs from the consensus step directly improve the foundational model card, creating a self-improving loop. The use of color-coding (e.g., green for safety, purple for external search) suggests a modular design where each component has a distinct role. The absence of numerical values implies this is a conceptual workflow rather than a data-driven analysis. The process emphasizes transparency (via source attribution in API results) and adaptability (through iterative framework updates).