2403.09724

# ClaimVer: Explainable Claim-Level Verification and Evidence Attribution of Text Through Knowledge Graphs > Work does not relate to position at Amazon. ## Abstract In the midst of widespread misinformation and disinformation through social media and the proliferation of AI-generated texts, it has become increasingly difficult for people to validate and trust information they encounter. Many fact-checking approaches and tools have been developed, but they often lack appropriate explainability or granularity to be useful in various contexts. A text validation method that is easy to use, accessible, and can perform fine-grained evidence attribution has become crucial. More importantly, building user trust in such a method requires presenting the rationale behind each prediction, as research shows this significantly influences people’s belief in automated systems. Localizing and bringing users’ attention to the specific problematic content is also paramount, instead of providing simple blanket labels. In this paper, we present ClaimVer, a human-centric framework tailored to meet users’ informational and verification needs by generating rich annotations and thereby reducing cognitive load. Designed to deliver comprehensive evaluations of texts, it highlights each claim, verifies it against a trusted knowledge graph (KG), presents the evidence, and provides succinct, clear explanations for each claim prediction. Finally, our framework introduces an attribution score, enhancing applicability across a wide range of downstream tasks. ClaimVer: Explainable Claim-Level Verification and Evidence Attribution of Text Through Knowledge Graphs Preetam Prabhu Srikar Dammu 1, Himanshu Naidu 1, Mouly Dewan 1, YoungMin Kim 1, Tanya Roosta 2,4, thanks: Work does not relate to position at Amazon., Aman Chadha 3,4, footnotemark: , Chirag Shah 1 1 University of Washington 2 UC Berkeley 3 Stanford University 4 Amazon GenAI <details> <summary>x1.png Details</summary>  ### Visual Description ## Diagram: Fact-Checking Claim Analysis Comparison ### Overview The image is a two-part diagram (labeled A and B) illustrating and comparing the analysis of two different factual claims by two different systems: a traditional fact-checking source (HealthFeedback.org in part A, Google Fact Check Tools in part B) and a system called "ClaimVer." The diagram demonstrates how ClaimVer deconstructs claims into components and evaluates them against structured knowledge (represented by Q-codes and triplets). ### Components/Axes The diagram is divided into two horizontal sections, A and B, separated by a dashed line. Each section follows a similar flow: 1. **Claim Source:** A silhouette of a person with a speech bubble containing the original claim. 2. **Traditional Fact-Check:** A logo and a box containing the fact-checking organization's verdict. 3. **ClaimVer Analysis:** A detailed breakdown featuring: * **Knowledge Definitions:** Boxes with Q-codes (e.g., Q134808) defining key terms. * **Claim Deconstruction:** The original claim text, segmented and color-coded (yellow for premises, red for the main assertion). * **Structured Responses:** Dotted-line boxes labeled R1, R2, R3 containing ClaimVer's analysis of each segment. * **Knowledge Triplets:** In part B, additional Q-code boxes and a list of triplets in brackets (e.g., `[('Apollo 11', 'crew member(s)', 'Neil Armstrong')]`). ### Detailed Analysis #### **Section A: Vaccine-Autism Claim** * **Original Claim (Top Center):** "Autism used to be 1 in 10,000. Now it's 1 in 50. Now, where it all coming from? Vaccines are doing it." * **Traditional Fact-Check (Left):** * **Source:** HealthFeedback.org (logo with DNA helix). * **Verdict:** "Inaccurate: The link between vaccines and autism has already been disproved in several studies." * **ClaimVer Analysis (Bottom):** * **Knowledge Definitions (Top):** * `Q134808 (Vaccine):` biological preparatory medicine that improves immunity to a particular disease. * `Q38404 (Autism):` neurodevelopmental condition. * **Claim Deconstruction & Responses:** * Segment 1 (Yellow): `Autism used to be 1 in 10,000. Now it's 1 in 50.` -> **R1:** Prevalence of autism is not directly supported or refuted. * Segment 2 (Yellow): `Now, where it all coming from?` -> **R2:** Origin of the increase in autism prevalence is not addressed. * Segment 3 (Red, underlined): `Vaccines are doing it.` -> **R3:** Statement that vaccines are causing the increase in autism prevalence is directly contradicted by the triplet `[('autism', 'does not have cause', 'vaccine')]`. #### **Section B: Moon Landing Hoax Claim** * **Original Claim (Top Center):** "Image shows mismatch between Neil Armstrong’s spacesuit and boot print left on the Moon, therefore Moon landing was a hoax." * **Traditional Fact-Check (Left):** * **Source:** Google Fact Check Tools (logo). * **Verdict:** "AFP Fact Check rating: False". * **ClaimVer Analysis (Bottom):** * **Knowledge Definitions (Top):** * `Q1615 (Neil Armstrong):` American astronaut; first person to walk on the moon. * `Q495307 (Moon landing):` arrival of a spacecraft on the surface of the Moon. * **Claim Deconstruction & Responses:** * Segment 1 (Yellow): `Image shows mismatch between Neil Armstrong’s spacesuit and boot print left on the Moon,` -> **R1:** specific claim about the mismatch between the spacesuit and boot print is not directly supported or refuted. * Segment 2 (Red, underlined): `therefore Moon landing was a hoax.` -> **R2:** The triplets directly state that the Moon landing was a significant event and an instance of the Apollo 11 mission, which contradicts the claim that the Moon landing was a hoax. `[('Apollo 11', 'crew member(s)', 'Neil Armstrong'), ('Apollo 11','significant event', 'Moon landing'), ('Apollo 11', 'instance of', 'Moon landing')]`. * **Additional Knowledge Base (Bottom Left):** A cluster of Q-code boxes: `Q223571`, `Q190868`, `Q18218093`, `Q190084`, `Q190084`, `Q405`, `....`. ### Key Observations 1. **Structural Parallelism:** Both sections A and B follow an identical visual and analytical structure, facilitating direct comparison. 2. **Color Coding:** ClaimVer consistently uses yellow text for factual premises or observations and red, underlined text for the core causal or conclusive assertion being evaluated. 3. **Response Hierarchy:** The R1, R2, R3 responses correspond directly to the segmented parts of the claim, moving from peripheral facts to the central assertion. 4. **Knowledge Representation:** ClaimVer relies on a structured knowledge base (Q-codes for entities/concepts and triplets for relationships) to generate its responses, contrasting with the narrative verdicts of the traditional fact-checkers. 5. **Handling of Uncertainty:** In both examples, ClaimVer's R1 response explicitly states when a specific sub-claim (e.g., the prevalence statistic, the image mismatch) is "not directly supported or refuted" by its knowledge base, demonstrating a granular approach to verification. ### Interpretation This diagram serves as a conceptual model or advertisement for the "ClaimVer" system. It argues that traditional fact-checking provides a binary, holistic verdict (Inaccurate/False), while ClaimVer offers a more nuanced, transparent, and auditable analysis. The core demonstration is that ClaimVer: * **Decomposes** complex claims into atomic statements. * **Contextualizes** key terms using a formal knowledge base (Wikidata-style Q-codes). * **Evaluates** each component against structured knowledge triplets. * **Identifies** the precise point of failure in a claim (e.g., the causal link in part A, the conclusion in part B) while acknowledging areas of uncertainty. The inclusion of the "traditional" fact-check verdicts (from HealthFeedback.org and AFP via Google) likely serves as a benchmark, showing that ClaimVer arrives at the same ultimate conclusion (the claim is false) but through a method that exposes its reasoning chain. The diagram suggests ClaimVer's value lies in its systematic, evidence-linked deconstruction, which could be more resistant to context manipulation and more informative for understanding *why* a claim is wrong. The final cluster of Q-codes in part B hints at the extensive underlying knowledge graph that powers such analysis. </details> Figure 1: Demonstration of ClaimVer for claim verification and evidence attribution. (A) Text labeled as Inaccurate by HealthFeedback and ClaimVer’s predictions, rationale, and evidence. (B) Text labeled as False by Google Fact Check Tools and ClaimVer’s outputs. Predictions are color-coded (amber: extrapolatory, red: contradictory); $R_i$ : rationale; related wiki entities are displayed in boxes. ## 1 Introduction Misinformation and disinformation are longstanding issues, but the proliferation of AI tools that can generate information on demand has amplified these issues. Tools for fact-checking are not keeping pace with sophisticated text generation techniques. Even when they are effective, they lack appropriate explainability and granularity to be useful to users. Studies have shown that explanations are crucial for users to build trust in AI systems Rechkemmer and Yin (2022); Weitz et al. (2019); Shin (2021). Therefore, there is a need for a novel human-centric approach to text verification that offers usable and sufficiently granular explanations to inform and educate the user. Most fact-checkers, including widely used ones in deployment, issue blanket predictions that can lead to user misunderstandings. For instance, in Figure 1 (A), we observe that HealthFeedback https://healthfeedback.org/, a fact-checker for medical text, indicates that a misleading statement about the increase in Autism is inaccurate. However, there are multiple claims made in that text, which are not addressed by this tool. In fact, research does show that Autism cases have increased, but this is mostly attributed to increased testing Russell et al. (2015). Our method accurately breaks down the text into multiple claims and shows that the specific claim that vaccines are causing autism is indeed incorrect, attributing it to a fact from the Wikidata Vrandečić and Krötzsch (2014). It also provides a clear rationale as to why the first two claims cannot be determined, as there’s no conclusive evidence present in the KG. Such granular predictions, supported by justifications, significantly improve user confidence Rechkemmer and Yin (2022); Weitz et al. (2019); Shin (2021). Similarly, in Figure 1 (B), we notice that Google Fact Check Tools https://toolbox.google.com/factcheck/explorer provides a blanket label for an utterance denying the moon landing. In contrast, ClaimVer identifies the exact text span that can be conclusively proven incorrect and proceeds to provide specific information about the Apollo 11 mission and its crew members to refute the claim. All verified entities present in the text, along with their Wiki IDs and descriptions, are displayed for user reference. Prior research Rashkin et al. (2023); Yue et al. (2023); Thorne et al. (2019); Aly et al. (2021) typically validates text at the paragraph or sentence level without adequately enhancing user awareness by supplying key details such as rationale, match scores, or evidence. A KG-based approach allows for finer granularity, aiding in pinpointing specific inaccuracies like hallucinations in LLM-generated text or false claims in misleading text. Furthermore, if needed, broader-level metrics can be extracted from this detailed attribution. The assumption of one-to-one mapping between input and reference texts, prevalent in previous methods Rashkin et al. (2023); Yue et al. (2023); Thorne et al. (2019); Aly et al. (2021), does not hold if the given text consists of claims that can be mapped to more than one source. In contrast, utilizing a KG, which represents a consolidated body of knowledge, results in a more comprehensive evaluation. While most previous methods may not support scenarios with information spread across various references, querying a KG can yield triplets originally sourced from multiple documents. Additionally, procuring the specific spans of text required to evaluate claims from large text sources that may span several pages presents many challenges. In contrast, a KG captures only the most important relationships as nodes and links, providing a more efficient way to evaluate the claims. ## 2 Related Work Research on validating text has been ongoing for the past decade, while the concept of evidence attribution has gained increased attention in recent years, following the advent of generative models. Our method integrates fact verification and evidence attribution. In this section, we discuss recent advancements in both domains. ### 2.1 Fact Verification Fact verification is a task that is closely related to natural language inference (NLI) Conneau et al. (2017); Schick and Schütze (2020), in which given a premise, the task is to verify whether a hypothesis is an entailment, contradiction, or neutral. Similarly, in fact verification, the task is to check if a given text can be supported, refuted, or indeterminable, given a reference text. Recent studies in this domain show that LLMs can achieve high performance, and can be considerably reliable for verification tasks, even though they are prone to hallucations Guan et al. (2023). In Lee et al. (2020), the authors show that the inherent knowledge of LLMs could be used to perform fact verification. Other works Yao et al. (2022); Jiang et al. (2023b) have shown that using external knowledge is helpful for many reasoning-intensive tasks, and report enhanced performance on HotPotQA Yang et al. (2018) and FEVER Thorne et al. (2018). A wide variety of studies have established that LLMs are suitable for fact verification. For example, Dong and Smith (2021) enhanced accuracy of table-based fact verification by incorporating column-level cell rank information into pre-training. In FactScore, authors Min et al. (2023), introduce a new evaluation that breaks a long-form text generated by large language models (LMs) into individual atomic facts and calculates the proportion of these atomic facts that are substantiated by a credible knowledge base. ### 2.2 Evidence Attribution The distinction between evidence attribution and fact verification lies in the emphasis on identifying a source that can be attributed to the information. This task is becoming increasingly important, as generative models produce useful and impressive outputs, but without a frame of reference to validate them. In Rashkin et al. (2023), the authors present a framework named AIS (Attributable to Identified Sources) that specifies annotation guidelines and underlines the importance of attributing text to an external, verifiable, and independent source. Yue et al. (2023) demonstrate that LLMs can be utilized for automatic evaluation of attribution, operationalizing the guidelines presented in Rashkin et al. (2023). However, both of these works are primarily designed for the question-answering (QA) task. In contrast, our method is not restricted to QA and is designed to work with text in general. Furthermore, while these previous studies focus on sentence or paragraph levels, our approach extends to a more detailed and granular level of analysis. <details> <summary>x2.png Details</summary>  ### Visual Description ## Diagram: Claim Verification System Flowchart ### Overview The image is a technical flowchart illustrating an automated claim verification system. It demonstrates the end-to-end process of taking a textual claim as input, processing it through a series of steps involving a knowledge graph and a specialized language model, and producing a structured output with a verification verdict, supporting evidence, and a confidence score. The specific example used is the claim: "Steven Tyler has never been a part of the band Aerosmith." ### Components/Axes The diagram is organized as a left-to-right process flow with the following major components: 1. **Input Text (Far Left):** A rectangular box containing the claim to be verified. 2. **Preprocessing (Center-Left):** A rounded rectangle listing four processing steps. 3. **Knowledge Graph (Top-Center):** A visual representation of a graph with interconnected nodes (circles of varying sizes and colors: dark blue, light blue, teal). 4. **KG Triplet Retrieval Algorithm (Center):** A circular icon containing a gear and flowchart symbols, representing the core retrieval mechanism. 5. **Finetuned ClaimVer LLM (Center-Right):** A circular icon containing a stylized unicorn logo, representing a specialized language model. 6. **Outputs (Far Right):** A large, dotted-border rectangle containing five stacked data fields that present the verification results. **Flow Direction:** Arrows indicate the data flow: `Input Text` → `Preprocessing` → `KG Triplet Retrieval Algorithm`. The `Knowledge Graph` has bidirectional arrows with the retrieval algorithm, indicating querying and data retrieval. The output from the retrieval algorithm and the original input text (via a bypass line) are combined (indicated by a `+` symbol) and fed into the `Finetuned ClaimVer LLM`, which then produces the final `Outputs`. ### Detailed Analysis **1. Input Text:** * **Content:** "Steven Tyler has never been a part of the band Aerosmith." **2. Preprocessing:** * **Listed Steps:** * NER (Named Entity Recognition) * Coreference * KG Entity Linking * Compartmentalization **3. Knowledge Graph:** * **Visual:** A network diagram with approximately 10-12 nodes connected by lines. Nodes are circles of different sizes and colors (dark blue, light blue, teal), suggesting different entity types or importance. No specific labels are visible on the nodes or edges in this schematic representation. **4. KG Triplet Retrieval Algorithm:** * **Function:** This component interacts with the Knowledge Graph to find relevant factual triplets (subject-predicate-object relationships) related to the entities in the input text. **5. Finetuned ClaimVer LLM:** * **Function:** A language model fine-tuned for claim verification. It receives the preprocessed information and the retrieved knowledge graph triplets to generate a verdict and rationale. **6. Outputs:** * **Structure:** Five labeled fields within a dotted border. * **Claim:** "Steven Tyler has never been a part of the band Aerosmith." (Identical to the input). * **Prediction:** "Contradictory" * **Relevant Triplets & TMS:** `([('Aerosmith', 'has part(s)', 'Steven Tyler')], 1.0)` * This indicates one triplet was retrieved: Subject=`Aerosmith`, Predicate=`has part(s)`, Object=`Steven Tyler`. * The `1.0` likely represents a Truth Mean Score (TMS) or confidence for this triplet's validity. * **Rationale:** "This triplet establishes a clear relationship between Steven Tyler and Aerosmith, refuting the claim that he has never been associated with the band." * **Score (KAS):** `0.047` * KAS likely stands for Knowledge-Aware Score or a similar confidence metric. A low score (0.047) in this context appears to correlate with high confidence in the "Contradictory" prediction. ### Key Observations * **System Logic:** The system correctly identifies the claim as false by retrieving a direct, contradictory fact from its knowledge base. * **Evidence Specificity:** The retrieved triplet (`Aerosmith, has part(s), Steven Tyler`) is a precise and semantically strong refutation of the claim. * **Score Interpretation:** The very low KAS score (0.047) suggests the model is highly confident in its "Contradictory" verdict. The relationship between the TMS (1.0 for the triplet) and the final KAS (0.047) is not explicitly defined but indicates a composite scoring mechanism. * **Process Transparency:** The system provides a clear rationale linking the retrieved evidence to the final decision, enhancing interpretability. ### Interpretation This diagram outlines a sophisticated, knowledge-grounded approach to automated fact-checking. Unlike simple text-matching systems, it: 1. **Leverages Structured Knowledge:** It uses a Knowledge Graph to access factual relationships, moving beyond statistical patterns in text. 2. **Employs Specialized AI:** The "Finetuned ClaimVer LLM" indicates the use of a language model adapted specifically for the task of verification, likely trained to weigh evidence from knowledge graphs. 3. **Provides Explainable Results:** The output includes not just a verdict ("Contradictory") but also the specific evidence (the triplet) and a human-readable rationale. This is crucial for trust and debugging. 4. **Quantifies Uncertainty:** The inclusion of scores (TMS and KAS) allows for assessing the system's confidence in both the retrieved facts and the final judgment. The example demonstrates a successful verification where a false claim is decisively refuted by a single, high-confidence fact from the knowledge base. The system's design suggests it would be effective for verifying claims about entities (people, organizations, etc.) and their relationships, provided that information is encoded in its underlying knowledge graph. The "Compartmentalization" step in preprocessing hints at handling complex or multi-faceted claims by breaking them down. </details> Figure 2: Flow of operations in the ClaimVer framework. Identified KG entity nodes during preprocessing inform the extraction of relevant triplets by the KG algorithm. Subsequently, these triplets and preprocessed text are then fed to a ClaimVer LLM, fine-tuned to operationalize the objective function. For each claim, the corresponding text span, prediction, relevant triplets, attribution scores, and rationale are generated. ## 3 Methodology In this section, we present the methodology for retrieving relevant triplets from the KG, fine-tuning LLM to process text at claim-level, verifying claims, tagging evidence for each prediction, and generating a rationale along with an attribution score that reflects the text’s validity. ### 3.1 Preprocessing Preprocessing involves multiple steps required to make the input text suitable for the subsequent operations. Since the nodes in a KG typically represent entities, performing Named Entity Recognition (NER) is necessary. In our work, we chose Wikidata Vrandečić and Krötzsch (2014) as the KG source; thus, we use an NER module suitable for Wiki entities Gerber (2023). However, the framework is sufficiently generic to support any kind of KG that models information in the form of triplets. As our analysis is performed at the claim level, coreference resolution Lee et al. (2017) becomes a necessary step to form localized claims that are semantically self-contained. If input text exceeds the context length, which depends on design choices, compartmentalization would be required. As a final step in preprocessing, we perform KG entity linking. This step tags all entities in the text that are present in the KG as nodes. ### 3.2 Relevant Triplets Retrieval Retrieving relevant triplets is a complex problem that has attracted attention from various research communities, and resulted in multiple approaches to address the challenge. While retrieving direct links between two given nodes in a KG is relatively straightforward, identifying complex paths that involve multiple hops is challenging. In our framework, we use Woolnet Gutiérrez and Patricio (2023), a multi-node Breadth-First Search (BFS) algorithm, to retrieve the most relevant triplets for a given claim present in the KG. This BFS algorithm initiates from multiple starting points and, at each step, searches for and processes all adjacent neighbors before advancing. It constructs a subgraph of visited nodes, tracking their origins, and distances from each BFS’s start. The algorithm expands each search tree one node at a time until paths intersect or reach a predefined maximum length. Upon intersection, it assesses if the discovered path meets the length criteria. If so, it logs the route, utilizing backtracking to trace the path to its origins, while ensuring there are no repetitions or cycles, thus maintaining a connection to a starting node. In our experiments, we allow for a maximum of three hops between any two given nodes, and a maximum of four potential paths. Adopting less stringent conditions leads to less relevant triplets. ### 3.3 Objective Function Previous works on evidence attribution tasks have established definitions for the categorization of input text with reference to a supporting source Rashkin et al. (2023); Gao et al. (2023); Bohnet et al. (2022); Yue et al. (2023). Similar to the formulation in Yue et al. (2023), we use three categories: Attributable, Extrapolatory, and Contradictory. However, there are two main differences that distinguish our approach from previous methods. First, we verify the input text against facts present in a KG, an aggregated information source constructed by integrating numerous data sources into a structure of triplets, instead of relying on a single reference. This approach eliminates the one-to-one dependency between the text and its information source. Second, we perform attribution with finer granularity, specifically at the claim level, involving a subtask of decomposing the input text into individual claims. We define our categories as follows: - Attributable: Triplets fully support the claim. - Extrapolatory: Triplets lack sufficient information to evaluate the claim. - Contradictory: Triplets contradict the claim. We formulate the objective function of our task as follows: where: - $input\_text$ : input text containing claim(s). - $ret\_triplets$ : retrieved triplets for the input text. - $claim\_span_i$ : $i^th$ claim extracted as a substring from $input\_text$ . - $claim\_pred_i$ : label predicted for $claim\_span_i$ . - $rel\_triplets_i$ : relevant subset of $ret\_triplets$ for $claim\_span_i$ . - $rationale_i$ : justification for $claim\_pred_i$ . - $n$ : total number of claims in $input\_text$ . This objective function encompasses two main sub-tasks: 1. Decomposing input text into claims. 1. Generating prediction and corresponding rationale for each claim by identifying relevant supporting triplets. ### 3.4 Fine-tuning LLMs The objective function shares similarities with the well-studied task of NLI Conneau et al. (2017); Schick and Schütze (2020). LLMs achieve state-of-the-art performance for NLI Chowdhery et al. (2023), making them a suitable choice to operationalize the objective function. Additionally, Yue et al. (2023) shows that LLMs can be used to automatically evaluate attribution to a given information source. However, these prior methods do not involve a complex sub-task, which is central to the proposed objective function, i.e., decomposing the input text into text spans that correspond to separate claims in the presence of multiple claims. It is crucial to perform both claim decomposition and attribution for all claims in a single step, as processing each claim individually can lead to an exponential increase in LLM queries, leading to significantly higher computational costs and latency issues. In order to perform attribution at the claim level, we need to fine-tune LLMs specifically for the proposed objective function (see § 3.3) using a custom dataset. This is necessary because, as of this writing, even the state-of-the-art model, OpenAI’s GPT-4 Achiam et al. (2023), does not perform satisfactorily right out of the box. Our custom dataset, built using two sequential complex prompts with GPT-4, enables us to fine-tune significantly smaller models. This approach distills the performance of a large proprietary model using a multi-query prompt pipeline into small open-source models with a compact zero-shot prompt. We make the weights of the fine-tuned models publicly available weights available on HuggingFace. We selected eight open-source LLMs with diverse sizes, ranging from 2B parameters to 10B parameters, to perform the fine-tuning: Gemma-2B-IT-Chat Team et al. (2024), Phi-3-mini-4k-Chat Javaheripi et al. (2023), Zephyr-7B-Beta-Chat Tunstall et al. (2023), Mistral-7B-v0.3-Chat Jiang et al. (2023a), Llama3-8B-Chat Touvron et al. (2023), and Solar-10.7B-Chat Kim et al. (2023). The models were fine-tuned using LoRA Hu et al. (2021) with 4-bit quantization and adapters with rank 8 Dettmers et al. (2024). The context length was set to 4096 tokens (for additional training details, refer § A.1) All models converged after 2 epochs, and high ROUGE-L Lin (2004) scores greater than 0.658 were achieved for each model. The instruction prompt used for fine-tuning is presented in Figure 3. Analyze text against provided triplets, classifying claims as "Attributable", "Contradictory", or "Extrapolatory". Justify your classification using the following structure: - "text_span": Text under evaluation. - "prediction": Category of the text (Attributable / Contradictory / Extrapolatory). - "triplets": Relevant triplets (if any, else "NA"). - "rationale": Reason for classification. For multiple claims, number each component (e.g., "text_span1", "prediction1",..). Use "NA" for inapplicable keys. Example: "text_span1": "Specific claim", "prediction1": "Attributable/Contradictory/Extrapolatory", "triplets1": "Relevant triplets", "rationale1": "Prediction justification", ... Input for analysis: -Text: {Input Text} -Triplets: {Retrieved Triplets} Figure 3: Instruction prompt for fine-tuned LLMs. ### 3.5 Computing Attribution Scores For various downstream tasks, such as ranking and filtering, a continuous score that reflects the validity of a given piece of text with respect to a KG is desirable. We propose the KG Attribution Score (KAS), which accomplishes this task with a high level of granularity, and is detailed in this section. #### 3.5.1 Claim Scores where, $y_i$ is $claim\_pred_i$ . For each claim, we assign a score that reflects the level of its validity, ranging from -1 (contradictory) to 2 (attributable). If a claim is predicted to be extrapolatory, yet has one or more relevant triplets, we assign that claim a score of 1, as there is still relevant information available even though it may not be sufficient to completely support or refute the claim. However, if there are no triplets at all, along with an extrapolatory prediction, we assign 0 as it does not add any useful information. While decomposing claims, the model might occasionally omit words, typically stop-words, and we assign 0 in those cases as well. #### 3.5.2 Triplets Match Score (TMS) This score reflects the extent of the match between the relevant triplets and the corresponding claim, and it can also serve as a proxy for the prediction confidence. Even though the prediction is made at the claim level, the triplets match score considers word-level matches in the computation. It can be computed as follows: where, $E(claim\_span_i)$ and $E(rel\_triplet_i)$ represent the sets of entities in $claim\_span_i$ and $rel\_triplet_i$ , respectively. $SS$ is the semantic similarity computed using the cosine similarity of text embeddings, and $EPR$ represents the ratio of entities in $E(claim\_span_i)$ that are also present in $E(rel\_triplet_i)$ . The parameters $α$ and $β$ can be adjusted as needed; in our experiments, we use 0.5 for both. In cases where examples of an entity retrieved from the KG are used to support the prediction, instead of the entity itself, we may not have a direct overlap, and thus semantic similarity would be helpful. $EPR$ rewards the direct use of the entity, so a balance between both may be ideal in most cases. #### 3.5.3 KG Attribution Score (KAS) For the final KG Attribution Score (KAS), a continuous score between 0 and 1 is desirable, as this facilitates various downstream applications such as ranking, fine-tuning, and filtering. This can be achieved using a Sigmoid function. However, the standard Sigmoid function treats positive and negative scores equally. In most cases, higher penalties should be assigned for erroneous text than rewards for valid text. This requirement can be met using a modified Sigmoid function that penalizes mistakes by a factor of $γ$ : $$ \displaystyleσ_mod(x,γ)=\frac{1}{1+e^-γ· x}, \displaystylewhere γ=\begin{cases}γ=3&if x<0,\\ γ=1&if x≥ 0,\end{cases} \tag{4} $$ In our experiments, we set the value of $γ$ to 3. Finally, the modified Sigmoid function, applied to the summation of triplet match scores and claim scores, is used to generate KAS: $$ \displaystyleKAS=σ_mod(∑_i=1^n[ \displaystyle TMS_i· cs(y_i)],γ) \tag{5} $$ ## 4 Dataset | Train Test | 3400 1000 | 5342 1677 | 998 316 | 3546 1068 | 798 293 | | --- | --- | --- | --- | --- | --- | Table 1: Distribution of fine-tuning dataset. Att: Attributable, Ext: Extrapolatory, Con: Contradictory. <details> <summary>x3.png Details</summary>  ### Visual Description ## Table: Claim Verification with Knowledge Triplets ### Overview The image displays a structured table with six rows and five columns. It appears to be an evaluation or analysis of textual claims against a knowledge base represented as "Relevant Triplets." Each claim is assessed for its supportability, resulting in a "Prediction (TMS)" with a confidence score, a "Rationale," and a final "KAS" score. The text in the "Input Text" column is color-coded (green, yellow, red), which likely corresponds to the prediction type (Attributable, Extrapolatory, Contradictory). ### Components/Axes **Column Headers:** 1. **Input Text**: Contains the claim to be verified. Text is color-coded and includes superscript numbers (¹, ², ³) to segment the claim into parts. 2. **Relevant Triplets**: Lists knowledge base entries (structured as (subject, predicate, object) tuples) relevant to the claim. Entries are numbered to correspond with the claim segments. "NA" indicates no relevant triplet was found for a segment. 3. **Prediction (TMS)**: Provides a classification for each claim segment (e.g., "Attributable," "Contradictory," "Extrapolatory") along with a confidence score in parentheses. 4. **Rationale**: A textual explanation justifying the prediction for each claim segment, referencing the relevant triplets. 5. **KAS**: A single numerical score (likely a Knowledge Accuracy Score or similar aggregate metric) for the entire input text claim. ### Detailed Analysis **Row 1:** * **Input Text (Green)**: "¹George O'Malley is a fictional character from the medical drama television series Grey's Anatomy, ²which airs on the American Broadcasting Company (ABC) in the United States." * **Relevant Triplets**: * 1: `[("Grey's Anatomy", 'characters', "George O'Malley")]` * 2: `[("Grey's Anatomy", 'original broadcaster', 'American Broadcasting Company'), ('American Broadcasting Company', 'country', 'United States of America')]` * **Prediction (TMS)**: * 1: Attributable (0.852) * 2: Attributable (0.637) * **Rationale**: * 1: The triplet directly supports the claim that George O'Malley is a character in Grey's Anatomy. * 2: The triplets confirm that Grey's Anatomy airs on ABC, which is based in the United States, directly supporting the claim about the show's broadcasting and location. * **KAS**: 0.818 **Row 2:** * **Input Text (Green & Yellow)**: "¹Bane was portrayed as a tertiary villain by Robert Swenson in Batman & Robin, ²directed by Joel Schumacher, ³and Batman's Back" * **Relevant Triplets**: * 1: `[('Batman & Robin', 'cast member', 'Robert Swenson')]` * 2: `[('Batman & Robin', 'director', 'Joel Schumacher')]` * 3: NA * **Prediction (TMS)**: * 1: Attributable (0.788) * 2: Attributable (0.882) * 3: Extrapolatory (0.0) * **Rationale**: * 1: The triplet directly supports the claim that Robert Swenson was involved in Batman & Robin, which is a requirement for the statement about his portrayal of Bane. * 2: The triplet directly supports the claim about the director of Batman & Robin, which is relevant to the context of the film. * 3: There are no triplets that directly support or refute the claim about Batman's back. * **KAS**: 0.752 **Row 3:** * **Input Text (Green & Yellow)**: "¹Crater Lake is the main feature of Crater Lake National Park ²and famous for its deep blue color and water clarity." * **Relevant Triplets**: * 1: `[('Crater Lake', 'located in protected area', 'Crater Lake National Park')]` * 2: NA * **Prediction (TMS)**: * 1: Attributable (0.942) * 2: Extrapolatory (0.0) * **Rationale**: * 1: The triplet directly supports the claim that Crater Lake is a significant feature within Crater Lake National Park, as it is located within the protected area. * 2: There are no triplets provided that directly support or refute the claim about the deep blue color and water clarity of Crater Lake. * **KAS**: 0.719 **Row 4:** * **Input Text (Green & Yellow)**: "¹Based in Blagnac , France, a suburb of Toulouse, ²and with significant activity across Europe, ³airbus produces approximately half of the world's jet airliners ." * **Relevant Triplets**: * 1: `[('Airbus Operations S.A.S.', 'country', 'France'), ('Airbus Corporate Jets', 'headquarters location', 'Toulouse'), ('Blagnac', 'country', 'France')]` * 2: NA * 3: NA * **Prediction (TMS)**: * 1: Attributable (0.505) * 2: Extrapolatory (0.0) * 3: Extrapolatory (0.0) * **Rationale**: * 1: The triplets confirm that Airbus Operations S.A.S. is in France, Airbus Corporate Jets is headquartered in Toulouse, and Blagnac is a suburb of Toulouse in France, supporting the statement about Airbus's location in France and its proximity to Toulouse. * 2: The triplets do not provide information about Airbus's activity across Europe. * 3: The triplets do not provide any information about Airbus's production output or market share. * **KAS**: 0.583 **Row 5:** * **Input Text (Red & Yellow)**: "¹Pope Benedict XVI never appointed anyone significant within the Catholic Church, ²nor did he ever teach the importance of understanding God's redemptive love." * **Relevant Triplets**: * 1: `[('Rutilio del Riego Jáñez', 'appointed by', 'Benedict XVI'), ('Rutilio del Riego Jáñez', 'religion or worldview', 'Catholic Church')]` * 2: `[('God', 'said to be the same as', 'love')]` * **Prediction (TMS)**: * 1: Contradictory (0.781) * 2: Extrapolatory (0.065) * **Rationale**: * 1: The triplets directly contradict the claim by showing that Pope Benedict XVI did indeed appoint someone (Rutilio del Riego Jáñez) who is associated with the Catholic Church, indicating that he did appoint significant individuals within the Church. * 2: While the triplets indicate that God is equated with love, it does not directly address whether Pope Benedict XVI taught the importance of understanding God's redemptive love. * **KAS**: 0.248 **Row 6:** * **Input Text (Red)**: "¹Southwest Airlines has never operated any Boeing 737 models." * **Relevant Triplets**: * 1: `[('Boeing 737 MAX', 'operator', 'Southwest Airlines'), ('Boeing 737 #1491', 'operator', 'Southwest Airlines')]` * **Prediction (TMS)**: * 1: Contradictory (0.933) * **Rationale**: * 1: The triplets directly contradict the claim by indicating that Southwest Airlines has operated both the Boeing 737 MAX and Boeing 737 #1491, which are specific models of the Boeing 737. This refutes the statement that Southwest Airlines has never operated any Boeing 737 models. * **KAS**: 0.057 ### Key Observations 1. **Color-Prediction Correlation**: There is a clear visual correlation between the color of the "Input Text" and the "Prediction (TMS)" for the primary claim segment. * **Green Text**: Corresponds to "Attributable" predictions (Rows 1, 2, 3, 4). * **Yellow Text**: Corresponds to "Extrapolatory" predictions, indicating a lack of direct supporting or contradicting evidence (Segments in Rows 2, 3, 4, 5). * **Red Text**: Corresponds to "Contradictory" predictions (Rows 5, 6). 2. **KAS Score Pattern**: The "KAS" score appears to be an aggregate measure of the claim's overall supportability. It is highest for claims with all-green, attributable text (Row 1: 0.818) and lowest for claims with red, contradictory text (Row 6: 0.057). 3. **Triplet Granularity**: The "Relevant Triplets" column shows that the knowledge base contains specific factual relationships (e.g., cast members, directors, locations, operators). The system matches claim segments to these granular facts. 4. **Handling of Missing Evidence**: When no relevant triplet is found ("NA"), the system defaults to an "Extrapolatory" prediction with a score of 0.0, as seen in multiple rows. ### Interpretation This table demonstrates a **knowledge-grounded claim verification system**. Its core function is to assess the veracity of natural language statements by decomposing them into segments and checking each against a structured knowledge base of factual triplets. * **What it shows**: The system can identify when a claim is directly supported by known facts ("Attributable"), when it is directly refuted by known facts ("Contradictory"), and when the knowledge base lacks evidence to make a determination ("Extrapolatory"). The confidence scores (e.g., 0.852, 0.933) suggest the system quantifies the strength of the match between the claim and the evidence. * **How elements relate**: The "Input Text" is the query. The "Relevant Triplets" are the retrieved evidence. The "Prediction (TMS)" is the system's judgment based on that evidence. The "Rationale" provides explainability by linking the judgment back to specific evidence. The "KAS" is a final, aggregated score for the entire claim. * **Notable patterns**: The strong correlation between text color and prediction type indicates this is likely a visualization of the system's output, where color is used for quick human interpretation. The system appears robust in handling partial claims, evaluating each segment independently (e.g., Row 2 has two attributable segments and one extrapolatory). The very low KAS scores for contradictory claims (0.248, 0.057) highlight the system's ability to confidently flag misinformation when direct counter-evidence exists. </details> Table 2: Examples of claim-level attribution by the proposed method. The first column shows the numbered claims in the input text. Second column lists relevant triplets for each claim. Predictions and Triplets Match Score (TMS) are in the third column, while the rationale behind each prediction is in the fourth column. The Knowledge Graph Attribution Score (KAS) is shown in the last column. Model: Solar-10.7B-Chat. Open-domain Question Answering (QA) datasets, such as WikiQA Yang et al. (2015), HotPotQA Yang et al. (2018), PopQA Mallen et al. (2022), and EntityQuestions Sciavolino et al. (2021), as well as Fact Verification datasets like FEVER Thorne et al. (2019), FEVEROUS Aly et al. (2021), TabFacT Chen et al. (2019), and SEM-TAB-FACTS Wang et al. (2021a), provide texts along with corresponding reference contexts or attributable information sources. However, these datasets significantly differ from the type of data required to train and test our proposed objective function, primarily due to two major factors: (i) these datasets predominantly offer samples that are inherently attributable, and (ii) consist of atomic claims and/or one-to-one mappings between input and reference texts. To address the first limitation, prior work Yue et al. (2023) in attribution evaluation introduced new samples by modifying correct answers to generate contradictory instances. Yet, this adjustment alone is not sufficient for our use case because our method requires attribution at the claim level, and necessitates the automatic decomposition input text to claims. Consequently, as this task represents a novel challenge, we developed a new dataset that enables effective training and testing of the objective function. Considering the choice of our KG, which is Wikidata Vrandečić and Krötzsch (2014), we opted for WikiQA Yang et al. (2015) as it is closely associated with the Wiki ecosystem. Given that our method is designed for text validation in general, and is not limited to question answering, we retain only answers and discard the questions. Subsequently, we processed the answers following the steps detailed in Section 3.1, selecting entries containing two or more Wiki entities. This approach resulted in the exclusion of most single-word answers and other responses that are dependent on their corresponding questions and may lack comprehensibility without them. We utilize GPT-4 Achiam et al. (2023) to generate the initial version of the ground truth. Although GPT-4 can adhere to the instructions (refer to Figure 3) to a reasonable degree and responds in the required format with all necessary keys, it still underperforms in the overall task. The most frequent issue observed is the erroneous assignment of prediction labels. To remedy this issue, we designed a detailed prompt tailored for the given task, incorporating techniques such as few-shot, chain-of-thought Kojima et al. (2022), and other strategies OpenAI (2024); Nori et al. (2023) (full prompt in § A Figure 12). We also conducted manual checks to ensure only high-quality samples were retained, as research indicates that high alignment can be achieved with as few as 1,000 samples, provided they are of superior quality Zhou et al. (2023). The final dataset is comprised of two splits: the training split, based on the training split of WikiQA Yang et al. (2015), and a test split, derived from both the test and validation splits. The training split contains 3,400 samples, and since some entries feature multiple claims, there are a total of 5,342 claims within this split. Similarly, the test split includes 1,000 samples and 1,677 claims. The label counts for the claims are tabulated in Table 1. The dataset is publicly shared to facilitate further research in this direction dataset available on HuggingFace. ## 5 Experiments and Results In this section, we present the evaluation of our claim-level attribution method. The performance metrics of the fine-tuned LLMs, which operationalize the objective function, are presented in Tables 3 and 4. In Table 3, we observe that all models converge and achieve sufficiently high ROUGE-L and ROUGE-1 scores, with Mistral-7B-v0.3-Chat achieving the highest of 0.694 and 0.719 respectively. We also observe that the smaller model, Gemma-2B-IT-Chat with just 2B parameters, is also sufficiently compatible for this task as it attained a decent ROUGE-L score of 0.667. The first task of the proposed objective function (refer § 3.3), decomposing text into multiple claims, is somewhat subjective, and there could be multiple valid approaches due to linguistic complexities. For instance, example 4 in Table 2 has been decomposed into three claims, but the first could arguably be further decomposed to verify whether Blagnac is in France, and whether it is a suburb of Toulouse. Controlling the precise manner of decomposition is challenging, and might necessitate an additional step before the prediction step, involving separate processing for each claim. However, this option could prove to be impractical, as the number of LLM queries could increase exponentially. To accurately compute classification performance, we impose a strict strategy: the text span of the claim, the identified relevant triplets, and the prediction label must all exactly match the ground truth to be considered accurate. In Table 4, the second column indicates number of claims with text spans exactly matching the ground truth responses. Columns 3 to 6 present the accuracy, precision, recall, and F1 scores for these matching claims. The most performant model is Solar-10.7B-Chat, with 1031 exact matches out of 1677 claims in the test set. Additionally, the classification scores in all metrics are above 89%, which clearly demonstrates that the model can reliably differentiate between the classes attributable, extrapolatory, and contradictory. Table 2 showcases the claim-level attribution performed by our method. Each claim in the input text is numbered and color-coded to reflect its prediction: green for attributable, amber for extrapolatory, and red for contradictory. The examples are sorted in descending order by their KAS scores, which reflect the validity of the text. As expected, we observe more green at the top of the table and more amber and eventually red as we move down. Since the Wiki ecosystem is open-domain, we observe that the examples cover a wide range of topics, demonstrating that the method is adaptable to diverse inputs. In the first example in 2, the input text is decomposed into two claims, both of which are attributable. The first claim is supported by a single triplet in the KG, while the second claim can be supported by combining two triplets. The second example presents more challenges for evaluation due to its complex sentence structure, but ClaimVer accurately identifies that the third claim regarding Batman’s Back is neither supported nor refuted by the triplets, as indicated in the rationale. In the third example, we note that the first claim is predicted to be attributable with a high triplet match score of 0.942 since there is a triplet that clearly supports the location description of Crater Lake. However, as there is no information regarding the water characteristics, the second claim is categorized as extrapolatory. In the fourth example, the first claim alone requires three triplets combined as supporting evidence, illustrating the method’s ability to handle complex multi-hop paths within the KG. The second and third claims are predicted to be extrapolatory, since there are no triplets concerning Airbus’s market share, or its activities in Europe, as highlighted in the model’s rationale. It is noteworthy that the context provided in the third claim is crucial for the first claim to be comprehensible, demonstrating why individual claim evaluation may be suboptimal. Interestingly, in the fifth example, the method identifies a specific instance from the KG to refute a general claim, citing the appointment of Rutilio del Riego Jáñez. Similarly, in the sixth example, the method provides specific instances, quoting two distinct Boeing 737 models to demonstrate contradiction with a high triplet match score. | Gemma-2B-IT-Chat Phi-3-mini-4k-Chat Zephyr-7B-Beta-Chat | 2B 4B 7B | 0.667 0.658 0.686 | 0.692 0.685 0.712 | | --- | --- | --- | --- | | Vicuna-7B-v1.5-Chat | 7B | 0.676 | 0.700 | | Mistral-7B-v0.3-Chat | 7B | 0.694 | 0.719 | | Gemma-7B-IT-Chat | 7B | 0.678 | 0.703 | | Llama3-8B-Chat | 8B | 0.679 | 0.705 | | Solar-10.7B-Chat | 10B | 0.689 | 0.714 | Table 3: ROUGE scores on the test set ( $n=1,000$ ). | Gemma-2B-IT-Chat Phi-3-mini-4k-Chat Zephyr-7B-Beta-Chat | 895 882 978 | 77.09 72.22 85.89 | 77.20 78.10 87.41 | 77.09 72.22 85.89 | 74.24 72.86 86.16 | | --- | --- | --- | --- | --- | --- | | Vicuna-7B-v1.5-Chat | 898 | 79.62 | 78.83 | 79.62 | 78.84 | | Mistral-7B-v0.3-Chat | 1002 | 86.63 | 87.03 | 86.63 | 86.73 | | Gemma-7B-IT-Chat | 940 | 82.87 | 84.09 | 82.87 | 83.17 | | Llama3-8B-Chat | 959 | 80.92 | 85.48 | 80.92 | 81.36 | | Solar-10.7B-Chat | 1031 | 89.23 | 89.52 | 89.23 | 89.30 | Table 4: Scores on matching claims in the test set ( $n=1677$ ). #MC: number of matching claims. ## 6 Discussion The susceptibility of LLMs to generating factually incorrect statements is an alarming concern as LLM-powered services become increasingly popular for seeking advice and information. The democratization of generative models has also had adverse effects, such as increasing misinformation Monteith et al. (2024). To arm end-users with the tools necessary to combat being misinformed, it is crucial to develop text-validation methods that are human-centric, and prioritize user engagement, understanding, and informativeness. We design our method with these principles in mind: we make predictions at the claim level, and identify text spans within the given text, that can be color-coded and presented to the user. The proposed method also generates easily comprehensible explanations along with the prediction and evidence, thus reducing the cognitive burden on the end-user, and making the process user-friendly. The usability and evaluation of these systems should align with human needs and capabilities. Chatbots, such as ChatGPT Achiam et al. (2023), serve a wide array of tasks; therefore, the text validation method should be adaptable to various domains. While KGs like Wikidata Vrandečić and Krötzsch (2014) are considered open-domain, the implementation of more specialized KGs, along with corresponding routing algorithms may be necessary to support a broader range of topics. For instance, a common-sense KG Hwang et al. (2020) would be more useful in validating non-factoid answers that involve logic. Furthermore, the maintenance efficiency of our approach aligns well with the need for sustainable, long-term AI solutions. In a world where information is constantly evolving, the ability to update and maintain AI systems with minimal effort is not just a convenience, but a necessity. This directly ties into the ethical implications of AI, where outdated or incorrect information can lead to harmful decisions. By leveraging existing, well-maintained KGs, we can ensure that AI systems remain accurate and relevant over time. ## 7 Conclusion In this paper, we present ClaimVer, a framework for text verification and evidence attribution at the claim level by leveraging information present in KGs. In contrast to other methods, ClaimVer eliminates the one-to-one mapping between input and reference text, allowing for layered interpretation and handling of distributed information. In addition to these primary functions, ClaimVer incorporates human-centric design principles by offering clear, concise explanations for each claim prediction—an important characteristic for building user trust and enhancing usability. Furthermore, we introduce an attribution score, which enhances its applicability across a wide range of downstream tasks. Finally, we share ClaimVer fine-tuned LLMs to facilitate further exploration of this research direction. ## 8 Limitations Limitations of LLMs for Fact Verification. Like most ML models, LLMs are prone to erroneous predictions, which is particularly concerning in sensitive applications such as handling misinformation. Despite this, they remain the most performant techniques for fact verification and related tasks like NLI Yue et al. (2023); Wang et al. (2021b). Therefore, while it is reasonable to use the best option available, fact verification systems relying on LLMs should be utilized with caution and necessary validations. Limitations of Knowledge Graphs. While there are several advantages associated with using KGs, we also acknowledge the presence of known issues, such as knowledge coverage and the efforts required to keep these sources up-to-date. For our solution, we assume that the KG is up-to-date and possesses adequate coverage. However, this may not always be the case, and thus the most suitable technique should be adopted after considering the specific requirements of a particular use case. Another point to consider is that the proposed method does not provide traditional citations to articles, although it may be possible to retrieve that information from the KG, if information source mapping has been properly maintained. Variations in Claim Decomposition. Decomposing text into multiple claims is a complex linguistic task that often results in multiple valid decompositions. Although this may not impact usability if the prediction, rationale, and text spans are comprehensible and supported by facts from the KG, it poses a challenge for evaluating model performance. One potential approach is to operate at the token level instead of the span level, but this would significantly complicate the problem space. Additionally, token-level verification and attribution would require substantially higher compute resources, necessitating further studies to assess their value and impact on system usability and reliability. LLM Reasoning Errors. Previous works have demonstrated that using LLM reasoning for tasks like fact verification, evidence attribution, and NLI can yield impressive results, surpassing alternative approaches Yue et al. (2023); Wang et al. (2021b). However, LLM reasoning can still be flawed. In our work, we impose validations to minimize LLM mistakes by performing membership checks for supporting triplets and string matching for text spans. Yet, validating reasoning remains an open problem with ongoing research efforts. Fine-tuning Dataset Limitations. To build the fine-tuning dataset, we utilized GPT-4 with two detailed sequential prompts designed in accordance with OpenAI’s recommendations OpenAI (2024) and previous works Nori et al. (2023). Despite employing techniques like few-shot prompting with state-of-the-art LLMs, we still observe mistakes, indicating the complexity of this problem. To address this, we conducted manual checks to minimize errors and share the dataset with the research community for further improvement. ## 9 Ethical Concerns Our work presents a scalable and interpretable framework for fact-checking textual claims. To promote the exploration of this important problem space, we fine-tune and share open-source LLMs that are well-aligned to the framework’s objective function. While the models we provide perform better than the publicly available base models for our specific task, they still share similar weaknesses such as erroneous reasoning. To address this as best as we can, we have incorporated and described ways to mitigate these issues to the extent possible. We believe the benefits of this work outweigh any potential risks. ## References - Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. - Aly et al. (2021) Rami Aly, Zhijiang Guo, Michael Schlichtkrull, James Thorne, Andreas Vlachos, Christos Christodoulopoulos, Oana Cocarascu, and Arpit Mittal. 2021. Feverous: Fact extraction and verification over unstructured and structured information. arXiv preprint arXiv:2106.05707. - Bohnet et al. (2022) Bernd Bohnet, Vinh Q Tran, Pat Verga, Roee Aharoni, Daniel Andor, Livio Baldini Soares, Massimiliano Ciaramita, Jacob Eisenstein, Kuzman Ganchev, Jonathan Herzig, et al. 2022. Attributed question answering: Evaluation and modeling for attributed large language models. arXiv preprint arXiv:2212.08037. - Chen et al. (2019) Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. 2019. Tabfact: A large-scale dataset for table-based fact verification. arXiv preprint arXiv:1909.02164. - Chowdhery et al. (2023) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2023. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1–113. - Conneau et al. (2017) Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. arXiv preprint arXiv:1705.02364. - Dettmers et al. (2024) Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2024. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36. - Dong and Smith (2021) Rui Dong and David A Smith. 2021. Structural encoding and pre-training matter: Adapting bert for table-based fact verification. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2366–2375. - Gao et al. (2023) Luyu Gao, Zhuyun Dai, Panupong Pasupat, Anthony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vincent Zhao, Ni Lao, Hongrae Lee, Da-Cheng Juan, et al. 2023. Rarr: Researching and revising what language models say, using language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16477–16508. - Gerber (2023) Emanuel Gerber. 2023. spacy module for linking text to wikidata items. https://github.com/egerber/spaCy-entity-linker. Accessed: 2024-02-26. - Guan et al. (2023) Jian Guan, Jesse Dodge, David Wadden, Minlie Huang, and Hao Peng. 2023. Language models hallucinate, but may excel at fact verification. arXiv preprint arXiv:2310.14564. - Gutiérrez and Patricio (2023) Torres Gutiérrez and Cristóbal Patricio. 2023. Sistema visual para explorar subgrafos temáticos en wikidata. - Hu et al. (2021) Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685. - Hwang et al. (2020) Jena D. Hwang, Chandra Bhagavatula, Ronan Le Bras, Jeff Da, Keisuke Sakaguchi, Antoine Bosselut, and Yejin Choi. 2020. Comet-atomic 2020: On symbolic and neural commonsense knowledge graphs. In AAAI Conference on Artificial Intelligence. - Javaheripi et al. (2023) Mojan Javaheripi, Sébastien Bubeck, Marah Abdin, Jyoti Aneja, Sebastien Bubeck, Caio César Teodoro Mendes, Weizhu Chen, Allie Del Giorno, Ronen Eldan, Sivakanth Gopi, et al. 2023. Phi-2: The surprising power of small language models. Microsoft Research Blog. - Jiang et al. (2023a) Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023a. Mistral 7b. arXiv preprint arXiv:2310.06825. - Jiang et al. (2023b) Zhengbao Jiang, Frank F Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023b. Active retrieval augmented generation. arXiv preprint arXiv:2305.06983. - Kim et al. (2023) Dahyun Kim, Chanjun Park, Sanghoon Kim, Wonsung Lee, Wonho Song, Yunsu Kim, Hyeonwoo Kim, Yungi Kim, Hyeonju Lee, Jihoo Kim, et al. 2023. Solar 10.7 b: Scaling large language models with simple yet effective depth up-scaling. arXiv preprint arXiv:2312.15166. - Kojima et al. (2022) Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199–22213. - Lee et al. (2017) Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017. End-to-end neural coreference resolution. arXiv preprint arXiv:1707.07045. - Lee et al. (2020) Nayeon Lee, Belinda Z Li, Sinong Wang, Wen-tau Yih, Hao Ma, and Madian Khabsa. 2020. Language models as fact checkers? arXiv preprint arXiv:2006.04102. - Lin (2004) Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics. - Mallen et al. (2022) Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Hannaneh Hajishirzi, and Daniel Khashabi. 2022. When not to trust language models: Investigating effectiveness and limitations of parametric and non-parametric memories. arXiv preprint arXiv:2212.10511. - Min et al. (2023) Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. Preprint, arXiv:2305.14251. - Monteith et al. (2024) Scott Monteith, Tasha Glenn, John R Geddes, Peter C Whybrow, Eric Achtyes, and Michael Bauer. 2024. Artificial intelligence and increasing misinformation. The British Journal of Psychiatry, 224(2):33–35. - Nori et al. (2023) Harsha Nori, Yin Tat Lee, Sheng Zhang, Dean Carignan, Richard Edgar, Nicolo Fusi, Nicholas King, Jonathan Larson, Yuanzhi Li, Weishung Liu, et al. 2023. Can generalist foundation models outcompete special-purpose tuning? case study in medicine. arXiv preprint arXiv:2311.16452. - OpenAI (2024) OpenAI. 2024. Best practices for prompt engineering with the openai api. https://help.openai.com/en/articles/6654000-best-practices-for-prompt-engineering-with-the-openai-api. Accessed:2024-01-11. - Rashkin et al. (2023) Hannah Rashkin, Vitaly Nikolaev, Matthew Lamm, Lora Aroyo, Michael Collins, Dipanjan Das, Slav Petrov, Gaurav Singh Tomar, Iulia Turc, and David Reitter. 2023. Measuring attribution in natural language generation models. Computational Linguistics, pages 1–64. - Rechkemmer and Yin (2022) Amy Rechkemmer and Ming Yin. 2022. When confidence meets accuracy: Exploring the effects of multiple performance indicators on trust in machine learning models. In Proceedings of the 2022 chi conference on human factors in computing systems, pages 1–14. - Russell et al. (2015) Ginny Russell, Stephan Collishaw, Jean Golding, Susan E Kelly, and Tamsin Ford. 2015. Changes in diagnosis rates and behavioural traits of autism spectrum disorder over time. BJPsych open, 1(2):110–115. - Schick and Schütze (2020) Timo Schick and Hinrich Schütze. 2020. Exploiting cloze questions for few shot text classification and natural language inference. arXiv preprint arXiv:2001.07676. - Sciavolino et al. (2021) Christopher Sciavolino, Zexuan Zhong, Jinhyuk Lee, and Danqi Chen. 2021. Simple entity-centric questions challenge dense retrievers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6138–6148, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. - Shin (2021) Donghee Shin. 2021. The effects of explainability and causability on perception, trust, and acceptance: Implications for explainable ai. International Journal of Human-Computer Studies, 146:102551. - Team et al. (2024) Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. 2024. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295. - Thorne et al. (2018) James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. Fever: a large-scale dataset for fact extraction and verification. arXiv preprint arXiv:1803.05355. - Thorne et al. (2019) James Thorne, Andreas Vlachos, Oana Cocarascu, Christos Christodoulopoulos, and Arpit Mittal. 2019. The FEVER2.0 shared task. In Proceedings of the Second Workshop on Fact Extraction and VERification (FEVER), pages 1–6, Hong Kong, China. Association for Computational Linguistics. - Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. - Tunstall et al. (2023) Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al. 2023. Zephyr: Direct distillation of lm alignment. arXiv preprint arXiv:2310.16944. - Vrandečić and Krötzsch (2014) Denny Vrandečić and Markus Krötzsch. 2014. Wikidata: a free collaborative knowledgebase. Communications of the ACM, 57(10):78–85. - Wang et al. (2021a) Nancy XR Wang, Diwakar Mahajan, Marina Danilevsky, and Sara Rosenthal. 2021a. Semeval-2021 task 9: Fact verification and evidence finding for tabular data in scientific documents (sem-tab-facts). arXiv preprint arXiv:2105.13995. - Wang et al. (2021b) Sinong Wang, Han Fang, Madian Khabsa, Hanzi Mao, and Hao Ma. 2021b. Entailment as few-shot learner. arXiv preprint arXiv:2104.14690. - Weitz et al. (2019) Katharina Weitz, Dominik Schiller, Ruben Schlagowski, Tobias Huber, and Elisabeth André. 2019. " do you trust me?" increasing user-trust by integrating virtual agents in explainable ai interaction design. In Proceedings of the 19th ACM International Conference on Intelligent Virtual Agents, pages 7–9. - Yang et al. (2015) Yi Yang, Wen-tau Yih, and Christopher Meek. 2015. WikiQA: A challenge dataset for open-domain question answering. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2013–2018, Lisbon, Portugal. Association for Computational Linguistics. - Yang et al. (2018) Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv:1809.09600. - Yao et al. (2022) Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629. - Yue et al. (2023) Xiang Yue, Boshi Wang, Kai Zhang, Ziru Chen, Yu Su, and Huan Sun. 2023. Automatic evaluation of attribution by large language models. arXiv preprint arXiv:2305.06311. - Zhou et al. (2023) Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. 2023. Lima: Less is more for alignment. arXiv preprint arXiv:2305.11206. ## Appendix A Appendix ### A.1 Training Details In this section, we present the training parameters used for fine-tuning each model, along with their corresponding loss plots. All models converged after two epochs, achieving ROUGE-L Lin (2004) scores greater than 0.658, with the best model reaching 0.719. <details> <summary>x4.png Details</summary>  ### Visual Description ## Line Chart: Llama3-8B-Chat Training Loss ### Overview This image is a line chart displaying the training loss over epochs for a machine learning model identified as "Llama3-8B-Chat". It plots two versions of the loss metric: the raw, noisy "Original" data and a "Smoothed" version of the same data. The chart demonstrates the model's learning progress, showing a characteristic rapid initial decrease in loss followed by a plateau. ### Components/Axes * **Title:** "Llama3-8B-Chat" (centered at the top). * **Y-Axis:** Labeled "Loss". The scale runs from 0.00 to 2.00, with major gridlines and numerical markers at intervals of 0.25 (0.00, 0.25, 0.50, 0.75, 1.00, 1.25, 1.50, 1.75, 2.00). * **X-Axis:** Labeled "Epoch". The scale runs from 0.00 to 2.00, with major gridlines and numerical markers at intervals of 0.25 (0.00, 0.25, 0.50, 0.75, 1.00, 1.25, 1.50, 1.75, 2.00). * **Legend:** Located in the top-right corner of the plot area. It contains two entries: * "Original" - represented by a solid magenta/dark pink line. * "Smoothed" - represented by a solid light pink line. * **Plot Area:** Contains a light gray grid for reference. ### Detailed Analysis **Data Series Trends:** 1. **Original (Magenta Line):** * **Trend:** Shows a steep, exponential-like decay initially, which then flattens into a noisy plateau. * **Key Points (Approximate):** * Starts at a loss of ~1.00 at Epoch 0.00. * Drops sharply to ~0.60 by Epoch 0.10. * Continues a slower decline, reaching ~0.45 by Epoch 0.50. * From Epoch 1.00 to 2.00, the loss fluctuates noisily between approximately 0.30 and 0.40, showing no clear downward trend in this later phase. The final value at Epoch 2.00 is approximately 0.35. 2. **Smoothed (Light Pink Line):** * **Trend:** Shows a smooth, monotonic decrease that approximates the central tendency of the "Original" line, filtering out high-frequency noise. * **Key Points (Approximate):** * Starts at the same point as the Original line: ~1.00 at Epoch 0.00. * Declines smoothly, passing through ~0.70 at Epoch 0.25, ~0.55 at Epoch 0.75, and ~0.45 at Epoch 1.25. * The curve flattens significantly after Epoch 1.50, ending at approximately 0.35 at Epoch 2.00, converging with the general level of the noisy Original line. ### Key Observations 1. **Rapid Initial Learning:** The most significant reduction in loss occurs within the first 0.25 epochs, where the loss drops by nearly half (from ~1.00 to ~0.55). 2. **Convergence and Plateau:** After approximately 1.0 epoch, the rate of loss reduction slows dramatically. Both lines indicate the model's training has largely stabilized, with the loss value oscillating around a mean of ~0.35 for the final epoch. 3. **Noise in Raw Data:** The "Original" line exhibits significant step-like fluctuations, especially visible after Epoch 0.5. This is typical of stochastic gradient descent-based training, where loss is calculated on mini-batches. 4. **Effect of Smoothing:** The "Smoothed" line provides a clearer view of the underlying learning curve, removing the batch-to-batch variance to show the overall trajectory of model improvement. ### Interpretation This chart is a standard diagnostic tool for monitoring the training of a large language model (LLM). The "Loss" metric quantifies the model's prediction error on its training data; a decreasing trend indicates the model is learning. * **What the data suggests:** The Llama3-8B-Chat model successfully learned from its training data over the course of 2 epochs. The steep initial drop shows it quickly grasped the most obvious patterns. The subsequent plateau suggests it reached a point of diminishing returns, where additional training epochs yield only minor improvements in fitting the training data. * **Relationship between elements:** The "Smoothed" line is derived from the "Original" line, likely using a moving average or exponential smoothing. Its purpose is to aid human interpretation by highlighting the trend amidst the inherent noise of the training process. * **Notable patterns/anomalies:** There are no apparent anomalies. The curve shape is textbook for a well-behaved training run. The final loss value (~0.35) is a relative measure; its absolute significance depends on the specific loss function and task, but the clear convergence is the key positive indicator. The fact that the smoothed and original lines converge at the end confirms the smoothing window is appropriate and the model has stabilized. </details> Figure 4: Fine-tuning loss plots for Llama3-8B-Chat. | Parameter Base Model ROUGE-L | Value meta-llama/Meta-Llama-3-8B-Instruct 0.679 | | --- | --- | | ROUGE-1 | 0.705 | | Fine-Tuning Type | LoRA | | LoRA Alpha | 16 | | LoRA Rank | 8 | | Cutoff Length | 4096 | | Gradient Accumulation Steps | 8 | | Learning Rate | 5.0e-05 | | LR Scheduler Type | Cosine | | Number of Training Epochs | 2.0 | | Optimizer | AdamW | | Quantization Bit | 4 | Table 5: Fine-tuning Parameters for Llama3-8B-Chat <details> <summary>x5.png Details</summary>  ### Visual Description ## Line Chart: Training Loss Curve for Mistral-7B-v0.3-Chat ### Overview This image is a line chart displaying the training loss over epochs for a machine learning model identified as "Mistral-7B-v0.3-Chat". It plots two data series: the raw, "Original" loss values and a "Smoothed" version of the same data. The chart demonstrates the model's learning progress, showing a characteristic rapid initial decrease in loss followed by a plateau. ### Components/Axes * **Chart Title:** "Mistral-7B-v0.3-Chat" (centered at the top). * **X-Axis:** Labeled "Epoch". The scale runs from 0.00 to 2.00, with major tick marks at intervals of 0.25 (0.00, 0.25, 0.50, 0.75, 1.00, 1.25, 1.50, 1.75, 2.00). * **Y-Axis:** Labeled "Loss". The scale runs from 0.00 to 2.00, with major tick marks at intervals of 0.25 (0.00, 0.25, 0.50, 0.75, 1.00, 1.25, 1.50, 1.75, 2.00). * **Legend:** Positioned in the top-right corner of the plot area. It contains two entries: * "Original" - represented by a dark purple line. * "Smoothed" - represented by a light purple (lavender) line. * **Grid:** A light gray grid is present, with lines corresponding to the major tick marks on both axes. ### Detailed Analysis **Data Series Trends & Approximate Values:** 1. **Original (Dark Purple Line):** * **Trend:** The line exhibits a very steep, near-vertical drop at the beginning, followed by a sharp knee, and then a much more gradual, fluctuating decline that plateaus. * **Key Points (Approximate):** * Epoch 0.00: Loss starts at approximately **0.75**. * Epoch ~0.05: Loss drops sharply to ~**0.50**. * Epoch ~0.10: Loss is ~**0.45**. * Epoch ~0.25: Loss is ~**0.35**. * Epoch 0.50: Loss is ~**0.30**. * From Epoch 0.50 to 2.00: The loss fluctuates in a narrow band between approximately **0.25 and 0.30**, showing minor ups and downs but no significant downward trend. The final value at Epoch 2.00 is approximately **0.27**. 2. **Smoothed (Light Purple Line):** * **Trend:** This line shows a smooth, continuous, and decelerating decline from the start to the end of the plotted epochs. It lacks the noise and sharp initial drop of the original data. * **Key Points (Approximate):** * Epoch 0.00: Loss starts at approximately **0.75** (same starting point as Original). * Epoch 0.25: Loss is ~**0.60**. * Epoch 0.50: Loss is ~**0.50**. * Epoch 1.00: Loss is ~**0.35**. * Epoch 1.50: Loss is ~**0.28**. * Epoch 2.00: Loss ends at approximately **0.26**, very close to the final value of the Original line. ### Key Observations * **Rapid Initial Learning:** The most dramatic reduction in loss occurs within the first 0.1 epochs (approximately the first 5% of the displayed training time). * **Plateau Phase:** After approximately epoch 0.5, the model's loss enters a plateau phase, indicating that further training yields minimal improvement in the loss metric. * **Convergence of Series:** While the smoothed line starts above the original line after the initial drop, the two series converge and become nearly indistinguishable from approximately epoch 1.5 onward. * **Noise in Original Data:** The "Original" line shows visible high-frequency fluctuations, especially in the plateau region, which the "Smoothed" line successfully filters out to reveal the underlying trend. ### Interpretation This chart is a classic visualization of a model's training dynamics. The steep initial drop signifies that the model is quickly learning the most prominent patterns in the training data. The subsequent plateau suggests the model has reached a point of diminishing returns, where additional training epochs provide only marginal gains in fitting the training set. The "Smoothed" line is crucial for interpretation, as it clarifies the overall trajectory by removing the stochastic noise inherent in batch-based training loss calculations. The fact that both lines converge to a similar final value (~0.26-0.27) indicates the smoothing process is accurate and the model's performance has stabilized. For a practitioner, this chart would signal that training beyond 1.5-2.0 epochs may not be cost-effective unless the loss needs to be driven marginally lower for specific performance requirements. </details> Figure 5: Fine-tuning loss plots for Mistral-7B-v0.3-Chat. | Parameter Base Model ROUGE-L | Value mistralai/Mistral-7B-Instruct-v0.3 0.694 | | --- | --- | | ROUGE-1 | 0.719 | | Fine-Tuning Type | LoRA | | LoRA Alpha | 16 | | LoRA Rank | 8 | | Cutoff Length | 4096 | | Gradient Accumulation Steps | 8 | | Learning Rate | 5.0e-05 | | LR Scheduler Type | Cosine | | Number of Training Epochs | 2.0 | | Optimizer | AdamW | | Quantization Bit | 4 | Table 6: Fine-tuning Parameters for Mistral-7B-v0.3-Chat <details> <summary>x6.png Details</summary>  ### Visual Description ## Line Chart: Phi-3-mini-4k-Chat Training Loss ### Overview This image is a line chart visualizing the training loss of a machine learning model named "Phi-3-mini-4k-Chat" over a period of 2 epochs. It displays two data series: the raw, fluctuating loss values ("Original") and a smoothed version of the same data ("Smoothed"). ### Components/Axes * **Chart Title:** "Phi-3-mini-4k-Chat" (centered at the top). * **Y-Axis (Vertical):** Labeled "Loss". The scale runs from 0.00 to 2.00, with major gridlines and numerical markers at intervals of 0.25 (0.00, 0.25, 0.50, 0.75, 1.00, 1.25, 1.50, 1.75, 2.00). * **X-Axis (Horizontal):** Labeled "Epoch". The scale runs from 0.00 to 2.00, with major gridlines and numerical markers at intervals of 0.25 (0.00, 0.25, 0.50, 0.75, 1.00, 1.25, 1.50, 1.75, 2.00). * **Legend:** Located in the top-right corner of the plot area. It contains two entries: * "Original" - represented by a solid, darker orange line. * "Smoothed" - represented by a solid, lighter peach-colored line. * **Plot Area:** Contains a light gray grid for reference. ### Detailed Analysis **Trend Verification:** * **Original Line (Darker Orange):** This line exhibits a steep downward slope initially, followed by a gradual flattening with persistent small-scale fluctuations (noise). The overall trend is a clear decrease in loss. * **Smoothed Line (Lighter Peach):** This line shows a smooth, monotonic decrease, curving gently from the top-left to the bottom-right. It represents the underlying trend of the "Original" data without the noise. **Data Point Extraction (Approximate Values):** * **Epoch 0.00:** Both lines originate at the same point. Loss ≈ 0.85. * **Epoch 0.25:** * Original: Loss ≈ 0.55 (steep drop from start). * Smoothed: Loss ≈ 0.70. * **Epoch 0.50:** * Original: Loss ≈ 0.40 (with a small local peak just before). * Smoothed: Loss ≈ 0.55. * **Epoch 1.00:** * Original: Loss ≈ 0.30 (with minor oscillations). * Smoothed: Loss ≈ 0.40. * **Epoch 1.50:** * Original: Loss ≈ 0.28 (oscillating between ~0.27 and 0.30). * Smoothed: Loss ≈ 0.32. * **Epoch 2.00 (End):** * Original: Loss ≈ 0.29 (ending with a slight upward tick). * Smoothed: Loss ≈ 0.30. The two lines nearly converge at the end of the plotted period. ### Key Observations 1. **Rapid Initial Learning:** The most significant reduction in loss occurs within the first 0.5 epochs, where the "Original" loss drops by more than half its initial value. 2. **Noise vs. Trend:** The "Original" line contains high-frequency noise, which is effectively filtered out in the "Smoothed" line, making the long-term trend easier to interpret. 3. **Convergence:** The gap between the "Original" and "Smoothed" lines narrows considerably after epoch 1.00, suggesting the rate of change in the underlying trend is slowing and the noise is centered around this slower trend. 4. **Plateauing:** Both lines show clear signs of plateauing after epoch 1.25, indicating the model's rate of improvement (reduction in loss) has significantly diminished. ### Interpretation This chart demonstrates the typical learning curve of a neural network during training. The "Loss" metric quantifies the model's error; a decreasing trend indicates the model is learning from the training data. * **What the data suggests:** The Phi-3-mini-4k-Chat model learns rapidly at the beginning of training (first half-epoch). As training progresses, learning continues but at a diminishing rate, eventually approaching a plateau by the end of the second epoch. The presence of noise in the "Original" loss is normal and can be caused by mini-batch sampling during stochastic gradient descent. * **Relationship between elements:** The "Smoothed" line is derived from the "Original" line, likely using a moving average or similar technique. Its purpose is to reveal the central tendency of the loss, making it easier to assess whether the model is still fundamentally improving, despite step-to-step fluctuations. * **Notable trends/anomalies:** There are no major anomalies. The curve is well-behaved. The slight uptick in the "Original" loss at the very end (epoch 2.00) is minor and within the range of normal noise; it does not yet suggest overfitting, which would typically be indicated by a sustained increase in loss on a *validation* set (not shown here). The primary takeaway is that the model has largely converged within the observed 2-epoch window. </details> Figure 6: Fine-tuning loss plots for Phi-3-mini-4k-Chat. | Parameter Base Model ROUGE-L | Value microsoft/Phi-3-mini-4k-instruct 0.658 | | --- | --- | | ROUGE-1 | 0.685 | | Fine-Tuning Type | LoRA | | LoRA Alpha | 16 | | LoRA Rank | 8 | | Cutoff Length | 4096 | | Gradient Accumulation Steps | 8 | | Learning Rate | 5.0e-05 | | LR Scheduler Type | Cosine | | Number of Training Epochs | 2.0 | | Optimizer | AdamW | | Quantization Bit | 4 | Table 7: Fine-tuning Parameters for Phi-3-mini-4k-Chat <details> <summary>x7.png Details</summary>  ### Visual Description ## Line Chart: Solar-10.7B-Chat Training Loss ### Overview This image is a line chart titled "Solar-10.7B-Chat," displaying the training loss of a machine learning model over a series of epochs. It plots two data series: the raw, "Original" loss values and a "Smoothed" version of the same data. The chart demonstrates a classic learning curve where the model's error (loss) decreases as training progresses. ### Components/Axes * **Title:** "Solar-10.7B-Chat" (centered at the top). * **Y-Axis (Vertical):** Labeled "Loss". The scale runs from 0.00 to 2.00, with major gridlines and numerical markers at intervals of 0.25 (0.00, 0.25, 0.50, 0.75, 1.00, 1.25, 1.50, 1.75, 2.00). * **X-Axis (Horizontal):** Labeled "Epoch". The scale runs from 0.00 to 2.00, with major gridlines and numerical markers at intervals of 0.25 (0.00, 0.25, 0.50, 0.75, 1.00, 1.25, 1.50, 1.75, 2.00). * **Legend:** Located in the top-right corner of the plot area. It contains two entries: * A dark gray line labeled "Original". * A light gray line labeled "Smoothed". * **Plot Area:** Contains a light gray grid for reference. ### Detailed Analysis **Data Series & Trends:** 1. **"Original" Line (Dark Gray):** * **Trend:** Exhibits a very steep, near-vertical decline at the beginning of training, followed by a rapid deceleration in the rate of decrease, eventually plateauing with minor fluctuations. * **Approximate Data Points:** * Epoch 0.00: Loss ≈ 0.78 * Epoch 0.05: Loss ≈ 0.48 (sharp drop) * Epoch 0.25: Loss ≈ 0.35 * Epoch 0.50: Loss ≈ 0.30 * Epoch 1.00: Loss ≈ 0.26 * Epoch 1.50: Loss ≈ 0.25 * Epoch 2.00: Loss ≈ 0.25 2. **"Smoothed" Line (Light Gray):** * **Trend:** Shows a smooth, continuous, and decelerating downward curve. It starts at approximately the same point as the Original line but decreases more gradually, acting as a trendline that filters out the noise/fluctuations of the raw data. * **Approximate Data Points:** * Epoch 0.00: Loss ≈ 0.78 * Epoch 0.25: Loss ≈ 0.60 * Epoch 0.50: Loss ≈ 0.48 * Epoch 0.75: Loss ≈ 0.40 * Epoch 1.00: Loss ≈ 0.35 * Epoch 1.50: Loss ≈ 0.28 * Epoch 2.00: Loss ≈ 0.26 **Spatial Grounding & Cross-Reference:** The legend is positioned in the top-right quadrant. The dark gray "Original" line is consistently below the light gray "Smoothed" line after the initial point, confirming the legend's color mapping. Both lines converge to a very similar value (≈0.25-0.26) by Epoch 2.00. ### Key Observations 1. **Rapid Initial Learning:** The most significant reduction in loss occurs within the first 0.25 epochs, indicating the model quickly learns the most prominent patterns in the data. 2. **Diminishing Returns:** After approximately Epoch 0.50, the rate of loss reduction slows dramatically. The curve flattens, suggesting the model is fine-tuning or has reached a performance plateau for this training phase. 3. **Convergence:** The "Original" and "Smoothed" lines converge towards the end of the plotted range (Epoch 2.00), indicating that the smoothed trend accurately reflects the final, stable loss value. 4. **Low Final Loss:** The final loss value (≈0.25) is relatively low on the provided scale (0-2), suggesting successful model training on the given task. ### Interpretation This chart is a fundamental diagnostic tool for monitoring the training of the "Solar-10.7B-Chat" model. The data suggests the training process was effective and stable. * **What the data demonstrates:** The model successfully learned from the training data, as evidenced by the consistent decrease in loss. The shape of the curve is typical for gradient-based optimization, where large initial gains are followed by smaller, incremental improvements. * **Relationship between elements:** The "Smoothed" line provides a clear view of the underlying learning trend, while the "Original" line shows the inherent variability or noise in the loss calculation at each step. Their convergence validates that the smoothing process did not distort the final outcome. * **Notable implications:** The plateau after Epoch 1.00 could indicate several possibilities: the model may have reached its capacity for the given data and architecture, the learning rate might need adjustment for further gains, or additional training epochs may yield only marginal improvements. The chart does not show signs of overfitting (which would be indicated by a rising loss on a validation set, not shown here) or training instability (e.g., wild spikes). The primary takeaway is that the core learning phase was completed efficiently within the first half of the displayed training period. </details> Figure 7: Fine-tuning loss plots for SOLAR-10.7B-Chat. | Parameter Base Model ROUGE-L | Value upstage/SOLAR-10.7B-Instruct-v1.0 0.689 | | --- | --- | | ROUGE-1 | 0.714 | | Fine-Tuning Type | LoRA | | LoRA Alpha | 16 | | LoRA Rank | 8 | | Cutoff Length | 4096 | | Gradient Accumulation Steps | 8 | | Learning Rate | 5.0e-05 | | LR Scheduler Type | Cosine | | Number of Training Epochs | 2.0 | | Optimizer | AdamW | | Quantization Bit | 4 | Table 8: Fine-tuning Parameters for SOLAR-10.7B-Chat <details> <summary>x8.png Details</summary>  ### Visual Description ## Line Chart: Loss Curve for Vicuna-7B-v1.5-Chat Model Training ### Overview The image displays a line chart titled "Vicuna-7B-v1.5-Chat," illustrating the training loss of a machine learning model over a series of epochs. The chart compares two data series: the raw, fluctuating loss values ("Original") and a smoothed version of the same data ("Smoothed"). ### Components/Axes * **Title:** "Vicuna-7B-v1.5-Chat" (centered at the top). * **Y-Axis (Vertical):** Labeled "Loss". The scale ranges from 0.00 to 2.00, with major gridlines and numerical markers at intervals of 0.25 (0.00, 0.25, 0.50, 0.75, 1.00, 1.25, 1.50, 1.75, 2.00). * **X-Axis (Horizontal):** Labeled "Epoch". The scale ranges from 0.00 to 2.00, with major gridlines and numerical markers at intervals of 0.25 (0.00, 0.25, 0.50, 0.75, 1.00, 1.25, 1.50, 1.75, 2.00). * **Legend:** Positioned in the top-right corner of the chart area. It contains two entries: * A dark red line labeled "Original". * A light pink line labeled "Smoothed". * **Grid:** A light gray grid is present, with both horizontal and vertical lines corresponding to the axis markers. ### Detailed Analysis **1. "Original" Data Series (Dark Red Line):** * **Trend Verification:** This line exhibits a steep initial decline followed by a gradual, noisy descent. It shows significant high-frequency fluctuations throughout. * **Data Points (Approximate):** * Epoch 0.00: Loss ≈ 1.00 * Epoch 0.25: Loss ≈ 0.65 * Epoch 0.50: Loss ≈ 0.45 * Epoch 0.75: Loss ≈ 0.40 * Epoch 1.00: Loss ≈ 0.35 * Epoch 1.25: Loss ≈ 0.35 * Epoch 1.50: Loss ≈ 0.35 * Epoch 1.75: Loss ≈ 0.33 * Epoch 2.00: Loss ≈ 0.35 * **Spatial Grounding:** The line starts at the top-left of the plotted data area and moves towards the bottom-right, remaining consistently below the "Smoothed" line after the initial point. **2. "Smoothed" Data Series (Light Pink Line):** * **Trend Verification:** This line shows a smooth, monotonic decrease, curving gently downward from left to right without any visible fluctuations. * **Data Points (Approximate):** * Epoch 0.00: Loss ≈ 1.00 (coincides with the start of the "Original" line). * Epoch 0.50: Loss ≈ 0.70 * Epoch 1.00: Loss ≈ 0.50 * Epoch 1.50: Loss ≈ 0.40 * Epoch 2.00: Loss ≈ 0.35 * **Spatial Grounding:** This line originates at the same top-left point as the "Original" line but follows a higher, smoother path, converging with the "Original" line near the end of the plotted range at Epoch 2.00. ### Key Observations 1. **Convergence:** Both loss curves show a clear downward trend, indicating the model is learning and its performance is improving (loss is decreasing) as training progresses over the epochs. 2. **Smoothing Effect:** The "Smoothed" line effectively filters out the high-frequency noise present in the "Original" loss signal, revealing the underlying, steady trend of improvement. 3. **Rate of Learning:** The most significant reduction in loss occurs in the first quarter of the training period shown (Epoch 0.00 to 0.50). The rate of decrease slows considerably after Epoch 1.00, suggesting the model is approaching a plateau or convergence point. 4. **Final Value:** By Epoch 2.00, both the original and smoothed loss values converge to approximately 0.35. ### Interpretation This chart is a standard diagnostic tool for monitoring the training of a neural network, specifically the Vicuna-7B-v1.5-Chat large language model. The "Loss" metric quantifies the error between the model's predictions and the target outputs; a decreasing trend is the desired outcome. The relationship between the two lines demonstrates the utility of loss smoothing. The raw "Original" loss is volatile, which can make it difficult to discern the true training trajectory from batch-to-batch variance. The "Smoothed" line provides a clearer, more interpretable view of the model's consistent progress. The steep initial drop suggests effective early learning, while the flattening curve indicates diminishing returns as training continues, a common phenomenon as models fine-tune their parameters. The convergence of both lines at a low value (~0.35) by Epoch 2.00 suggests the training process for this segment was stable and successful in reducing error. </details> Figure 8: Fine-tuning loss plots for Vicuna-7B-v1.5-Chat. | Parameter Base Model ROUGE-L | Value lmsys/vicuna-7b-v1.5 0.676 | | --- | --- | | ROUGE-1 | 0.700 | | Fine-Tuning Type | LoRA | | LoRA Alpha | 16 | | LoRA Rank | 8 | | Cutoff Length | 4096 | | Gradient Accumulation Steps | 8 | | Learning Rate | 5.0e-05 | | LR Scheduler Type | Cosine | | Number of Training Epochs | 2.0 | | Optimizer | AdamW | | Quantization Bit | 4 | Table 9: Fine-tuning Parameters for Vicuna-7B-v1.5-Chat <details> <summary>x9.png Details</summary>  ### Visual Description ## Line Chart: Training Loss Curve for Zephyr-7B-Beta-Chat ### Overview The image displays a line chart titled "Zephyr-7B-Beta-Chat," plotting the training loss of a machine learning model over a series of training epochs. The chart contains two data series: the raw, fluctuating loss values ("Original") and a smoothed version of the same data ("Smoothed"). The overall trend shows a rapid initial decrease in loss, followed by a gradual convergence towards a stable, low value. ### Components/Axes * **Chart Title:** "Zephyr-7B-Beta-Chat" (centered at the top). * **Y-Axis (Vertical):** * **Label:** "Loss" (rotated 90 degrees). * **Scale:** Linear scale from 0.00 to 2.00. * **Tick Marks:** Major ticks at intervals of 0.25 (0.00, 0.25, 0.50, 0.75, 1.00, 1.25, 1.50, 1.75, 2.00). * **X-Axis (Horizontal):** * **Label:** "Epoch". * **Scale:** Linear scale from 0.00 to 2.00. * **Tick Marks:** Major ticks at intervals of 0.25 (0.00, 0.25, 0.50, 0.75, 1.00, 1.25, 1.50, 1.75, 2.00). * **Legend:** * **Position:** Top-right corner of the chart area. * **Entries:** 1. **"Original"**: Represented by a darker green line. 2. **"Smoothed"**: Represented by a lighter, pale green line. * **Grid:** A light gray grid is present, aligning with the major tick marks on both axes. ### Detailed Analysis **Data Series & Trends:** 1. **"Original" Line (Darker Green):** * **Trend:** Starts at a high loss value, experiences a very steep decline within the first ~0.1 epochs, then transitions to a much slower, noisy descent that gradually flattens out. * **Approximate Data Points:** * Epoch 0.00: Loss ≈ 0.85 * Epoch 0.10: Loss ≈ 0.55 (sharp drop) * Epoch 0.25: Loss ≈ 0.45 * Epoch 0.50: Loss ≈ 0.35 * Epoch 1.00: Loss ≈ 0.28 * Epoch 1.50: Loss ≈ 0.27 * Epoch 2.00: Loss ≈ 0.26 * **Character:** The line exhibits small, frequent fluctuations (noise) throughout its descent, which is typical of raw training loss metrics. 2. **"Smoothed" Line (Lighter Green):** * **Trend:** Starts at the same point as the Original line but follows a smooth, monotonic, and convex decay curve. It consistently lies above the Original line for most of the training process before converging with it near the end. * **Approximate Data Points:** * Epoch 0.00: Loss ≈ 0.85 * Epoch 0.25: Loss ≈ 0.65 * Epoch 0.50: Loss ≈ 0.50 * Epoch 1.00: Loss ≈ 0.35 * Epoch 1.50: Loss ≈ 0.28 * Epoch 2.00: Loss ≈ 0.26 * **Character:** This line represents a moving average or similar smoothing function applied to the "Original" data, filtering out high-frequency noise to reveal the underlying trend. **Spatial Relationship:** The "Smoothed" line acts as a central tendency guide for the "Original" line. The "Original" line oscillates around the "Smoothed" line, with the amplitude of oscillation decreasing as training progresses and the loss stabilizes. ### Key Observations 1. **Rapid Initial Learning:** The most significant reduction in loss (from ~0.85 to ~0.55) occurs within the first 10% of the displayed training period (Epoch 0.00 to 0.10). 2. **Convergence:** Both lines converge to a very similar value (≈0.26) by Epoch 2.00, indicating the training process has likely reached a plateau or a point of diminishing returns. 3. **Noise Reduction:** The "Smoothed" line successfully clarifies the overall trajectory, making it easier to see that the rate of improvement slows considerably after approximately Epoch 0.50. 4. **No Overfitting Signal:** The chart only shows training loss. There is no validation loss curve present to assess for potential overfitting. ### Interpretation This chart is a standard diagnostic tool for monitoring the training of a neural network, specifically the "Zephyr-7B-Beta-Chat" model. The loss function measures the discrepancy between the model's predictions and the target outputs; a decreasing loss indicates the model is learning from the training data. The data suggests a successful and stable training run. The steep initial drop is characteristic of the model quickly learning the most obvious patterns in the data. The subsequent slower decline represents the model fine-tuning its parameters to capture more nuanced relationships. The convergence of both the raw and smoothed loss to a low, stable value by epoch 2.00 implies that further training beyond this point, with the current setup, may yield minimal improvement in training loss. The presence of the smoothed line is crucial for interpretation, as it prevents the inherent noise in the raw loss from obscuring the clear, downward trend, confirming that the model's optimization process is functioning as intended. </details> Figure 9: Fine-tuning loss plots for Zephyr-7B-Beta-Chat. | Parameter Base Model ROUGE-L | Value HuggingFaceH4/zephyr-7b-beta 0.686 | | --- | --- | | ROUGE-1 | 0.712 | | Fine-Tuning Type | LoRA | | LoRA Alpha | 16 | | LoRA Rank | 8 | | Cutoff Length | 4096 | | Gradient Accumulation Steps | 8 | | Learning Rate | 5.0e-05 | | LR Scheduler Type | Cosine | | Number of Training Epochs | 2.0 | | Optimizer | AdamW | | Quantization Bit | 4 | Table 10: Fine-tuning Parameters for Zephyr-7B-Beta-Chat <details> <summary>x10.png Details</summary>  ### Visual Description ## Line Chart: Gemma-7B-IT-Chat Training Loss ### Overview The image is a line chart titled "Gemma-7B-IT-Chat" that plots the training loss of a machine learning model over a series of training epochs. It displays two data series: the raw, fluctuating "Original" loss and a "Smoothed" version of the same data. The chart demonstrates a classic learning curve where loss decreases rapidly at the start of training and then stabilizes. ### Components/Axes * **Title:** "Gemma-7B-IT-Chat" (centered at the top). * **Y-Axis:** Labeled "Loss". The scale runs from 0.00 to 2.00, with major gridlines and labels at intervals of 0.25 (0.00, 0.25, 0.50, 0.75, 1.00, 1.25, 1.50, 1.75, 2.00). * **X-Axis:** Labeled "Epoch". The scale runs from 0.00 to 2.00, with major gridlines and labels at intervals of 0.25 (0.00, 0.25, 0.50, 0.75, 1.00, 1.25, 1.50, 1.75, 2.00). * **Legend:** Located in the top-right corner of the plot area. It contains two entries: * A dark brown line labeled "Original". * A light beige/tan line labeled "Smoothed". * **Grid:** A light gray grid is present, aligning with the major ticks on both axes. ### Detailed Analysis **1. "Original" Data Series (Dark Brown Line):** * **Trend:** The line exhibits a very steep, near-vertical drop at the beginning of training, followed by a rapid deceleration in the rate of decrease. After approximately epoch 0.25, the line enters a phase of gradual decline with noticeable high-frequency fluctuations (noise). From around epoch 1.00 onward, the loss stabilizes, oscillating within a narrow band. * **Approximate Data Points (with uncertainty):** * Epoch 0.00: Loss ≈ 1.55 * Epoch 0.05: Loss ≈ 0.65 (sharp drop) * Epoch 0.25: Loss ≈ 0.45 * Epoch 0.50: Loss ≈ 0.40 * Epoch 1.00: Loss ≈ 0.30 * Epoch 1.50: Loss ≈ 0.28 * Epoch 2.00: Loss ≈ 0.30 (slight uptick at the very end) **2. "Smoothed" Data Series (Light Beige Line):** * **Trend:** This line represents a smoothed version of the "Original" data, likely using a moving average or similar filter. It shows a smooth, convex curve that decreases monotonically. The rate of decrease is fastest initially and slows over time, asymptotically approaching a stable value. * **Approximate Data Points (with uncertainty):** * Epoch 0.00: Loss ≈ 1.55 (coincides with Original start) * Epoch 0.25: Loss ≈ 0.90 * Epoch 0.50: Loss ≈ 0.60 * Epoch 0.75: Loss ≈ 0.45 * Epoch 1.00: Loss ≈ 0.35 * Epoch 1.50: Loss ≈ 0.28 * Epoch 2.00: Loss ≈ 0.27 **Spatial Relationship:** The "Smoothed" line consistently lies above the "Original" line during the initial rapid descent phase (epochs 0.00 to ~0.75), as the smoothing algorithm lags behind the sharp drop. The two lines converge and become nearly indistinguishable from approximately epoch 1.25 onward, both hovering around a loss value of 0.25-0.30. ### Key Observations 1. **Rapid Initial Learning:** The most significant reduction in loss occurs within the first 0.25 epochs, indicating the model quickly learns the most prominent patterns in the training data. 2. **Convergence and Stabilization:** Both lines show clear convergence. The loss stabilizes after epoch 1.00, suggesting the model has reached a plateau in its learning for this training run. 3. **Noise in Raw Signal:** The "Original" line contains significant high-frequency noise, which is typical of stochastic gradient descent-based training. The "Smoothed" line effectively removes this noise to reveal the underlying trend. 4. **Final Value:** The training loss stabilizes at a value of approximately 0.28 ± 0.02. ### Interpretation This chart visualizes the training dynamics of the "Gemma-7B-IT-Chat" model. The steep initial drop signifies effective early-stage learning. The subsequent plateau indicates that further training beyond 1.5-2.0 epochs, with the current setup (data, learning rate, etc.), may yield diminishing returns, as the model's performance on the training set (as measured by loss) is no longer improving significantly. The presence of noise in the "Original" signal is not an anomaly but a characteristic of the training process. The "Smoothed" line is crucial for interpreting the true learning trajectory, filtering out the batch-to-batch variance. The convergence of both lines at a low, stable loss value suggests the training process was stable and the model successfully fit the training data to the limit of its capacity or the provided optimization parameters. To assess if this loss value is "good," one would need to compare it to validation loss metrics, which are not shown here. </details> Figure 10: Fine-tuning loss plots for Gemma-7B-IT-Chat. | Parameter Base Model ROUGE-L | Value google/gemma-7b-it 0.678 | | --- | --- | | ROUGE-1 | 0.703 | | Fine-Tuning Type | LoRA | | LoRA Alpha | 16 | | LoRA Rank | 8 | | Cutoff Length | 4096 | | Gradient Accumulation Steps | 8 | | Learning Rate | 5.0e-05 | | LR Scheduler Type | Cosine | | Number of Training Epochs | 2.0 | | Optimizer | AdamW | | Quantization Bit | 4 | Table 11: Fine-tuning Parameters for Gemma-7B-IT-Chat <details> <summary>x11.png Details</summary>  ### Visual Description ## Line Chart: Gemma-2B-IT-Chat Training Loss ### Overview This image is a line chart titled "Gemma-2B-IT-Chat," displaying the training loss of a machine learning model over a period of 2 epochs. The chart plots two data series: the raw, fluctuating loss values ("Original") and a smoothed version of the same data ("Smoothed"). ### Components/Axes * **Title:** "Gemma-2B-IT-Chat" (centered at the top). * **Y-Axis:** Labeled "Loss." The scale runs from 0.00 to 2.00, with major gridlines and labels at intervals of 0.25 (0.00, 0.25, 0.50, 0.75, 1.00, 1.25, 1.50, 1.75, 2.00). * **X-Axis:** Labeled "Epoch." The scale runs from 0.00 to 2.00, with major gridlines and labels at intervals of 0.25 (0.00, 0.25, 0.50, 0.75, 1.00, 1.25, 1.50, 1.75, 2.00). * **Legend:** Located in the top-right corner of the plot area. It contains two entries: * A dark blue line labeled "Original." * A light blue line labeled "Smoothed." * **Grid:** A light gray grid is present, aligning with the major ticks on both axes. ### Detailed Analysis **Data Series & Trends:** 1. **Original (Dark Blue Line):** * **Trend:** Shows a steep, rapid descent initially, followed by a gradual, noisy decline that eventually plateaus. * **Key Data Points (Approximate):** * Epoch 0.00: Loss ≈ 1.58 (starting point). * Epoch 0.10: Loss ≈ 0.80 (sharp drop). * Epoch 0.25: Loss ≈ 0.60. * Epoch 0.50: Loss ≈ 0.50. * Epoch 1.00: Loss ≈ 0.40. * Epoch 1.50: Loss ≈ 0.38. * Epoch 2.00: Loss ≈ 0.42 (slight uptick at the very end). * **Behavior:** The line exhibits significant high-frequency noise or volatility throughout, with many small peaks and valleys, especially visible after epoch 0.5. 2. **Smoothed (Light Blue Line):** * **Trend:** Shows a smooth, continuous, and monotonic decrease, representing the underlying trend of the "Original" data without the noise. * **Key Data Points (Approximate):** * Epoch 0.00: Loss ≈ 1.58 (coincides with Original). * Epoch 0.25: Loss ≈ 1.05. * Epoch 0.50: Loss ≈ 0.70. * Epoch 0.75: Loss ≈ 0.55. * Epoch 1.00: Loss ≈ 0.48. * Epoch 1.50: Loss ≈ 0.40. * Epoch 2.00: Loss ≈ 0.38. * **Behavior:** This line acts as a trendline, clearly illustrating the exponential-decay-like pattern of the loss function. It converges with the general level of the "Original" line after epoch 1.0. ### Key Observations * **Rapid Initial Convergence:** The most dramatic reduction in loss occurs within the first 0.25 epochs, where the loss drops by over 60% from its starting value. * **Noise Profile:** The "Original" signal is consistently noisy, indicating high variance in the per-step or per-batch loss calculations during training. * **Plateau Phase:** After approximately epoch 1.0, the rate of loss reduction slows significantly. The loss value stabilizes in the range of 0.35 to 0.45 for the remainder of the charted training. * **Final Uptick:** There is a minor but noticeable increase in the "Original" loss at the final data point (epoch 2.00), which could be statistical noise or the beginning of overfitting, though more data would be needed to confirm. ### Interpretation This chart visualizes the training dynamics of the "Gemma-2B-IT-Chat" model. The steep initial descent indicates that the model is learning very effectively at the start of training, quickly adjusting its parameters to reduce prediction error. The subsequent plateau suggests the model is approaching a local minimum in the loss landscape, where further improvements become incremental. The presence of the "Smoothed" line is crucial for interpretation. It filters out the stochastic noise inherent in mini-batch gradient descent, revealing the true, underlying learning trajectory. The fact that the smoothed loss continues a gentle downward trend even in the plateau phase (from ~0.48 at epoch 1.0 to ~0.38 at epoch 2.0) suggests that training is still beneficial, albeit at a much slower pace. The final uptick in the raw loss warrants monitoring in subsequent epochs to ensure it is not a sign of training instability or overfitting. Overall, the chart demonstrates a healthy and typical loss curve for a well-configured training run. </details> Figure 11: Fine-tuning loss plots for Gemma-2B-IT-Chat. | Parameter Base Model ROUGE-L | Value google/gemma-2b-it 0.667 | | --- | --- | | ROUGE-1 | 0.692 | | Fine-Tuning Type | LoRA | | LoRA Alpha | 16 | | LoRA Rank | 8 | | Cutoff Length | 4096 | | Gradient Accumulation Steps | 8 | | Learning Rate | 5.0e-05 | | LR Scheduler Type | Cosine | | Number of Training Epochs | 2.0 | | Optimizer | AdamW | | Quantization Bit | 4 | Table 12: Fine-tuning Parameters for Gemma-2B-IT-Chat **Text Span Attribution Verification** **Objective:** Predict whether the text span is "Attributable", "Contradictory", or "Extrapolatory" based on the information provided in the triplets. **Instructions:** 1. **Read the Full Text:** - Understand the context and content of the full text string. 2. **Examine the Text Span:** - Determine the claims made within the text span. 3. **Analyze the Triplets:** - Evaluate if the triplets support, refute, or neither support nor refute the claims in the text span. 4. **Make Your Prediction:** - Classify the text span as "Attributable", "Contradictory", or "Extrapolatory" based on your analysis of the triplets. 5. **Provide Rationale:** - Clearly explain your reasoning for the classification. **Classification Criteria:** - **"Attributable"**: The text span is sufficiently supported by the triplet(s). All claims in the text span are directly present in the triplet information. - **"Contradictory"**: The text span is conclusively refuted by the triplet(s). All claims in the text span are directly contradicted by the triplet information. - **"Extrapolatory"**: The triplet(s) can neither support nor refute the text span. The information provided is either irrelevant, indirect, or related but not sufficient to support or refute the text span. **Example:** **Full Text:** "Albert Einstein is widely recognized as the father of modern physics. He was awarded the Nobel Prize in Physics for his services to Theoretical Physics." **Text Span:** "He was awarded the Nobel Prize in Physics." **Triplets:** [("Albert Einstein", "award received", "Nobel Prize in Physics")] **Sample Evaluation:** - **Prediction:** "Attributable" - **Rationale:** "The triplet directly supports the claim that Albert Einstein received the Nobel Prize in Physics." **Example:** **Full Text:** "Isaac Newton discovered the element radium." **Text Span:** "Isaac Newton discovered radium." **Triplets:** [("Marie Curie", "discovered", "radium")] **Sample Evaluation:** - **Prediction:** "Contradictory" - **Rationale:** "The triplet states that Marie Curie discovered radium, contradicting the claim that Isaac Newton discovered it." **Example:** **Full Text:** "The Eiffel Tower is a wrought-iron lattice tower that was opened in 1889." **Text Span:** "The Eiffel Tower is a wrought-iron lattice tower that was opened in 1889." **Triplets:** [("Eiffel Tower", "located in", "Paris")] **Sample Evaluation:** - **Prediction:** "Extrapolatory" - **Rationale:** "The triplet states that the Eiffel Tower is located in Paris, which is related but not sufficient to confirm or refute that it was opened in 1889." **Verification Checklist:** - [ ] The prediction accurately reflects the relationship between the text span and the triplets. - [ ] The rationale clearly explains the classification based on the triplets. - [ ] The explanation is free from irrelevant information. **Response Format:** Provide your evaluation in the following JSON format: - "prediction": "Attributable", "Contradictory", or "Extrapolatory" - "rationale": "Your comments here" **Inputs to Evaluate** **Full text:** "{full_text}" **Text span:** "{text_span}" **Triplets:** {triplets} Figure 12: Prompt for the text span attribution verification task, guiding the model to classify text spans as "Attributable," "Contradictory," or "Extrapolatory" based on the provided triplets. The prompt design incorporates concepts such as few-shot learning, chain-of-thought reasoning Kojima et al. (2022), and tailored prompt engineering OpenAI (2024); Nori et al. (2023)