# : Citation Verification with AI-Powered Full-Text Analysis and Evidence-Based Reasoning
**Authors**:
- Sebastian Haan (The University of Sydney, Australia)
## Abstract
Effective scientific communication depends on accurate citations that validate sources and guide readers to supporting evidence. Yet academic literature faces mounting challenges: semantic citation errors that misrepresent sources, AI-generated hallucinated references, and traditional citation formats that point to entire papers without indicating which sections substantiate specific claims. We introduce SemanticCite, an AI-powered system that verifies citation accuracy through full-text source analysis while providing rich contextual information via detailed reasoning and relevant text snippets. Our approach combines multiple retrieval methods with a four-class classification system (Supported, Partially Supported, Unsupported, Uncertain) that captures nuanced claim-source relationships and enables appropriate remedial actions for different error types. Our experiments show that fine-tuned lightweight language models achieve performance comparable to large commercial systems with significantly lower computational requirements, making large-scale citation verification practically feasible. The system provides transparent, evidence-based explanations that support user understanding and trust. We contribute a comprehensive dataset of over 1,000 citations with detailed alignments, functional classifications, semantic annotations, and bibliometric metadata across eight disciplines, alongside fine-tuned models and the complete verification framework as open-source software. SemanticCite addresses critical challenges in research integrity through scalable citation verification, streamlined peer review, and quality control for AI-generated content, providing an open-source foundation for maintaining citation accuracy at scale.
footnotetext: Email: sebastian.haan@sydney.edu.au footnotetext: Software Repository: https://github.com/sebhaan/semanticcite
<details>
<summary>x2.png Details</summary>
### Visual Description
Icon/Small Image (24x23)
</details>
Repository
<details>
<summary>hf_icon.png Details</summary>
### Visual Description
## Icon/Symbol: Yellow Emoji Character with Hands Covering Mouth
### Overview
The image is a digital illustration of a stylized, yellow, circular emoji-like character. The character is depicted with a joyful or surprised expression, with both hands raised to cover its open mouth. The style is simple, cartoonish, and uses flat colors with minimal shading. The image is presented on a plain white background with a subtle white border or sticker effect around the character.
### Components/Visual Elements
* **Primary Subject:** A yellow, circular face/head.
* **Facial Features:**
* **Eyes:** Two closed, curved black lines, suggesting a happy or laughing expression (like `^ ^`).
* **Mouth:** A wide, open, black semi-circle, indicating a big smile or gasp. Inside the mouth, a pink/red tongue is visible.
* **Blush:** Two small, orange, circular marks on the cheeks, positioned below and slightly outside the eyes.
* **Hands:** Two yellow hands, matching the face color, are placed in front of the lower half of the face. Each hand has four visible fingers, rendered as simple, rounded shapes. The palms face inward, covering the mouth area.
* **Color Palette:**
* **Primary Yellow:** The dominant color for the face and hands. (Approximate hex: `#FFCC00` or similar bright yellow).
* **Black:** Used for the eye lines and mouth outline.
* **Pink/Red:** Used for the tongue inside the mouth.
* **Orange:** Used for the blush circles.
* **White:** Used for the background and the sticker-like border around the character.
* **Style & Composition:** The character is centered in the frame. The lines are clean and bold. The overall composition is symmetrical and focused entirely on the character's expressive reaction.
### Detailed Analysis
* **Expression:** The combination of closed, upturned eyes and a wide-open mouth covered by hands strongly conveys an emotion of **overwhelming joy, excited surprise, or shy delight**. The gesture of covering the mouth is often associated with stifling a laugh, gasping in pleasant surprise, or expressing bashfulness.
* **Design Language:** The design uses universal, non-verbal cues (emoji style) to communicate emotion without text. The simplicity ensures immediate recognizability and emotional impact.
### Key Observations
1. **Emotional Focus:** Every design element (eyes, mouth, hands, blush) is coordinated to amplify a single, positive emotional state.
2. **Gesture as Key Identifier:** The hands covering the mouth are the most distinctive feature, differentiating this from a standard smiling or laughing emoji.
3. **Sticker Aesthetic:** The white border suggests this image is designed to be used as a digital sticker, a common format in messaging apps.
### Interpretation
This image is a **non-verbal emotional symbol**. It is designed to be inserted into digital communication to convey a complex feelingâlikely a mix of happiness, surprise, and modestyâmore efficiently and with more warmth than plain text. The character's design leverages universal facial expressions and a culturally recognized gesture (covering the mouth when laughing or surprised) to ensure its meaning is widely understood. Its purpose is to add emotional nuance and personality to a conversation, acting as a visual shorthand for a reaction that might otherwise require a full sentence to describe. The sticker format indicates it is meant for informal, personal communication.
</details>
Model & Data
Keywords: citation verification, semantic analysis, natural language processing, research integrity, automated fact-checking
## 1 Introduction
### 1.1 Problem Statement
Citations serve as fundamental channels through which scientific knowledge is shared and developed within the academic ecosystem. Despite their critical importance, scholarly literature suffers from a notable prevalence of citation errors that compromise research integrity and quality.
These errors manifest in several forms: citations supporting claims not made in referenced works, citations omitting critical qualifications present in sources, citations attributing findings to incorrect studies, and selective citation of results whilst ignoring contradictory evidence. Beyond these semantic errors, typographical mistakes in bibliographic information also occur, though digital citation management tools have reduced their frequency. Analysis of prestigious science journals reveals that semantic citation errors remain pervasive, with error rates of 25% on average [smith2020quotation].
The implications for research quality are profound. Incorrect citations inadvertently propagate erroneous ideas, contribute to the acceptance of scientific myths, and ultimately erode the credibility of academic scholarship [gotzsche2022citation]. Such inaccuracies can lead to unintended plagiarism, diminished academic standing, and misallocation of valuable research resources.
Manual citation verification during peer review faces substantial limitations. Reviewers are increasingly burdened by overwhelming workloads due to rapid expansion in academic publishing, making it difficult to dedicate sufficient time to meticulous citation evaluation. The lack of standardised review criteria and training contributes to inconsistent evaluations, whilst personal biases can subtly influence outcomes [smith2006peer, tennant2020limitations, mishra2025challenges, sizo2025defining].
A pressing new challenge has emerged with AI-generated content, where advanced language models produce convincing but non-existent âfake citationsâ that are extraordinarily difficult to detect manually at scale. Recent studies demonstrate that even advanced models like GPT-5 fabricate citations 39% of the time when operating without internet access, highlighting the fundamental nature of this problem [gibney2025hallucination]. These hallucinations arise from the statistical nature of large language models, which make predictions by generalising from learned associations, leading to plausible but sometimes incorrect outputs [gibney2025hallucination]. As AI-assisted writing becomes prevalent in academic contexts, traditional verification approaches designed for human authors prove insufficient for detecting AI-generated content that can produce highly plausible but entirely fabricated citations at scale.
Traditional citation presentation provides no mechanism for readers to quickly understand cited work context without manually retrieving and reading entire source documents. Citations merely point to entire papers without indicating specific supporting sections, forcing readers to interrupt their flow, locate sources, search through lengthy documents for relevant passages, and return to the original text. This lack of precise contextual guidance impedes efficient knowledge discovery and creates barriers to thorough fact-checking during both research and peer review processes.
Current automated citation tools remain severely limited across multiple dimensions. Reference management software (EndNote, Zotero, Mendeley) excel at bibliographic formatting but provide no semantic verification capabilities. Existing automated verification systems validate citations against authority databases (Web of Science, PubMed, CrossRef), detecting incorrect metadata or non-existent references, but cannot verify whether citation claims accurately reflect source document content. A reference may exist with correct metadata whilst the citing author mischaracterises the sourceâs actual findings. Similarly, format-checking systems verify style compliance but do not analyse semantic alignment between claims and sources.
Even systems attempting content-level verification typically rely on abstract-only analysis rather than examining complete source documents. This approach misses critical supporting evidence, methodological caveats, limitations discussions, and contradictory findings present in full papers. Important contextual information such as experimental conditions, population constraints, statistical significance details, and author-acknowledged limitations rarely appears in abstracts yet proves essential for accurate citation verification.
Furthermore, traditional verification approaches employ simple binary schemes (supported/unsupported) that fail to capture partial support, where core assertions align with sources but contextual elements are missing. These situations require different remedial actions than completely unsupported claims, yet binary systems cannot distinguish between them. Additionally, automated systems provide classifications without explaining reasoning or presenting supporting evidence, preventing independent validation.
These converging challenges create an urgent need for comprehensive citation verification systems. Pervasive semantic errors, limited verification tools, opaque automated systems, AI-generated hallucinations, and manual verification friction demand solutions that detect errors, explain reasoning, provide actionable guidance, and scale efficiently whilst maintaining the transparency essential for scholarly work.
### 1.2 Research Objectives
This work addresses these fundamental limitations through four primary objectives. First, we develop an automated full-text citation verification system that transcends surface-level bibliographic checks to perform deep semantic analysis of citation content against complete source documents. Second, we create a nuanced 4-class classification scheme (Supported, Partially Supported, Unsupported, Uncertain) that captures the complexity of citation-reference relationships beyond traditional binary approaches, enabling appropriate remedial actions for different types of citation errors. Third, we demonstrate that fine-tuned lightweight models can achieve competitive performance with large commercial language models whilst requiring significantly reduced computational resources. Fourth, we implement transparent evidence-based reasoning that enables users to understand and independently validate system decisions through relevant text snippets extracted from source documents and detailed explanations of classification rationale. This transparency is essential for maintaining the verifiability and trust required in academic contexts, addressing the opacity that has plagued previous automated verification systems.
### 1.3 Contributions
This paper introduces SemanticCite, the first comprehensive AI-powered framework for automated full-text citation verification. Unlike existing approaches that rely on abstract-level analysis or binary classification schemes, our system performs deep semantic analysis of complete source documents using a 4-class taxonomy that captures the nuanced relationships between citations and their references. SemanticCite bridges the gap between automated detection and actionable remediation by providing detailed evidence-based reasoning that enables authors and reviewers to understand and address citation inaccuracies systematically. This represents a fundamental advance from simple error detection to comprehensive citation quality assessment with practical guidance for manuscript improvement. By combining hybrid retrieval and reranking techniques with fine-tuned language models, SemanticCite enables thorough citation analysis while maintaining computational efficiency, making sophisticated verification accessible to individual researchers and institutions regardless of computational resources.
We contribute three key resources spanning data, models, and implementation. Our dataset comprises 1,111 analysed citations across 8 academic disciplines, including extracted claims, 4-class semantic classifications, reasoning explanations, confidence scores, and ranked evidence snippets. We release fine-tuned Qwen3 models (1.7B and 4B parameters) trained for both citation preprocessing and semantic classification tasks that achieve competitive performance with commercial systems while requiring significantly fewer computational resources. We open-source the SemanticCite framework code, including claim extraction pipelines, hybrid retrieval systems, and classification method.
## 2 Related Work
### 2.1 Citation Analysis and Verification
Traditional citation analysis has primarily focused on bibliometric approaches using citation counts as indicators of research impact, failing to capture semantic relationships between citing statements and referenced sources. The field of automated fact-checking has provided important foundations for citation verification. Wadden et al. (2020) [wadden2020fact] introduced scientific claim verification with their SciFact dataset comprising 1.4K expert-written claims paired with abstracts annotated for veracity, demonstrating that domain adaptation significantly improves performance versus general-purpose fact-checking models.
However, existing systems face significant limitations. Most rely on abstract-level analysis rather than full-text examination, potentially missing crucial evidence within complete documents. Binary classification schemes (SUPPORTS/REFUTES) fail to capture nuanced citation-reference relationships [wadden2020fact]. The emergence of AI-generated content has introduced new challenges, with advanced language models producing convincing but non-existent citations. Whilst tools like CiteSure [citesure2024] address this specific problem and systems like Morressierâs Integrity Manager detect various misconduct indicators, these specialised tools focus on specific error types rather than comprehensive citation verification [sae2024research].
Some existing platforms analyse citation usage within citing documents, extracting surrounding context and classifying whether the citing statement supports or contradicts the reference generally [nicholson2021scite]. However, these systems examine the directionality in reverse, i.e., they assess whether the citation supports the reference rather than whether the source document supports the specific citation claim. They cannot verify if the referenced source actually contains the evidence the citing author asserts, leaving the fundamental question of claim accuracy unaddressed.
### 2.2 Multi-class Citation Classification in NLP
The evolution from binary to multi-class classification reflects growing recognition that real-world relationships exist on spectra rather than simple dichotomies. Multi-class approaches offer significant advantages by capturing degrees of alignment rather than forcing binary decisions. Systems employing classifications such as âsupported,â âpartially supported,â âunsupported,â and âuncertainâ provide granular assessments that better reflect scientific discourse complexity [haan2025developing].
The Jurgens et al. (2018) taxonomy [jurgens2018measuring] represents a notable contribution, defining six categories (Background, Extends, Uses, Motivation, Compare/Contrast, Future work) for citation intent classification using the ACL-ARC dataset of 1,941 expert-annotated instances. Subsequent research expanded available data with the SciCite dataset, over five times larger than ACL-ARC, simplifying taxonomy to three categories (Background, Method, Result) whilst achieving superior performance with pre-trained language models like XLNet (88.93% F1 score) [mercier2020impactcite].
The diversity of taxonomies across datasets indicates lack of universal agreement on citation function definitions. This fragmentation, coupled with recognition that citations may serve multiple purposes, suggests current single-label classification may oversimplify relationships, necessitating flexible frameworks supporting customisable taxonomies and multi-label classification [abu-jbara2013, lauscher2021multicite].
### 2.3 Retrieval-Augmented Generation
Retrieval-augmented generation (RAG) has emerged as a powerful paradigm for fact-checking applications, combining information retrieval with language model generation capabilities [lewis2020retrieval, gao2023retrieval]. In citation verification contexts, RAG enables systems to retrieve relevant evidence from source documents and generate informed assessments of citation accuracy. Traditional RAG approaches face limitations when applied to academic texts due to their complexity and specialised terminology [li2024budget].
Recent advances in reverse RAG methodologies show promise for citation verification by first extracting or summarising information, then linking each data point back to source documents through systematic fact verification. This approach ensures every claim is traceable to original sources, addressing hallucination problems in standard generative models [rajpurkar2024claimify, metropolitansky2025towards].
Hybrid retrieval approaches combining dense semantic search with sparse keyword matching have demonstrated superior performance over individual methods [arivazhagan2023, sawarkar2024blended, sultania2024domain]. Dense retrieval using vector embeddings captures semantic relationships, whilst sparse methods like BM25 (Robertson & Zaragoza, 2009) ensure exact term correspondence. Neural reranking using cross-encoders further optimises candidate ordering based on query-document relevance, providing robust foundations for evidence-based citation verification systems.
## 3 Methodology
### 3.1 System Architecture
SemanticCite employs a four-stage pipeline that combines hybrid retrieval-augmented generation with large language models for automated citation verification. The system operates through sequential stages: text preprocessing, hybrid retrieval, neural reranking, and LLM-based analysis, implemented using the LangChain framework with support for multiple providers. Figure 1 provides an overview of the complete citation verification pipeline, illustrating the flow from input processing through to structured output generation.
<details>
<summary>pipeline_diagram.png Details</summary>
### Visual Description
## Diagram: Semantic Citation Pipeline
### Overview
The image is a technical flowchart titled "Semantic Citation Pipeline." It illustrates a multi-stage process for verifying citations against reference documents using a combination of traditional and AI-powered methods. The pipeline flows from left to right and top to bottom, with distinct, color-coded stages connected by directional arrows indicating the sequence of operations.
### Components
The diagram is composed of seven primary, color-coded rectangular boxes representing stages, connected by dark gray arrows. Each stage contains one or more sub-components.
**1. INPUT (Top-Left, Blue Box)**
* **Citation Text** (Solid border box)
* **Reference Document** (Solid border box)
* **Metadata** (Dashed border box, labeled "optional")
**2. PROCESSING (Top-Center, Orange Box)**
* **Document Processing**
* PDF extraction
* Text chunking
* **Citation Processing**
* Remove references
* Extract core claim
**3. EMBEDDING (Top-Right, Red Box)**
* **Vector Store**
* Embeddings
**4. HYBRID RETRIEVAL (Center-Left, Light Blue Box)**
* **Dense Retrieval**
* Vector similarity
* **Sparse Retrieval**
* BM25 keyword
**5. RERANKING (Center-Right, Purple Box)**
* **Neural Reranking**
* FlashRank
**6. LLM ANALYSIS (Bottom-Left, Green Box)**
* **Citation Verification**
* Citation vs Evidence
* Metadata context
* Structured output
**7. OUTPUT (Bottom-Right, Gray Box)**
* **Classification** (Green sub-box)
* 4 categories + confidence
* **Evidence** (Blue sub-box)
* Top 3 chunks + scores
* **Reasoning** (Orange sub-box)
* Detailed explanation
* **Metadata** (Gray sub-box)
* Timestamps + metrics
### Detailed Analysis: Flow and Relationships
The pipeline's data flow is explicitly defined by the arrows:
1. The **INPUT** stage (Citation Text, Reference Document, optional Metadata) feeds directly into the **PROCESSING** stage.
2. **PROCESSING** splits into two parallel paths:
* The processed document data flows to the **EMBEDDING** stage.
* The processed citation data flows down to the **HYBRID RETRIEVAL** stage.
3. The **EMBEDDING** stage (Vector Store) also feeds its output into the **HYBRID RETRIEVAL** stage.
4. The **HYBRID RETRIEVAL** stage, combining Dense and Sparse methods, sends its results to the **RERANKING** stage.
5. The **RERANKING** stage outputs to the **LLM ANALYSIS** stage.
6. Finally, the **LLM ANALYSIS** stage produces the structured **OUTPUT**.
### Key Observations
* **Hybrid Approach:** The core retrieval mechanism is explicitly hybrid, combining semantic "Dense Retrieval" (vector similarity) with lexical "Sparse Retrieval" (BM25 keyword matching).
* **Two-Stage Ranking:** Retrieval is followed by a dedicated "Neural Reranking" step using FlashRank, indicating a focus on precision after initial candidate selection.
* **LLM as Verifier:** The Large Language Model (LLM) is positioned not as a retriever but as an analyst that performs "Citation Verification" using the retrieved evidence and context.
* **Structured, Multi-faceted Output:** The final output is not a simple yes/no but a rich, structured object containing a classification, the top evidence chunks with scores, a reasoning explanation, and operational metadata.
* **Visual Coding:** Each major stage has a unique color (blue, orange, red, light blue, purple, green, gray), aiding in visual segmentation of the process.
### Interpretation
This diagram outlines a sophisticated system designed to move beyond simple keyword matching for citation verification. It represents a **Retrieval-Augmented Generation (RAG) pipeline specialized for academic or factual citation checking**.
The process begins by ingesting a citation and its purported source document. It then processes both into a searchable format. The use of a **hybrid retrieval** strategy (dense + sparse) suggests the system aims to balance semantic understanding with precise keyword matching to find the most relevant text chunks from the reference document.
The subsequent **neural reranking** step is critical for filtering and ordering these chunks by relevance before they are passed to the LLM. This prevents the LLM from being overwhelmed with irrelevant context and improves the accuracy of the final analysis.
The **LLM's role** is specifically defined as "Citation Verification." It compares the core claim of the citation against the retrieved evidence, considers any provided metadata, and generates a structured verdict. The final **OUTPUT** is designed for both human consumption (via the "Reasoning" explanation) and programmatic use (via the "Classification," "Evidence" scores, and "Metadata").
In essence, this pipeline automates the scholarly task of checking whether a citation accurately reflects the content of its source document, providing a transparent, auditable, and multi-faceted result. It emphasizes accuracy through a multi-stage filtering process and delivers a comprehensive report rather than a binary answer.
</details>
Figure 1: Semantic Citation Verification Pipeline: A multi-stage automated system for citation verification combining document processing (PDF extraction and text chunking), vector embedding storage, hybrid retrieval using both dense semantic similarity and sparse BM25 keyword matching, neural reranking with FlashRank, and LLM-based analysis. The pipeline outputs a classification result, supporting evidence, detailed reasoning, and confidence score for each citation verification task.
The citation verification system takes three inputs: citation text, reference document, and optional metadata. Then it produces a classification label, confidence score, supporting evidence, and detailed reasoning. The system processes reference documents through text extraction and chunking, whilst citations undergo preprocessing to remove attribution markers and extract core factual claims.
Document processing begins with text extraction using PyMuPDF PyMuPDF Documentation: https://github.com/pymupdf/PyMuPDF for PDF files, followed by recursive text splitting into 512-character chunks with 50-character overlap. The chunking strategy employs hierarchical separators including paragraph breaks, sentences, punctuation, and whitespace to maintain semantic coherence whilst ensuring manageable sizes for retrieval operations.
Citation preprocessing transforms raw citations into standardised claims by removing reference markers, author attributions, and publication identifiers whilst preserving numerical values and factual assertions. This transformation converts author-centric statements to fact-centric assertions suitable for independent verification. For example, âSmith et al. (2020) found that exercise reduces cardiovascular risk by 30%â becomes âExercise reduces cardiovascular risk by 30%â.
The system supports flexible configuration of LLM providers including OpenAI, Anthropic Claude, Google Gemini, and local models via LiteLLM LiteLLM Documentation: https://docs.litellm.ai/docs/. Similarly, embedding models can be configured across OpenAI, local SentenceTransformers, or custom endpoints, enabling deployment across diverse computational environments whilst maintaining consistent performance. Temperature settings are controlled for reproducible outputs, with deterministic citation processing using zero temperature.
### 3.2 Classification Scheme
Our 4-class taxonomy represents a fundamental contribution to citation verification methodology, providing nuanced assessment beyond traditional binary approaches. The classification scheme follows a logical progression from fully supported to unsupported citations, with an uncertainty class capturing ambiguous cases where clear determination cannot be made. Figure 2 illustrates the complete classification framework with actionable guidance for each category.
The development of this taxonomy addresses critical limitations in existing binary classification systems. Traditional SUPPORTS/REFUTES schemes, whilst computationally simpler, fail to capture the complexity of citation-reference relationships commonly encountered in academic literature. Many citations exist in intermediate states requiring different remedial actions, making binary classification insufficient for practical citation verification applications.
<details>
<summary>classification_scheme_diagram.png Details</summary>
### Visual Description
## Diagram: Classification Scheme for Claim-Source Alignment
### Overview
The image displays a structured classification scheme titled "Classification Scheme" for evaluating the alignment between a claim and its source material. It presents four distinct categories of alignment, each with defined indicators and recommended actions. The diagram uses a color-coded, columnar layout to organize this information.
### Components/Axes
The diagram is organized into four vertical columns, each representing a classification category. Each column consists of two main boxes:
1. **Top Box (Category & Indicators):** Contains the category name, a parenthetical alignment descriptor, and a list of indicators.
2. **Bottom Box (Actions):** Contains a list of recommended actions for that category.
A downward-pointing arrow connects the top and bottom boxes within each column.
**Categories (from left to right):**
1. **SUPPORTED (Fully Aligned)** - Green color scheme.
2. **PARTIALLY SUPPORTED (Partially Aligned)** - Yellow/Orange color scheme.
3. **UNSUPPORTED (Misaligned)** - Red color scheme.
4. **UNCERTAIN (Indeterminate)** - Gray color scheme.
### Detailed Analysis
| Category | Indicators | Actions |
| :--- | :--- | :--- |
| **SUPPORTED (Fully Aligned)** | ⢠Complete alignment<br>⢠Proper contextual usage<br>⢠Comprehensive representation | ⢠No changes needed<br>⢠Consider adding context<br>⢠Citation is accurate |
| **PARTIALLY SUPPORTED (Partially Aligned)** | ⢠Core claim supported<br>⢠Missing nuances/context<br>⢠Simplified representation | ⢠Review evidence snippets<br>⢠Revise to include key details<br>⢠Clarify/expand citation |
| **UNSUPPORTED (Misaligned)** | ⢠Claim absent from source<br>⢠Contradiction with source<br>⢠Misrepresentation | ⢠Remove/replace claim<br>⢠Search alternative references<br>⢠Review reasoning provided |
| **UNCERTAIN (Indeterminate)** | ⢠Ambiguous content<br>⢠Insufficient information<br>⢠Inconclusive relationship | ⢠Review evidence carefully<br>⢠Revise for clarity<br>⢠Consider removing citation |
### Key Observations
* The scheme progresses from positive alignment (green/SUPPORTED) to negative alignment (red/UNSUPPORTED), with intermediate (yellow/PARTIALLY SUPPORTED) and ambiguous (gray/UNCERTAIN) states.
* The "Actions" for each category are directly informed by its "Indicators." For example, a "SUPPORTED" claim requires no changes, while an "UNSUPPORTED" one may need removal or replacement.
* The "UNCERTAIN" category is distinct from "PARTIALLY SUPPORTED"; it implies a lack of clear evidence for or against the claim, rather than partial evidence.
### Interpretation
This diagram provides a systematic framework for assessing the fidelity of a written claim to its source material. It moves beyond a simple true/false binary by introducing granularity for partial support and uncertainty.
* **Purpose:** It is likely used in fields like fact-checking, academic research, legal review, or content moderation to standardize the evaluation of evidence and guide editorial decisions.
* **Relationships:** The categories are mutually exclusive and collectively exhaustive for the purpose of this evaluation model. The flow from Indicators to Actions creates a clear decision-making pathway: diagnose the alignment issue, then apply the prescribed remedy.
* **Underlying Logic:** The scheme emphasizes not just the presence or absence of support, but also the *quality* of representation (e.g., "comprehensive" vs. "simplified," "proper contextual usage" vs. "missing nuances"). The "UNCERTAIN" category acknowledges the real-world problem of ambiguous or insufficient data, preventing forced classifications. The recommended actions are pragmatic, focusing on improving the accuracy and clarity of the final document.
</details>
Figure 2: Four-Category Classification Scheme for Source-Claim Alignment Assessment
#### 3.2.1 Detailed Classification Categories
SUPPORTED (Fully Aligned): Citations demonstrate complete alignment between claims and source material, maintaining proper context and comprehensive representation. These citations accurately convey the referenced content without oversimplification, missing context, or distortion of meaning. The core indicators include complete factual alignment, appropriate contextual usage, and comprehensive representation of source nuances. For instance, a citation stating âThe human genome contains approximately 20,000 protein-coding genesâ fully aligns with source material stating âRecent genomic studies have confirmed that the human genome encodes about 20,000 protein-coding genes.â Such citations receive high confidence scores and require no modifications, though users may optionally add additional context from provided evidence snippets.
PARTIALLY SUPPORTED (Partially Aligned): Core claims are present in source material but lack critical nuances, contextual information, or important qualifications, leading to incomplete representation. These citations contain accurate fundamental assertions but oversimplify complex findings or omit essential caveats present in the source. Key indicators include core claim support with missing minor nuances, simplified representation, or incomplete contextual information. An example would be citing âExercise guarantees a 50% reduction in cardiovascular disease riskâ when the source states âRegular exercise has been shown to reduce cardiovascular disease risk by up to 50% in some studies, though results can vary.â The citation captures the core finding but overstates certainty and omits variability. Such citations receive medium confidence scores and require minor revisions to include qualifying details and expand scope for complete accuracy.
UNSUPPORTED (Misaligned): Claims are absent from, contradicted by, or significantly misrepresent source material. These citations demonstrate fundamental disconnection between stated claims and referenced content, ranging from complete absence of supporting evidence to direct contradiction of source findings. Critical indicators include claims absent from source material, direct contradiction between claims and evidence, or significant misrepresentation of source content. For example, claiming âDrinking coffee cures cancerâ when the source states âSome studies suggest compounds in coffee may lower risk of certain cancers, but evidence is not conclusiveâ represents serious misrepresentation. Such citations receive low confidence scores and require major revision, removal, or alternative reference identification.
UNCERTAIN (Indeterminate): Ambiguous relationships exist where insufficient evidence, unclear claim-reference relationships, or inadequate verification context prevent definitive classification. These cases often involve vague claims, complex source material with multiple interpretations, or insufficient contextual information for reliable assessment. Key indicators include ambiguous claim or source content, insufficient contextual information, or inconclusive claim-reference relationships. An example might be citing âVitamin D supplementation prevents all cases of osteoporosisâ when the source discusses âvarying degrees of effectiveness depending on factors such as age, dosage, and baseline deficiency levels.â The broad generalisation in the claim combined with nuanced source findings creates ambiguity requiring additional clarification. Such citations warrant careful review of evidence snippets to identify ambiguity sources.
#### 3.2.2 Actionable User Guidance Framework
The classification scheme provides specific, actionable guidance aligned with potential user responses:
- SUPPORTED citations receive confirmation that no changes are needed, with optional suggestions for additional context enhancement.
- PARTIALLY SUPPORTED citations receive specific guidance on reviewing evidence snippets to identify missing nuances and revising content to include qualifying details.
- UNSUPPORTED citations prompt users to remove inaccurate claims, replace them with evidence-based statements, or seek alternative supporting references.
- UNCERTAIN classifications guide users to carefully review evidence snippets, revise citations to include necessary contextual details, or consider citation removal if relationships remain unclear.
This structured guidance framework transforms abstract classification labels into concrete editorial actions, bridging the gap between automated analysis and practical manuscript improvement. The approach recognises that different classification outcomes require different remedial strategies, moving beyond simple accuracy scores to provide constructive improvement pathways.
#### 3.2.3 Advantages Over Binary Classification
The multi-class approach offers several critical advantages over binary schemes. Binary classification forces complex citation-reference relationships into oversimplified dichotomies, failing to distinguish between minor revisions and major corrections. Our taxonomy captures nuanced relationships requiring different remedial actions, providing more precise feedback for authors and reviewers.
Furthermore, the inclusion of an UNCERTAIN category acknowledges inherent limitations in automated analysis, providing transparent handling of ambiguous cases rather than forcing potentially incorrect binary decisions. This approach enhances system trustworthiness by explicitly flagging cases requiring human judgment whilst maintaining high confidence in clear-cut classifications.
The taxonomy also aligns with real-world editorial workflows where citations often require incremental improvements rather than complete acceptance or rejection. By providing graduated classification levels, the system supports iterative manuscript refinement and targeted quality improvement efforts.
### 3.3 Hybrid Retrieval System
The retrieval architecture combines dense semantic search with sparse keyword matching to maximise evidence recall whilst maintaining precision. This hybrid approach addresses individual method limitations: dense retrieval captures semantic relationships but may miss exact term matches, whilst sparse retrieval ensures keyword correspondence but lacks semantic understanding.
Dense Retrieval employs vector embeddings stored in ChromaDB using cosine similarity matching. Document chunks are mapped into high-dimensional vector spaces where semantically similar content clusters together. When processing citation claims, the system calculates similarity scores between query vectors and stored document vectors, retrieving the 15 most semantically relevant chunks. This approach excels at capturing conceptual relationships and paraphrased content that might use different terminology whilst conveying similar meaning.
Sparse Retrieval utilises BM25 algorithm for exact term matching, ensuring that citations requiring specific keyword correspondence are properly identified. BM25 considers term frequency, document length normalisation, and inverse document frequency to rank passages based on lexical overlap with citation claims. This method retrieves an additional 15 chunks focusing on exact term matches, complementing semantic retrieval by capturing precise terminology and specific numerical values.
Neural Reranking applies FlashRank cross-encoder optimisation to the combined retrieval results. After deduplication, the hybrid results undergo reranking based on query-document relevance scores computed by BERT-based cross-encoders. This process optimises candidate ordering and selects the top 3 most relevant passages for downstream analysis, ensuring focus on the most pertinent evidence whilst maintaining computational efficiency.
Document Chunking Strategy employs recursive character splitting with carefully chosen parameters to balance granularity with context preservation. The 512-character chunk size with 50-character overlap ensures sufficient context for semantic understanding whilst maintaining computational tractability. Hierarchical separators prioritise natural text boundaries, starting with paragraph breaks and progressively falling back to sentences, punctuation, and finally individual words when necessary.
The hybrid approach demonstrates superior performance compared to individual methods by leveraging complementary strengths. Dense retrieval captures nuanced semantic relationships crucial for academic content, whilst sparse retrieval ensures critical exact matches are not missed. Neural reranking provides final optimisation, resulting in high-quality evidence selection for citation verification.
### 3.4 Citation Processing Pipeline
The processing pipeline transforms raw academic citations into verifiable claims whilst preserving essential factual content. This preprocessing stage is critical for accurate verification as raw citations often contain attribution elements that obscure core factual assertions.
Preprocessing Operations systematically transform citations through several key steps. Reference markers such as brackets, parentheses, and citation numbers are removed to isolate factual content from bibliographic formatting. Author attributions are eliminated whilst preserving substantive claims, converting statements like âSmith et al. found that Xâ to âX was foundâ using passive voice constructions. Numerical values, statistical measures, and specific metrics are carefully preserved as these often represent the most verifiable aspects of citations. The process ensures all claims become independently verifiable statements that can be assessed against source material without requiring external attribution context.
Structured Output Parsing employs LLM-based analysis with explicit schema requirements to ensure consistent and comprehensive results. The system generates structured JSON outputs containing classification labels, detailed reasoning with both summary and point-by-point analysis, confidence scores, and ranked evidence snippets with relevance scores and source locations. This structured approach enables programmatic processing whilst maintaining human interpretability.
The verification prompt carefully incorporates processed claims, top-ranked evidence snippets with relevance scores, and optional metadata for enhanced contextual understanding. Prompts explicitly instruct models to assess factual alignment between citations and source material whilst considering broader study context when metadata is available. Instructions emphasise objective evaluation based on textual evidence rather than general knowledge or assumptions.
Evidence Snippet Ranking applies systematic relevance scoring to prioritise the most pertinent supporting material. Each evidence snippet receives numerical relevance scores indicating how well it addresses the citation claim, enabling users to focus on the most important supporting or contradicting evidence. The system tracks comprehensive metadata including processing timestamps, reference file identifiers, chunk locations, and system version information for complete result provenance and reproducibility analysis.
This systematic approach enables transparent verification where users understand classification rationale through extracted evidence and detailed reasoning, supporting both automated analysis and human review processes. The pipelineâs modular design allows for component-level optimisation and adaptation to different document types and verification requirements whilst maintaining consistent output quality.
### 3.5 Web Interface and Deployment
<details>
<summary>User_Interface.png Details</summary>
### Visual Description
## [Screenshot]: SemanticCite Web Application Interface
### Overview
This image is a screenshot of a web application interface for a tool named "SemanticCite." The application is designed for AI-powered semantic citation verification, allowing users to analyze whether citations in their text are accurately supported by provided reference documents. The interface is divided into two primary sections: a left-hand sidebar for configuration settings and a main content area for the core citation verification workflow.
### Components/Axes
The interface is structured into two main vertical panels.
**Left Sidebar (Configuration Panel):**
* **Header:** "Model Configurations"
* **Section 1: LLM Configuration**
* Label: "LLM Provider" | Dropdown Value: "local"
* **Section 2: Dual Model Configuration**
* Label: "Preprocessing Model" | Input Value: "ollama/SemanticCite-Refiner-Qwen3-1B"
* Label: "Classification Model" | Input Value: "ollama/SemanticCite-Checker-Qwen3-4B"
* Label: "Model endpoint URL" | Input Value: "http://localhost:11434"
* **Section 3: Embedding Configuration**
* Label: "Embedding Provider" | Dropdown Value: "local"
* Label: "Embedding Model Name" | Input Value: "Qwen/Qwen3-Embedding-0.6B"
* **Section 4: Disclaimers**
* A highlighted box contains the text: "Privacy: All processing occurs locally"
* Below the box: "AI-assisted analysis - verify results independently"
**Main Content Area (Workflow Panel):**
* **Header/Logo:** A shield icon with "AI" and the text "SemanticCite".
* **Title:** "AI-powered semantic citation verification"
* **Subtitle:** "Analyse whether your citations are accurately supported by reference documents."
* **Section 1: Citation Text**
* Label: "Enter citation text"
* A large, empty text input area.
* **Section 2: Reference Document**
* Two tabs: "Upload File" (selected) and "Download from URL".
* Label: "Upload reference document"
* A drag-and-drop file upload zone with the text: "Drag and drop file here" and "Limit 200MB per file ⢠TXT, PDF, MD". A "Browse files" button is on the right.
* **Section 3: Reference Metadata (Optional)**
* Label: "Enter reference metadata"
* A text input area with placeholder text: "Optional: Enter title, abstract, and other reference metadata... Supports plain text and markdown formatting."
* **Action Button:** A large blue button at the bottom labeled "Check Citation" with a magnifying glass icon.
### Detailed Analysis
The interface is a form-based application. The user is expected to:
1. Configure the underlying AI models (LLM and Embedding) in the left sidebar, which are pre-filled with specific model names and a local endpoint.
2. Input the citation text they want to verify in the "Citation Text" field.
3. Provide a reference document by either uploading a file (TXT, PDF, MD, up to 200MB) or downloading from a URL.
4. Optionally, provide additional metadata about the reference document.
5. Click the "Check Citation" button to initiate the analysis.
All configuration fields in the sidebar have a help icon (a question mark in a circle) next to their labels. The main content area's input fields also have these help icons.
### Key Observations
* **Local Processing Emphasis:** The configuration defaults to "local" for both LLM and Embedding providers, and the endpoint URL points to `localhost:11434`. This is reinforced by the prominent privacy disclaimer stating "All processing occurs locally."
* **Specific Model Stack:** The application is configured to use a specific suite of models from the "Qwen3" family, hosted via "ollama": a Refiner model (1B parameters), a Checker model (4B parameters), and an Embedding model (0.6B parameters).
* **Dual-Model Architecture:** The "Dual Model Configuration" suggests a pipeline where one model preprocesses or refines the input, and another performs the core classification or checking task.
* **User Guidance:** The interface includes placeholder text and disclaimers to guide the user, such as the note to "verify results independently" and the metadata field's explanation of supported formatting.
### Interpretation
This interface represents a specialized tool for academic or research writing integrity. Its purpose is to automate the laborious task of cross-referencing citations against source material to detect inaccuracies or misrepresentations.
The design prioritizes transparency and control. By exposing the specific model names and the local endpoint, it caters to technical users who may want to understand or modify the underlying AI stack. The strong emphasis on local processing addresses potential privacy concerns, assuring users that sensitive documents and citations are not sent to external servers.
The workflow is linear and logical: configure models, provide inputs (citation + reference), and execute the check. The optional metadata field suggests the tool might use this additional context to improve verification accuracy. The "Check Citation" button is the clear call-to-action, positioned as the final step in the process.
Overall, SemanticCite appears to be a focused application leveraging local, specialized AI models to provide a privacy-conscious solution for verifying the fidelity of scholarly citations.
</details>
Figure 3: SemanticCite web interface showing citation input, reference document upload options (file upload or URL download), optional metadata entry, and configurable model parameters. The interface supports multiple LLM providers and embedding models, enabling flexible deployment across different resource constraints and institutional requirements.
To facilitate widespread adoption and ensure accessibility for researchers without programming expertise, we developed a web-based interface using Streamlit (Figure 3). The interface provides intuitive controls for citation verification, allowing users to input citation text, upload reference documents (PDF, TXT, or Markdown formats), or download references directly from URLs. Users can configure both LLM providers (OpenAI, Anthropic Claude, Google Gemini, or local Ollama models) and embedding models (local SentenceTransformers, OpenAI embeddings, or custom endpoints) through an interactive sidebar, enabling flexible deployment across different institutional resource constraints and privacy requirements. The system displays results including classification labels with color coding, confidence scores, detailed reasoning explanations, and ranked evidence snippets extracted from the reference document. All analysis results can be exported to Markdown format for documentation and integration into research workflows. This implementation demonstrates the practical deployment of the SemanticCite framework whilst maintaining full analytical capabilities and transparency in the verification process.
## 4 Model Development and Evaluation
This section details the development and evaluation of fine-tuned large language models for automated citation verification within the SemanticCite framework. We present our complete experimental pipeline for training specialized citation analysis models, beginning with the systematic construction of a balanced, multi-disciplinary dataset spanning eight academic fields (§4.1). We describe our automated training data generation methodology that leverages LLM-based filtering and annotation to create high-quality datasets for both citation preprocessing and classification tasks (§4.2). The fine-tuning process employs QLoRA techniques with Qwen3 models [qwen2024] of varying scales to develop efficient, specialized systems for citation verification (§4.3). We introduce a weighted evaluation framework specifically designed to assess the ordinal nature of citation support categories (§4.4), and conclude with comprehensive experimental results demonstrating model performance across different scales and computational requirements (§4.5). This experimental framework establishes the effectiveness of fine-tuned language models for citation verification while providing insights into the trade-offs between model complexity, computational efficiency, and task-specific performance in academic text analysis.
### 4.1 Data Selection
We constructed a high-quality training dataset using a multi-stage approach designed to ensure representative coverage across academic disciplines while maintaining rigorous quality standards for bibliometric metadata and full-text content. The resulting corpus targets balanced representation across citation impact levels, disciplinary boundaries, and functional citation categories, providing a robust empirical foundation for training and evaluating citation analysis models. An overview of the data selection process is presented in Figure 4.
<details>
<summary>data_selection_diagram.png Details</summary>
### Visual Description
## Flowchart: Research Paper Processing and Citation Analysis Workflow
### Overview
The image displays a vertical flowchart outlining a four-stage systematic process for acquiring, processing, and categorizing academic papers and their citations to create a balanced training dataset. The workflow progresses from top to bottom, with one stage splitting into two parallel processes before converging again.
### Components/Axes
The diagram consists of four primary rectangular stages connected by downward-pointing arrows, indicating sequential flow. The third stage is split into two parallel, side-by-side boxes.
**Stage 1: Stratified Paper Selection & Metadata Acquisition**
- **Process:** Semantic Scholar Graph with quality filters, citation impact stratification, and metadata collection.
- **Academic Fields Listed:** Computer Science ⢠Medicine ⢠Chemistry ⢠Biology ⢠Materials Science ⢠Physics ⢠Geology ⢠Psychology
**Stage 2: Full-Text Acquisition**
- **Process:** Systematic PDF download with integrity verification and full-text extraction.
**Parallel Processes (Following Stage 2):**
1. **Functional Citation Classification**
- **Method:** LLM-based functional categorisation.
- **Citation Function Categories (in boxes):** BACKGROUND, METHOD, COMPARISON, MOTIVATION, EXTENSION, FUTURE.
2. **Citation Selection**
- **Method:** Two-stage filtering and prioritisation.
- **Selection Criteria (bullet points):**
- Single-reference contexts
- Factual assertions
- Quantitative claims
- High verification potential
**Stage 3: Reference Document Acquisition**
- **Process:** Acquisition of cited reference documents for verification.
- **Formats Processed:** Full-text ⢠Abstracts ⢠Multi-format processing
**Final Output: Balanced Training Dataset**
This final box contains four key metrics presented in individual sub-boxes:
- **1,111** Citations Analysed
- **8** Academic Fields
- **6** Citation Types
- **27%** Numerical Citations
- **Additional Notes:** Equal distribution across citation impact levels. Comprehensive metadata, citation claims, alignment classification, reasoning, and reference text.
### Detailed Analysis
The workflow is a linear pipeline with a bifurcation for specialized processing. The initial stage casts a wide net across eight major scientific disciplines. The core processing involves two complementary actions on the acquired full-texts: classifying the *function* of citations (e.g., to provide background or propose future work) and *selecting* citations based on their verifiability (prioritizing factual and quantitative claims). The process concludes by acquiring the referenced documents themselves and compiling all information into a structured dataset characterized by specific quantitative metrics (1,111 citations, 27% numerical) and qualitative balance (across fields and impact levels).
### Key Observations
1. **Multi-Disciplinary Scope:** The process explicitly targets eight distinct academic fields, ensuring broad coverage.
2. **LLM Integration:** Large Language Models are specified as the tool for the functional classification task.
3. **Verification Focus:** The "Citation Selection" criteria heavily emphasize verifiable content (factual assertions, quantitative claims).
4. **Dataset Composition:** The final dataset is defined by precise numbers: 1,111 citations from 8 fields, categorized into 6 types, with a notable 27% being numerical citations.
5. **Parallel Processing:** The workflow intelligently separates the *analysis* of citation function from the *selection* of citations for verification, allowing these tasks to proceed concurrently after full-text acquisition.
### Interpretation
This flowchart details a rigorous, automated methodology for constructing a high-quality, verifiable citation analysis dataset. The process is designed to move from broad acquisition to highly specific selection and classification.
The **Peircean investigative reading** suggests this is a system built for **abductive reasoning**âit doesn't just collect data but actively selects elements (factual/quantitative claims) that are most suitable for generating and testing hypotheses about scientific discourse. The emphasis on "verification potential" and the acquisition of reference documents indicates the ultimate goal is likely to train or evaluate models on their ability to fact-check or validate scientific claims against source material.
The **balanced nature** of the final dataset (across fields and impact levels) is critical for ensuring any model trained on it generalizes well and doesn't inherit biases toward specific disciplines or citation popularity. The 27% figure for numerical citations highlights a targeted focus on quantitative scientific claims, which are often central to verification tasks. This workflow represents a sophisticated pipeline for creating ground-truth data to advance research in automated scientific claim verification, citation analysis, and scholarly document understanding.
</details>
Figure 4: Overview of the data selection and processing pipeline for semantic citation verification
#### 4.1.1 Data Collection and Processing Pipeline
The training dataset was constructed through a systematic three-stage pipeline designed to collect academic papers with diverse citation characteristics across multiple disciplines, targeting a corpus of 4,000 papers with balanced representation across citation impact levels.
Stage 1: Stratified Paper Selection
The collection process employed a stratified sampling approach across eight academic fields (Computer Science, Medicine, Chemistry, Biology, Materials Science, Physics, Geology, and Psychology) using the Semantic Scholar Graph API. This selection ensures comprehensive coverage of distinct disciplinary citation practices and scholarly communication patterns. Papers were filtered according to stringent criteria including publication year (2019-2023), minimum citation count ( $⼠5$ ), minimum reference count ( $⼠10$ ), and open access availability. However, open-access designation did not guarantee that all papers were actually available for download. The temporal constraint ensures currency whilst providing sufficient time for citation accumulation, and the citation thresholds guarantee papers with meaningful scholarly impact and comprehensive reference networks.
To ensure balanced representation across citation impact levels, papers were categorised into three citation rate bins (Low, Mid, High) based on field-specific empirically-derived thresholds that account for citations-per-year rates rather than absolute counts. This normalisation approach addresses the substantial variation in citation velocity across academic disciplines. These thresholds varied considerably across disciplines, reflecting domain-specific citation practicesâfor instance, Medicine exhibited substantially higher thresholds (Mid: 14, High: 31 citations/year) compared to Geology (Mid: 4, High: 7 citations/year). This stratification ensures that the dataset captures papers across the full spectrum of scholarly impact within each discipline, avoiding bias towards highly-cited outliers whilst maintaining representation of influential work.
Stage 2: Metadata Acquisition
For each selected paper, extensive metadata was retrieved including bibliographic information, abstracts, author details, venue information, external identifiers, and complete citation and reference networks. This metadata collection provides essential context for understanding citation patterns and enables sophisticated analysis of citation behaviour across different paper characteristics. Quality assurance measures excluded papers lacking valid open access PDF URLs or failing PDF format validation, ensuring accessibility of full-text content essential for detailed citation analysis.
Stage 3: Full-Text Document Acquisition
The final stage involved systematic download of PDF documents using validated open access URLs. Each download was verified for successful completion and proper PDF format before inclusion in the dataset, with verification including file size validation, PDF structure integrity checks, and preliminary text extraction testing. This methodology yielded a balanced, high-quality corpus of approximately 500 papers per academic field, with equal distribution across citation impact categories. The approach ensures representative coverage across academic disciplines and citation patterns whilst maintaining access to both structured bibliometric data and full-text content, providing a robust foundation for training citation analysis models.
#### 4.1.2 Citation Function Classification Methodology
Classification Framework
The citation classification system employs a systematic approach to categorise citation functions within academic texts, building upon the established taxonomy of Jurgens et al. (2018) [jurgens2018measuring]. The methodology implements a six-category schema designed to capture the primary functional roles of citations in scholarly discourse. This approach balances thorough coverage of citation purposes with tractable computational requirements for automated classification.
- BACKGROUND citations establish the general field, provide contextual information, or reference foundational knowledge essential for understanding the current work.
- METHOD citations reference specific algorithms, techniques, datasets, software, or experimental procedures employed or adapted in the current work, representing instrumental citations that directly inform methodological choices.
- RESULT_COMPARISON citations directly compare or contrast findings, results, or conclusions with those of the cited work, often appearing in results and discussion sections where authors evaluate their findings against existing evidence.
- MOTIVATION citations indicate that the cited work directly inspired the research problem or provided primary motivation for the current study, representing foundational influences on research direction.
- EXTENSION citations signal that the current work builds upon, extends, improves, or adapts methods or ideas from the cited work, indicating direct intellectual lineage.
- FUTURE_WORK citations mention prior work specifically in the context of suggested research directions or open problems, typically appearing in concluding sections where authors outline potential developments.
Dual-Dimensional Classification Approach
The classification system operates along two complementary dimensions to capture both functional and quantitative aspects of citation usage. The primary dimension assigns each citation context to one of the six functional categories through systematic analysis of the surrounding textual context, considering both immediate sentence-level context and broader paragraph-level discourse patterns. The secondary dimension employs binary classification to identify whether the citation context explicitly involves numerical or quantitative informationâincluding percentages, specific values, metrics, performance scores, or statistical dataâdirectly related to the claim or comparison involving the cited work. This dual-dimensional approach recognises that quantitative citations often require different verification strategies compared to qualitative references, as numerical claims demand precise fact-checking against source documents whilst qualitative citations may require more nuanced semantic analysis. The quantitative dimension proves particularly valuable for identifying citations that make specific empirical claims requiring direct verification against source data.
Computational Implementation
The classification process uses large language models (LLMs) with structured prompting to ensure consistency and reproducibility across the entire dataset. Citation contexts are processed in configurable batches to optimise computational efficiency whilst maintaining classification accuracy. Each classification decision is accompanied by a concise rationale explaining the categorical assignment, providing transparency and enabling quality assessment of the automated classification process. The methodology incorporates error handling and validation mechanisms, including verification of output format compliance, categorical validity checks, and fallback procedures for problematic cases.
#### 4.1.3 Citation Selection Methodology
To generate high-quality training data for fine-tuning large language models on citation analysis tasks, we implemented a two-stage automated selection pipeline using LLM-based filtering and ranking designed to identify citation instances most suitable for verification analysis.
The first stage applied strict filtering criteria to identify valid citation instances, requiring that: (1) the context contained only a single reference rather than multiple citations, ensuring unambiguous attribution and avoiding complications from citation clustering; (2) the context represented an actual citation sentence rather than a bibliography entry, maintaining focus on in-text scholarly discourse; and (3) the citation text was properly embedded within the contextual passage, ensuring sufficient surrounding context for meaningful analysis.
From the filtered candidates, a second LLM stage selected the top 5 citations per paper that warranted verification analysis. The selection prioritised citations containing specific claims, numerical statements, factual assertions, and substantial content that could be meaningfully verified against their source references, rather than brief mentions or general acknowledgments. This prioritisation ensures that the resulting training data focuses on citations with sufficient semantic content to support meaningful verification tasks. Files with fewer than 3 citations were excluded to maintain dataset quality and ensure sufficient citation diversity within each source document. This systematic approach ensured the selection of citation instances most suitable for training models to assess citation-reference alignment and factual accuracy.
### 4.2 Training Data Generation
Training datasets are generated for two distinct model types addressing different aspects of citation analysis. The Citation Preprocessing Model transforms raw citation statements into cleaned claims through systematic processing, creating input-output pairs mapping original citations to their standardised, processed forms. This preprocessing removes reference markers, author attributions, and publication identifiers whilst preserving numerical values and factual assertions, converting author-centric statements to fact-centric representations suitable for verification.
The Classification Model generates training examples for citation support classification through a more complex pipeline involving: (1) preparing vector stores from reference documents using hybrid retrieval methods, (2) extracting semantically relevant text chunks through embedding-based similarity matching using Qwen3-Embedding-0.6B, and (3) generating structured outputs containing the four classification labels (SUPPORTED, PARTIALLY SUPPORTED, UNSUPPORTED, UNCERTAIN), detailed reasoning, and confidence scores.
Ground truth annotations were generated using GPT-4.1 (gpt-4.1-2025-04-14) with carefully designed prompts ensuring consistent and reliable classifications. The citation preprocessing employed prompts instructing the model to remove reference markers, convert author-centred statements to fact-centred assertions, maintain numerical values, and ensure claims stand alone as verifiable statements.
For classification analysis, GPT-4 was provided with processed citations, reference document metadata, and relevant text snippets, then instructed to classify citations using our 4-class taxonomy. The structured output schema required classification labels, reasoning dictionaries containing summary and detailed analysis, and numerical confidence ratings (0.0-1.0).
### 4.3 Model Fine-tuning
The fine-tuning methodology employed QLoRA (Quantised Low-Rank Adaptation) techniques with Qwen3 language models to develop specialised systems for citation verification.
Model Architecture and Quantisation
The system used Qwen3 models in three configurations (1.7B, 4B, and 8B parameters) with 4-bit quantisation via BitsAndBytes to enable efficient training on consumer hardware. The QLoRA configuration employed a rank (r) of 16 with an alpha value of 32, targeting attention projection layers (q_proj, k_proj, v_proj, o_proj) and feed-forward network components (gate_proj, up_proj, down_proj). Gradient checkpointing through Unsloth optimisation reduced memory consumption whilst maintaining training stability, enabling deployment across diverse computational environments.
Training Data Preparation and Configuration
Three distinct training datasets were prepared for different citation analysis tasks. The Preprocessing Dataset contained 1,111 examples for text simplification and paraphrasing tasks, with input-output pairs averaging 49.2 and 38.8 tokens respectively. The Classification Dataset comprised 1,111 examples for citation verification tasks, featuring significantly longer sequences (average 810.9 input tokens, 178.5 output tokens) due to inclusion of reference snippets and metadata.
The training protocol implemented a conservative approach with 2 epochs, a learning rate of $2Ă 10^-4$ , and AdamW optimisation with 8-bit precision. Batch sizes were dynamically adjusted based on GPU memory (1-4 per device) with gradient accumulation steps (2-8) to maintain effective batch sizes. The system employed linear learning rate scheduling with 5% warmup ratio and 1% weight decay for regularisation.
Task-Specific Training Approaches
Text Preprocessing Models were trained with sequence lengths of 1024 tokens while the Citation Classification Models used an extended sequence lengths (4608 tokens). Structured Output Models generated JSON-formatted responses containing classification labels, detailed reasoning, and confidence scores (0.0-1.0), enabling programmatic integration and uncertainty quantification. Training data underwent systematic preprocessing whereby complex nested JSON structures were flattened into coherent text sequences, with task-specific instructions integrated using Alpaca formatting templates to create standardised prompt structures.
Models were exported in multiple formats including LoRA adapters, merged weights, and GGUF quantised versions for deployment flexibility. The system maintained reproducibility through fixed random seeds and deterministic training procedures, with data splitting following a 90:10 train-test partition using stratified sampling.
### 4.4 Evaluation Methodology
Our classification task involves predicting four categories with natural ordinal structure: SUPPORTED (3), PARTIALLY SUPPORTED (2), UNSUPPORTED (1), and UNCERTAIN (0). Unlike traditional multi-class evaluation that treats all misclassifications equally, we implement a weighted evaluation framework that penalizes errors proportionally to their semantic distance.
#### 4.4.1 Weighted Evaluation Framework
We define a weight matrix where misclassification penalty equals the absolute difference between ordinal values: $W(i,j)=|v(i)-v(j)|$ . This ensures that extreme errors (e.g., SUPPORTED $â$ UNCERTAIN, weight = 3) receive higher penalties than adjacent-level misclassifications (e.g., SUPPORTED $â$ PARTIALLY SUPPORTED, weight = 1).
Weighted accuracy is calculated as:
$$
ACC_weighted=1-\frac{â_i,jW(i,j)ĂCM(i,j)}{NĂ\max(W)} \tag{1}
$$
where CM $(i,j)$ represents confusion matrix entries, $N$ is total predictions, and $\max(W)=3$ . We also calculate ordinal Mean Absolute Error (MAE) treating classes as ordered variables, providing intuitive 0-1 scale performance assessment.
Our evaluation combines standard metrics (accuracy, F1-scores, Cohenâs $Îş$ ) with ordinal-aware measures (weighted accuracy, ordinal MAE) to capture both traditional classification performance and task-specific semantic alignment.
#### 4.4.2 Text Generation Quality Metrics
Beyond classification accuracy, we assess text generation quality using character-level similarity and length calibration metrics. Character similarity is calculated using Jaccard similarity at the character level, measuring the overlap of unique characters between predictions and ground truth texts. Specifically, we convert both texts to lowercase, remove spaces, create sets of unique characters, and calculate Jaccard similarity as the intersection divided by the union of character sets. This metric captures lexical overlap and vocabulary consistency regardless of word order, providing insight into how well models preserve semantic content during text generation. Length difference measures the deviation in word count between predictions and ground truth, indicating model calibration in output verbosity.
### 4.5 Results
#### 4.5.1 Model Performance Overview
We evaluated three fine-tuned Qwen3 models (1.7B, 4B, 8B parameters) using QLoRA techniques on a stratified test set of 112 examples. Models were assessed across both preprocessing and classification tasks, with preprocessing involving citation text cleaning and standardization, while classification represents the primary citation verification task.
#### 4.5.2 Preprocessing Task Performance
Table 1: Preprocessing Performance Results
| Qwen3 1.7B Qwen3 4B Qwen3 8B | 112 112 112 | 112 (100%) 112 (100%) 112 (100%) | 93.32% 94.15% 94.30% | 26.7 words 26.7 words 26.4 words | 26.3 words 26.3 words 26.3 words | +0.4 words +0.4 words +0.1 words |
| --- | --- | --- | --- | --- | --- | --- |
All models achieved exceptional performance on the preprocessing task, demonstrating 100% valid prediction rates with high character similarity ( $>93\$ ) and excellent length calibration. The preprocessing task involves cleaning citation text by removing reference markers, author attributions, and standardizing claims into fact-centric statements suitable for verification. The consistently high performance across all model sizes indicates that text preprocessing represents a simpler task compared to classification, where models can reliably learn pattern-based transformations. The 8B model achieved marginally superior character similarity (94.30%) and best length calibration (+0.1 words), though all models performed comparably, suggesting that even the 1.7B model is sufficient for preprocessing tasks in resource-constrained deployments.
#### 4.5.3 Classification Task Performance
Table 2: Classification Performance Results
| Qwen3 1.7B Qwen3 4B Qwen3 8B | 50.00% 66.36% 66.07% | 75.15% 83.64% 83.93% | 33.50% 49.15% 50.13% | 46.61% 65.31% 65.73% | 88.07% 90.01% 89.33% | +4.7 words -0.7 words +3.6 words |
| --- | --- | --- | --- | --- | --- | --- |
#### 4.5.4 Key Findings
1. Weighted vs. Standard Performance: The two evaluation approaches reveal different aspects of model performance. Standard accuracy shows clear scaling with model size (1.7B: 50.00%, 4B: 66.36%, 8B: 66.07%), while weighted accuracy demonstrates that all models achieve substantially higher performance when accounting for ordinal relationships (1.7B: 75.15%, 4B: 83.64%, 8B: 83.93%). This indicates that models predominantly make semantically reasonable ânear-missâ errors rather than extreme misclassifications.
2. Text Generation Quality and Calibration: The 4B model achieves superior text generation performance with the highest character similarity (90.01%) and best length calibration (-0.7 words from ground truth). The 8B model shows slightly lower character similarity (89.33%) and moderate length deviation (+3.6 words), while the 1.7B model exhibits the lowest character similarity (88.07%) and largest length deviation (+4.7 words). This suggests the 4B model achieves optimal balance between accuracy and text generation fidelity.
3. Model Scaling Effects: Performance scaling is non-linear across different metrics. While weighted accuracy scales consistently with model size, the 4B model outperforms the 8B model in standard accuracy, character similarity, and length calibration. All models maintain high character similarity ( $>88\$ ), indicating robust text generation capabilities across different scales.
4. Deployment Considerations: The 4B model emerges as the optimal choice, achieving near-best weighted accuracy (83.64% vs. 83.93% for 8B) while excelling in text generation quality and computational efficiency. The 1.7B model, despite lower absolute performance, maintains meaningful weighted accuracy (75.15%) and reasonable text generation quality (88.07% character similarity) suitable for resource-constrained applications.
#### 4.5.5 Implications
Comparison with base models reveals the critical importance of fine-tuning for this task: all base Qwen3 models demonstrated substantially lower accuracy, generated significantly longer outputs that deviated from expected length targets, and consistently failed to conform to required JSON output structures for structured classification tasks. The evaluation results demonstrate the complementary insights provided by standard and weighted evaluation approaches for ordinal classification tasks, while revealing important patterns in text generation quality across model scales. The 4B model emerges as optimally calibrated, achieving competitive weighted accuracy (83.64%) while excelling in character similarity (90.01%) and length precision (-0.7 words). This multi-dimensional superiority suggests that intermediate-scale models may achieve better overall performance than larger alternatives, possibly due to reduced over-parameterization effects. The 1.7B modelâs 75.15% weighted accuracy alongside 88.07% character similarity demonstrates that even lightweight models can achieve meaningful performance for resource-constrained deployments where understanding ordinal error patterns and maintaining text generation quality are crucial for practical utility.
## 5 Discussion
### 5.1 Novel Contributions
This work advances the field of automated citation verification through several key innovations. First, we introduce the first full-text citation verification system that moves beyond abstract-level analysis to examine complete source documents, addressing a critical limitation in existing approaches [wadden2020fact]. Our hybrid retrieval architecture, combining dense semantic search with sparse BM25 matching and neural reranking, demonstrates superior evidence selection compared to individual methods, particularly important for academic texts with specialised terminology.
The 4-class classification taxonomy provides nuanced assessment beyond traditional binary schemes. Unlike SciFactâs SUPPORTS/REFUTES framework [wadden2020fact], our taxonomy captures intermediate states requiring different remedial actions. The inclusion of confidence scoring and actionable user guidance transforms abstract classifications into concrete editorial recommendations, bridging automated analysis with human decision-making.
We introduce evidence-based transparency through detailed reasoning explanations and ranked evidence snippets that enable users to understand and verify system decisions. This addresses a critical gap in existing citation verification tools that provide classifications without supporting evidence, making our system interpretable and trustworthy for academic applications where understanding the basis for verification decisions is essential.
Our framework incorporates confidence calibration and uncertainty quantification, enabling practical deployment scenarios where low-confidence predictions can be flagged for human review. This bridges the gap between fully automated processing and human oversight, addressing real-world deployment needs where system reliability must be transparent and actionable.
Our demonstration that fine-tuned lightweight models achieve competitive performance with commercial LLMs whilst requiring significantly reduced computational resources addresses accessibility concerns in academic technology deployment. The Qwen3 4B modelâs 90.01% character similarity and superior length calibration (-0.7 words deviation) demonstrate effective performance-efficiency trade-offs for practical deployment.
To maximise research impact and enable widespread adoption, we commit to open-sourcing both the fine-tuned models and the complete SemanticCite framework. This open-source approach will facilitate reproducible research, enable customisation for domain-specific applications, and reduce barriers to implementation across diverse institutional contexts. The modular architecture supports easy integration with existing research workflows and citation management systems.
### 5.2 Limitations and Future Work
Several limitations warrant acknowledgement and suggest directions for future research. The evaluation framework relies exclusively on OpenAI GPT-4.1 ground truth annotations, potentially introducing systematic biases inherent in commercial language models. Future work should incorporate human expert annotations and inter-annotator agreement studies to validate automated ground truth generation and assess potential model-specific biases.
The current dataset, whilst comprehensive across eight academic disciplines, remains limited to English-language publications and open-access materials. Expanding to multilingual contexts and proprietary publications would enhance generalisability. Additionally, the temporal focus on 2019-2023 publications may not capture citation patterns across different research paradigms or historical periods.
The framework currently processes multi-reference citations by evaluating each referenced source individually rather than assessing their collective evidential support. This approach can result in citations being classified as PARTIALLY SUPPORTED when individual references each provide partial evidence, even though their combined evidence may constitute full support for the assertion. Future development should implement advanced reasoning mechanisms that evaluate the collective strength of evidence across multiple references, incorporating logical frameworks to assess whether partial support from individual sources aggregates to comprehensive claim validation. Such systems would require sophisticated evidence synthesis algorithms capable of identifying complementary, reinforcing, and contradictory information patterns across reference sets to provide more nuanced and accurate verification assessments.
A critical limitation affecting system accuracy is the availability of full-text reference documents for verification. While the framework implements fallback mechanisms to use reference abstracts when full texts are unavailable, this significantly constrains verification depth and accuracy. Many academic publications remain behind paywalls or lack open-access versions, forcing the system to rely on incomplete abstract-level information that may miss crucial supporting or contradictory evidence present in complete documents. This limitation represents the primary impediment to achieving higher accuracy rates, as comprehensive citation verification inherently requires access to the complete source material being referenced.
The frameworkâs applicability extends beyond academic citation verification to broader fact-checking applications for AI-generated content. The same 4-class taxonomy (SUPPORTED, PARTIALLY SUPPORTED, UNSUPPORTED, UNCERTAIN) can be directly applied to verify claims in AI-generated reports, articles, and summaries against their purported sources. This extension addresses the critical challenge of AI hallucinations in content generation, providing systematic verification capabilities for automated content production systems.
Future development should focus on AI-assisted improvement suggestions that go beyond classification to provide specific, actionable recommendations for citation enhancement. Such systems could suggest alternative phrasings, identify missing contextual information, recommend additional supporting sources, and propose precise modifications to transform PARTIALLY SUPPORTED citations into SUPPORTED ones. This evolution from detection to correction would create comprehensive AI-powered writing assistance tools.
The systemâs reliance on text-based analysis limits applicability to multimodal citations involving figures, tables, or mathematical expressions common in scientific literature. Extending the framework to incorporate visual and mathematical content represents a significant technical challenge with substantial potential impact.
Deployment scalability remains a practical consideration. Whilst fine-tuned models reduce computational requirements compared to commercial APIs, processing large document corpora still requires substantial resources. Future work should explore efficient inference techniques and distributed processing architectures for institutional-scale implementation.
### 5.3 Implications for Research Quality and Broader Applications
The demonstrated effectiveness of SemanticCite has broader implications for research quality and scholarly communication. Automated citation verification tools can significantly enhance peer review efficiency by identifying problematic citations before human review, allowing reviewers to focus on higher-level scientific assessment rather than time-consuming fact-checking.
The systemâs ability to detect subtle citation issues through the PARTIALLY SUPPORTED category addresses a critical gap in existing verification approaches. These cases often require minor corrections that substantially improve citation accuracy without major manuscript revisions.
The framework advances contextual citation understanding by providing detailed reasoning and ranked evidence that illuminate the nuanced relationships between claims and their supporting sources. This enables researchers to better comprehend how individual citations fit within broader scholarly discourse, identify gaps in supporting evidence, and understand the strength of connections between claims and references across different academic contexts and disciplinary conventions.
SemanticCite enables systematic retrospective analysis of published work, allowing institutions to assess citation quality in their research output and identify patterns that may require additional training or support. This capability supports institutional research integrity initiatives and provides objective metrics for evaluating citation practices across departments and disciplines.
The frameworkâs direct applicability to AI-generated content verification represents a significant advancement in addressing widespread concerns about AI hallucinations. While recent improvements in models like GPT-5 have reduced hallucination rates to levels that âseem acceptable to usersâ for most cases, eliminating hallucinations entirely may prove impossible due to the fundamental statistical nature of how large language models operate [gibney2025hallucination]. As AI writing assistants become prevalent in content creation, the same classification taxonomy can verify claims in automatically generated reports, summaries, and articles against their sources, providing essential quality control for AI-powered publishing workflows.
## 6 Conclusion
This paper introduces SemanticCite, an AI-driven framework for automated citation verification that addresses fundamental limitations in existing approaches. Our 4-class classification taxonomy enables nuanced assessment beyond traditional binary schemes, whilst the hybrid retrieval architecture supports effective full-text analysis for accurate evidence extraction.
The demonstration that fine-tuned Qwen3 models achieve competitive performance with significantly reduced computational requirements establishes the feasibility of deploying sophisticated citation verification in resource-constrained environments. The optimal performance of the 4B model (83.64% weighted accuracy, 90.01% character similarity) demonstrates effective practical trade-offs for diverse deployment scenarios.
The frameworkâs practical implications extend beyond academic manuscript review. By providing actionable guidance through detailed reasoning and evidence extraction, SemanticCite bridges the gap between automated analysis and human editorial decision-making, supporting iterative manuscript improvement rather than binary acceptance/rejection outcomes. The same 4-class taxonomy applies directly to AI-generated content verification, addressing critical quality control challenges in automated reports, summaries, and AI-powered publishing workflows.
Future research directions include expanding multimodal capabilities, incorporating multilingual support, and developing AI-assisted improvement features that shift the system from detecting citation issues to actively supporting authors with specific recommendations for enhancement and source integration. These extensions will strengthen the frameworkâs applicability to fact-checking in automated content production systems.
SemanticCite represents a significant step towards research quality tools that augment rather than replace human expertise, providing scalable, open-source solutions for maintaining citation accuracy in an era of rapidly expanding scientific output and AI-generated content.