2511.16198

# : Citation Verification with AI-Powered Full-Text Analysis and Evidence-Based Reasoning **Authors**: - Sebastian Haan (The University of Sydney, Australia) ## Abstract Effective scientific communication depends on accurate citations that validate sources and guide readers to supporting evidence. Yet academic literature faces mounting challenges: semantic citation errors that misrepresent sources, AI-generated hallucinated references, and traditional citation formats that point to entire papers without indicating which sections substantiate specific claims. We introduce SemanticCite, an AI-powered system that verifies citation accuracy through full-text source analysis while providing rich contextual information via detailed reasoning and relevant text snippets. Our approach combines multiple retrieval methods with a four-class classification system (Supported, Partially Supported, Unsupported, Uncertain) that captures nuanced claim-source relationships and enables appropriate remedial actions for different error types. Our experiments show that fine-tuned lightweight language models achieve performance comparable to large commercial systems with significantly lower computational requirements, making large-scale citation verification practically feasible. The system provides transparent, evidence-based explanations that support user understanding and trust. We contribute a comprehensive dataset of over 1,000 citations with detailed alignments, functional classifications, semantic annotations, and bibliometric metadata across eight disciplines, alongside fine-tuned models and the complete verification framework as open-source software. SemanticCite addresses critical challenges in research integrity through scalable citation verification, streamlined peer review, and quality control for AI-generated content, providing an open-source foundation for maintaining citation accuracy at scale. footnotetext: Email: sebastian.haan@sydney.edu.au footnotetext: Software Repository: https://github.com/sebhaan/semanticcite <details> <summary>x2.png Details</summary>  ### Visual Description Icon/Small Image (24x23) </details> Repository <details> <summary>hf_icon.png Details</summary>  ### Visual Description ## Icon/Symbol: Smiling Face with Hands ### Overview The image depicts a stylized yellow emoji face with a wide, open-mouthed smile, black eyes, and rosy cheeks. Two yellow hands with five fingers each are positioned below the face, palms facing outward. The design uses bold, flat colors with no shading or gradients. No textual elements are present. ### Components/Axes - **Face**: - Shape: Perfect circle - Color: Bright yellow (#FFD700) - Features: - Two black circular eyes (positioned at 10 and 2 o'clock) - Black crescent-shaped mouth (open wide, revealing a red tongue) - Two small orange circles (cheeks) at 11 and 1 o'clock - **Hands**: - Shape: Simplified human hands with five fingers each - Color: Yellow (#FFD700) - Positioning: Centered below the face, palms facing outward - **Background**: Pure white - **Border**: Thin white outline around the entire icon ### Detailed Analysis - **Color Analysis**: - Primary color: Yellow (face and hands) - Secondary colors: Black (eyes/mouth), Orange (cheeks), Red (tongue) - No gradients or shadows - **Typography**: No text present - **Spatial Relationships**: - Face occupies upper 60% of the icon - Hands occupy lower 40%, centered beneath the face - All elements are perfectly symmetrical ### Key Observations 1. The open mouth with visible tongue suggests exaggerated joy/excitement 2. Raised hands indicate surprise, celebration, or "stop" gesture 3. Simplified design follows standard emoji conventions 4. No contextual elements (e.g., background scenes) 5. No variation in facial expression beyond the base smile ### Interpretation This emoji represents intense positive emotion, likely conveying: - Overjoyed happiness - Surprised excitement - Celebratory gesture - "I'm thrilled!" expression The combination of facial expression and hand positioning creates a universally recognizable symbol for extreme positive reactions in digital communication. The absence of text ensures cross-linguistic understanding. The design prioritizes clarity and immediate emotional recognition over realism, following established emoji design principles. </details> Model & Data Keywords: citation verification, semantic analysis, natural language processing, research integrity, automated fact-checking ## 1 Introduction ### 1.1 Problem Statement Citations serve as fundamental channels through which scientific knowledge is shared and developed within the academic ecosystem. Despite their critical importance, scholarly literature suffers from a notable prevalence of citation errors that compromise research integrity and quality. These errors manifest in several forms: citations supporting claims not made in referenced works, citations omitting critical qualifications present in sources, citations attributing findings to incorrect studies, and selective citation of results whilst ignoring contradictory evidence. Beyond these semantic errors, typographical mistakes in bibliographic information also occur, though digital citation management tools have reduced their frequency. Analysis of prestigious science journals reveals that semantic citation errors remain pervasive, with error rates of 25% on average [smith2020quotation]. The implications for research quality are profound. Incorrect citations inadvertently propagate erroneous ideas, contribute to the acceptance of scientific myths, and ultimately erode the credibility of academic scholarship [gotzsche2022citation]. Such inaccuracies can lead to unintended plagiarism, diminished academic standing, and misallocation of valuable research resources. Manual citation verification during peer review faces substantial limitations. Reviewers are increasingly burdened by overwhelming workloads due to rapid expansion in academic publishing, making it difficult to dedicate sufficient time to meticulous citation evaluation. The lack of standardised review criteria and training contributes to inconsistent evaluations, whilst personal biases can subtly influence outcomes [smith2006peer, tennant2020limitations, mishra2025challenges, sizo2025defining]. A pressing new challenge has emerged with AI-generated content, where advanced language models produce convincing but non-existent “fake citations” that are extraordinarily difficult to detect manually at scale. Recent studies demonstrate that even advanced models like GPT-5 fabricate citations 39% of the time when operating without internet access, highlighting the fundamental nature of this problem [gibney2025hallucination]. These hallucinations arise from the statistical nature of large language models, which make predictions by generalising from learned associations, leading to plausible but sometimes incorrect outputs [gibney2025hallucination]. As AI-assisted writing becomes prevalent in academic contexts, traditional verification approaches designed for human authors prove insufficient for detecting AI-generated content that can produce highly plausible but entirely fabricated citations at scale. Traditional citation presentation provides no mechanism for readers to quickly understand cited work context without manually retrieving and reading entire source documents. Citations merely point to entire papers without indicating specific supporting sections, forcing readers to interrupt their flow, locate sources, search through lengthy documents for relevant passages, and return to the original text. This lack of precise contextual guidance impedes efficient knowledge discovery and creates barriers to thorough fact-checking during both research and peer review processes. Current automated citation tools remain severely limited across multiple dimensions. Reference management software (EndNote, Zotero, Mendeley) excel at bibliographic formatting but provide no semantic verification capabilities. Existing automated verification systems validate citations against authority databases (Web of Science, PubMed, CrossRef), detecting incorrect metadata or non-existent references, but cannot verify whether citation claims accurately reflect source document content. A reference may exist with correct metadata whilst the citing author mischaracterises the source’s actual findings. Similarly, format-checking systems verify style compliance but do not analyse semantic alignment between claims and sources. Even systems attempting content-level verification typically rely on abstract-only analysis rather than examining complete source documents. This approach misses critical supporting evidence, methodological caveats, limitations discussions, and contradictory findings present in full papers. Important contextual information such as experimental conditions, population constraints, statistical significance details, and author-acknowledged limitations rarely appears in abstracts yet proves essential for accurate citation verification. Furthermore, traditional verification approaches employ simple binary schemes (supported/unsupported) that fail to capture partial support, where core assertions align with sources but contextual elements are missing. These situations require different remedial actions than completely unsupported claims, yet binary systems cannot distinguish between them. Additionally, automated systems provide classifications without explaining reasoning or presenting supporting evidence, preventing independent validation. These converging challenges create an urgent need for comprehensive citation verification systems. Pervasive semantic errors, limited verification tools, opaque automated systems, AI-generated hallucinations, and manual verification friction demand solutions that detect errors, explain reasoning, provide actionable guidance, and scale efficiently whilst maintaining the transparency essential for scholarly work. ### 1.2 Research Objectives This work addresses these fundamental limitations through four primary objectives. First, we develop an automated full-text citation verification system that transcends surface-level bibliographic checks to perform deep semantic analysis of citation content against complete source documents. Second, we create a nuanced 4-class classification scheme (Supported, Partially Supported, Unsupported, Uncertain) that captures the complexity of citation-reference relationships beyond traditional binary approaches, enabling appropriate remedial actions for different types of citation errors. Third, we demonstrate that fine-tuned lightweight models can achieve competitive performance with large commercial language models whilst requiring significantly reduced computational resources. Fourth, we implement transparent evidence-based reasoning that enables users to understand and independently validate system decisions through relevant text snippets extracted from source documents and detailed explanations of classification rationale. This transparency is essential for maintaining the verifiability and trust required in academic contexts, addressing the opacity that has plagued previous automated verification systems. ### 1.3 Contributions This paper introduces SemanticCite, the first comprehensive AI-powered framework for automated full-text citation verification. Unlike existing approaches that rely on abstract-level analysis or binary classification schemes, our system performs deep semantic analysis of complete source documents using a 4-class taxonomy that captures the nuanced relationships between citations and their references. SemanticCite bridges the gap between automated detection and actionable remediation by providing detailed evidence-based reasoning that enables authors and reviewers to understand and address citation inaccuracies systematically. This represents a fundamental advance from simple error detection to comprehensive citation quality assessment with practical guidance for manuscript improvement. By combining hybrid retrieval and reranking techniques with fine-tuned language models, SemanticCite enables thorough citation analysis while maintaining computational efficiency, making sophisticated verification accessible to individual researchers and institutions regardless of computational resources. We contribute three key resources spanning data, models, and implementation. Our dataset comprises 1,111 analysed citations across 8 academic disciplines, including extracted claims, 4-class semantic classifications, reasoning explanations, confidence scores, and ranked evidence snippets. We release fine-tuned Qwen3 models (1.7B and 4B parameters) trained for both citation preprocessing and semantic classification tasks that achieve competitive performance with commercial systems while requiring significantly fewer computational resources. We open-source the SemanticCite framework code, including claim extraction pipelines, hybrid retrieval systems, and classification method. ## 2 Related Work ### 2.1 Citation Analysis and Verification Traditional citation analysis has primarily focused on bibliometric approaches using citation counts as indicators of research impact, failing to capture semantic relationships between citing statements and referenced sources. The field of automated fact-checking has provided important foundations for citation verification. Wadden et al. (2020) [wadden2020fact] introduced scientific claim verification with their SciFact dataset comprising 1.4K expert-written claims paired with abstracts annotated for veracity, demonstrating that domain adaptation significantly improves performance versus general-purpose fact-checking models. However, existing systems face significant limitations. Most rely on abstract-level analysis rather than full-text examination, potentially missing crucial evidence within complete documents. Binary classification schemes (SUPPORTS/REFUTES) fail to capture nuanced citation-reference relationships [wadden2020fact]. The emergence of AI-generated content has introduced new challenges, with advanced language models producing convincing but non-existent citations. Whilst tools like CiteSure [citesure2024] address this specific problem and systems like Morressier’s Integrity Manager detect various misconduct indicators, these specialised tools focus on specific error types rather than comprehensive citation verification [sae2024research]. Some existing platforms analyse citation usage within citing documents, extracting surrounding context and classifying whether the citing statement supports or contradicts the reference generally [nicholson2021scite]. However, these systems examine the directionality in reverse, i.e., they assess whether the citation supports the reference rather than whether the source document supports the specific citation claim. They cannot verify if the referenced source actually contains the evidence the citing author asserts, leaving the fundamental question of claim accuracy unaddressed. ### 2.2 Multi-class Citation Classification in NLP The evolution from binary to multi-class classification reflects growing recognition that real-world relationships exist on spectra rather than simple dichotomies. Multi-class approaches offer significant advantages by capturing degrees of alignment rather than forcing binary decisions. Systems employing classifications such as “supported,” “partially supported,” “unsupported,” and “uncertain” provide granular assessments that better reflect scientific discourse complexity [haan2025developing]. The Jurgens et al. (2018) taxonomy [jurgens2018measuring] represents a notable contribution, defining six categories (Background, Extends, Uses, Motivation, Compare/Contrast, Future work) for citation intent classification using the ACL-ARC dataset of 1,941 expert-annotated instances. Subsequent research expanded available data with the SciCite dataset, over five times larger than ACL-ARC, simplifying taxonomy to three categories (Background, Method, Result) whilst achieving superior performance with pre-trained language models like XLNet (88.93% F1 score) [mercier2020impactcite]. The diversity of taxonomies across datasets indicates lack of universal agreement on citation function definitions. This fragmentation, coupled with recognition that citations may serve multiple purposes, suggests current single-label classification may oversimplify relationships, necessitating flexible frameworks supporting customisable taxonomies and multi-label classification [abu-jbara2013, lauscher2021multicite]. ### 2.3 Retrieval-Augmented Generation Retrieval-augmented generation (RAG) has emerged as a powerful paradigm for fact-checking applications, combining information retrieval with language model generation capabilities [lewis2020retrieval, gao2023retrieval]. In citation verification contexts, RAG enables systems to retrieve relevant evidence from source documents and generate informed assessments of citation accuracy. Traditional RAG approaches face limitations when applied to academic texts due to their complexity and specialised terminology [li2024budget]. Recent advances in reverse RAG methodologies show promise for citation verification by first extracting or summarising information, then linking each data point back to source documents through systematic fact verification. This approach ensures every claim is traceable to original sources, addressing hallucination problems in standard generative models [rajpurkar2024claimify, metropolitansky2025towards]. Hybrid retrieval approaches combining dense semantic search with sparse keyword matching have demonstrated superior performance over individual methods [arivazhagan2023, sawarkar2024blended, sultania2024domain]. Dense retrieval using vector embeddings captures semantic relationships, whilst sparse methods like BM25 (Robertson & Zaragoza, 2009) ensure exact term correspondence. Neural reranking using cross-encoders further optimises candidate ordering based on query-document relevance, providing robust foundations for evidence-based citation verification systems. ## 3 Methodology ### 3.1 System Architecture SemanticCite employs a four-stage pipeline that combines hybrid retrieval-augmented generation with large language models for automated citation verification. The system operates through sequential stages: text preprocessing, hybrid retrieval, neural reranking, and LLM-based analysis, implemented using the LangChain framework with support for multiple providers. Figure 1 provides an overview of the complete citation verification pipeline, illustrating the flow from input processing through to structured output generation. <details> <summary>pipeline_diagram.png Details</summary>  ### Visual Description ## Flowchart: Semantic Citation Pipeline ### Overview The image depicts a multi-stage pipeline for processing citations and reference documents using a combination of traditional NLP techniques, vector embeddings, and large language model (LLM) analysis. The workflow progresses from raw input data through processing, embedding, retrieval, reranking, and finally to structured output with verification. ### Components/Axes **Input Stage (Blue Box):** - Citation Text - Reference Document - Metadata (optional) **Processing Stage (Orange Box):** - Document Processing - PDF extraction - Text chunking - Citation Processing - Remove references - Extract core claim **Embedding Stage (Red Box):** - Vector Store (Embeddings) **Hybrid Retrieval Stage (Light Blue Box):** - Dense Retrieval (Vector similarity) - Sparse Retrieval (BM25 keyword) **Reranking Stage (Purple Box):** - Neural Reranking (FlashRank) **LLM Analysis Stage (Green Box):** - Citation Verification - Citation vs Evidence - Metadata context - Structured output **Output Stage (Gray Box):** - Classification (4 categories) - Evidence (Top 3 chunks + scores) - Reasoning (Detailed explanation) - Metadata (Timestamps + metrics) ### Detailed Analysis 1. **Flow Direction**: - Input → Processing → Embedding → Hybrid Retrieval → Reranking → LLM Analysis → Output - Embedding branches to both Hybrid Retrieval and Reranking - Hybrid Retrieval branches to Reranking 2. **Color Coding**: - Blue: Input - Orange: Processing - Red: Embedding - Light Blue: Hybrid Retrieval - Purple: Reranking - Green: LLM Analysis - Gray: Output 3. **Key Techniques**: - BM25 keyword retrieval - FlashRank neural reranking - Vector similarity matching - 4-category classification system ### Key Observations - The pipeline integrates both traditional (BM25) and modern (vector similarity) retrieval methods - Metadata is used throughout the process, from input to final output - Verification occurs at two stages: during citation processing and final LLM analysis - The output includes both structured data (classification) and explanatory text (reasoning) ### Interpretation This pipeline demonstrates a hybrid approach to citation verification that combines: 1. **Semantic Search**: Through vector embeddings and similarity matching 2. **Keyword Matching**: Using BM25 for sparse retrieval 3. **Neural Reranking**: To improve relevance of retrieved documents 4. **LLM Verification**: For final citation validation and explanation generation The system appears designed to handle complex citation verification tasks by: - First processing raw documents into structured chunks - Creating both dense and sparse representations - Using multiple retrieval strategies to find relevant evidence - Finally using an LLM to verify claims against evidence and generate explanations The presence of metadata at multiple stages suggests the system can handle temporal or contextual verification needs, while the 4-category classification implies a structured approach to categorizing citation types or verification outcomes. </details> Figure 1: Semantic Citation Verification Pipeline: A multi-stage automated system for citation verification combining document processing (PDF extraction and text chunking), vector embedding storage, hybrid retrieval using both dense semantic similarity and sparse BM25 keyword matching, neural reranking with FlashRank, and LLM-based analysis. The pipeline outputs a classification result, supporting evidence, detailed reasoning, and confidence score for each citation verification task. The citation verification system takes three inputs: citation text, reference document, and optional metadata. Then it produces a classification label, confidence score, supporting evidence, and detailed reasoning. The system processes reference documents through text extraction and chunking, whilst citations undergo preprocessing to remove attribution markers and extract core factual claims. Document processing begins with text extraction using PyMuPDF PyMuPDF Documentation: https://github.com/pymupdf/PyMuPDF for PDF files, followed by recursive text splitting into 512-character chunks with 50-character overlap. The chunking strategy employs hierarchical separators including paragraph breaks, sentences, punctuation, and whitespace to maintain semantic coherence whilst ensuring manageable sizes for retrieval operations. Citation preprocessing transforms raw citations into standardised claims by removing reference markers, author attributions, and publication identifiers whilst preserving numerical values and factual assertions. This transformation converts author-centric statements to fact-centric assertions suitable for independent verification. For example, “Smith et al. (2020) found that exercise reduces cardiovascular risk by 30%” becomes “Exercise reduces cardiovascular risk by 30%”. The system supports flexible configuration of LLM providers including OpenAI, Anthropic Claude, Google Gemini, and local models via LiteLLM LiteLLM Documentation: https://docs.litellm.ai/docs/. Similarly, embedding models can be configured across OpenAI, local SentenceTransformers, or custom endpoints, enabling deployment across diverse computational environments whilst maintaining consistent performance. Temperature settings are controlled for reproducible outputs, with deterministic citation processing using zero temperature. ### 3.2 Classification Scheme Our 4-class taxonomy represents a fundamental contribution to citation verification methodology, providing nuanced assessment beyond traditional binary approaches. The classification scheme follows a logical progression from fully supported to unsupported citations, with an uncertainty class capturing ambiguous cases where clear determination cannot be made. Figure 2 illustrates the complete classification framework with actionable guidance for each category. The development of this taxonomy addresses critical limitations in existing binary classification systems. Traditional SUPPORTS/REFUTES schemes, whilst computationally simpler, fail to capture the complexity of citation-reference relationships commonly encountered in academic literature. Many citations exist in intermediate states requiring different remedial actions, making binary classification insufficient for practical citation verification applications. <details> <summary>classification_scheme_diagram.png Details</summary>  ### Visual Description ## Classification Scheme: Citation Alignment Framework ### Overview The image presents a four-quadrant classification system for evaluating citation alignment, with color-coded categories (green, yellow, red, gray) and corresponding indicators and actions. The framework guides reviewers in assessing and addressing citation accuracy and contextual relevance. ### Components/Axes 1. **Legend (Top Section)**: - **Green**: SUPPORTED (Fully Aligned) - **Yellow**: PARTIALLY SUPPORTED (Partially Aligned) - **Red**: UNSUPPORTED (Misaligned) - **Gray**: UNCERTAIN (Indeterminate) 2. **Categories (Left-to-Right Flow)**: - **SUPPORTED (Fully Aligned)**: - Indicators: Complete alignment, proper contextual usage, comprehensive representation - Actions: No changes needed, consider adding context, citation is accurate - **PARTIALLY SUPPORTED (Partially Aligned)**: - Indicators: Core claim supported, missing nuances/context, simplified representation - Actions: Review evidence snippets, revise to include key details, clarify/expand citation - **UNSUPPORTED (Misaligned)**: - Indicators: Claim absent from source, contradiction with source, misrepresentation - Actions: Remove/replace claim, search alternative references, review reasoning provided - **UNCERTAIN (Indeterminate)**: - Indicators: Ambiguous content, insufficient information, inconclusive relationship - Actions: Review evidence carefully, revise for clarity, consider removing citation ### Detailed Analysis - **Color Coding**: - Green (SUPPORTED) represents optimal alignment. - Yellow (PARTIALLY SUPPORTED) indicates partial but incomplete alignment. - Red (UNSUPPORTED) signals direct contradictions or missing evidence. - Gray (UNCERTAIN) denotes ambiguous or inconclusive cases. - **Indicators**: - **SUPPORTED**: Emphasizes full fidelity to source material. - **PARTIALLY SUPPORTED**: Highlights gaps in nuance or context. - **UNSUPPORTED**: Focuses on direct conflicts or absence of evidence. - **UNCERTAIN**: Addresses cases where evidence is unclear or insufficient. - **Actions**: - **SUPPORTED**: Minimal intervention required. - **PARTIALLY SUPPORTED**: Targeted revisions to strengthen alignment. - **UNSUPPORTED**: Radical changes or replacement of claims. - **UNCERTAIN**: Cautious evaluation with potential removal. ### Key Observations 1. **Hierarchical Structure**: Categories progress from fully aligned (green) to misaligned (red), with uncertain (gray) as a neutral outlier. 2. **Action Gradient**: Interventions escalate from "no changes" (green) to "consider removal" (gray). 3. **Contextual Focus**: All categories emphasize contextual accuracy, with SUPPORTED requiring only minor adjustments and UNSUPPORTED demanding complete overhaul. ### Interpretation This framework operationalizes citation evaluation by linking alignment quality to specific corrective actions. The color-coded system enables rapid assessment, while the actionable indicators provide clear pathways for improvement. Notably: - **SUPPORTED** (green) prioritizes preservation of accurate citations. - **PARTIALLY SUPPORTED** (yellow) bridges minor gaps through targeted revisions. - **UNSUPPORTED** (red) enforces strict adherence to source material, rejecting misrepresentations. - **UNCERTAIN** (gray) introduces a risk-averse approach, acknowledging limitations in evidence quality. The framework reflects a Peircean investigative logic: starting with the assumption of alignment (green) and progressively addressing contradictions (red) or ambiguities (gray) through iterative refinement. The emphasis on contextual usage across all categories underscores the importance of semantic fidelity beyond mere textual matching. </details> Figure 2: Four-Category Classification Scheme for Source-Claim Alignment Assessment #### 3.2.1 Detailed Classification Categories SUPPORTED (Fully Aligned): Citations demonstrate complete alignment between claims and source material, maintaining proper context and comprehensive representation. These citations accurately convey the referenced content without oversimplification, missing context, or distortion of meaning. The core indicators include complete factual alignment, appropriate contextual usage, and comprehensive representation of source nuances. For instance, a citation stating “The human genome contains approximately 20,000 protein-coding genes” fully aligns with source material stating “Recent genomic studies have confirmed that the human genome encodes about 20,000 protein-coding genes.” Such citations receive high confidence scores and require no modifications, though users may optionally add additional context from provided evidence snippets. PARTIALLY SUPPORTED (Partially Aligned): Core claims are present in source material but lack critical nuances, contextual information, or important qualifications, leading to incomplete representation. These citations contain accurate fundamental assertions but oversimplify complex findings or omit essential caveats present in the source. Key indicators include core claim support with missing minor nuances, simplified representation, or incomplete contextual information. An example would be citing “Exercise guarantees a 50% reduction in cardiovascular disease risk” when the source states “Regular exercise has been shown to reduce cardiovascular disease risk by up to 50% in some studies, though results can vary.” The citation captures the core finding but overstates certainty and omits variability. Such citations receive medium confidence scores and require minor revisions to include qualifying details and expand scope for complete accuracy. UNSUPPORTED (Misaligned): Claims are absent from, contradicted by, or significantly misrepresent source material. These citations demonstrate fundamental disconnection between stated claims and referenced content, ranging from complete absence of supporting evidence to direct contradiction of source findings. Critical indicators include claims absent from source material, direct contradiction between claims and evidence, or significant misrepresentation of source content. For example, claiming “Drinking coffee cures cancer” when the source states “Some studies suggest compounds in coffee may lower risk of certain cancers, but evidence is not conclusive” represents serious misrepresentation. Such citations receive low confidence scores and require major revision, removal, or alternative reference identification. UNCERTAIN (Indeterminate): Ambiguous relationships exist where insufficient evidence, unclear claim-reference relationships, or inadequate verification context prevent definitive classification. These cases often involve vague claims, complex source material with multiple interpretations, or insufficient contextual information for reliable assessment. Key indicators include ambiguous claim or source content, insufficient contextual information, or inconclusive claim-reference relationships. An example might be citing “Vitamin D supplementation prevents all cases of osteoporosis” when the source discusses “varying degrees of effectiveness depending on factors such as age, dosage, and baseline deficiency levels.” The broad generalisation in the claim combined with nuanced source findings creates ambiguity requiring additional clarification. Such citations warrant careful review of evidence snippets to identify ambiguity sources. #### 3.2.2 Actionable User Guidance Framework The classification scheme provides specific, actionable guidance aligned with potential user responses: - SUPPORTED citations receive confirmation that no changes are needed, with optional suggestions for additional context enhancement. - PARTIALLY SUPPORTED citations receive specific guidance on reviewing evidence snippets to identify missing nuances and revising content to include qualifying details. - UNSUPPORTED citations prompt users to remove inaccurate claims, replace them with evidence-based statements, or seek alternative supporting references. - UNCERTAIN classifications guide users to carefully review evidence snippets, revise citations to include necessary contextual details, or consider citation removal if relationships remain unclear. This structured guidance framework transforms abstract classification labels into concrete editorial actions, bridging the gap between automated analysis and practical manuscript improvement. The approach recognises that different classification outcomes require different remedial strategies, moving beyond simple accuracy scores to provide constructive improvement pathways. #### 3.2.3 Advantages Over Binary Classification The multi-class approach offers several critical advantages over binary schemes. Binary classification forces complex citation-reference relationships into oversimplified dichotomies, failing to distinguish between minor revisions and major corrections. Our taxonomy captures nuanced relationships requiring different remedial actions, providing more precise feedback for authors and reviewers. Furthermore, the inclusion of an UNCERTAIN category acknowledges inherent limitations in automated analysis, providing transparent handling of ambiguous cases rather than forcing potentially incorrect binary decisions. This approach enhances system trustworthiness by explicitly flagging cases requiring human judgment whilst maintaining high confidence in clear-cut classifications. The taxonomy also aligns with real-world editorial workflows where citations often require incremental improvements rather than complete acceptance or rejection. By providing graduated classification levels, the system supports iterative manuscript refinement and targeted quality improvement efforts. ### 3.3 Hybrid Retrieval System The retrieval architecture combines dense semantic search with sparse keyword matching to maximise evidence recall whilst maintaining precision. This hybrid approach addresses individual method limitations: dense retrieval captures semantic relationships but may miss exact term matches, whilst sparse retrieval ensures keyword correspondence but lacks semantic understanding. Dense Retrieval employs vector embeddings stored in ChromaDB using cosine similarity matching. Document chunks are mapped into high-dimensional vector spaces where semantically similar content clusters together. When processing citation claims, the system calculates similarity scores between query vectors and stored document vectors, retrieving the 15 most semantically relevant chunks. This approach excels at capturing conceptual relationships and paraphrased content that might use different terminology whilst conveying similar meaning. Sparse Retrieval utilises BM25 algorithm for exact term matching, ensuring that citations requiring specific keyword correspondence are properly identified. BM25 considers term frequency, document length normalisation, and inverse document frequency to rank passages based on lexical overlap with citation claims. This method retrieves an additional 15 chunks focusing on exact term matches, complementing semantic retrieval by capturing precise terminology and specific numerical values. Neural Reranking applies FlashRank cross-encoder optimisation to the combined retrieval results. After deduplication, the hybrid results undergo reranking based on query-document relevance scores computed by BERT-based cross-encoders. This process optimises candidate ordering and selects the top 3 most relevant passages for downstream analysis, ensuring focus on the most pertinent evidence whilst maintaining computational efficiency. Document Chunking Strategy employs recursive character splitting with carefully chosen parameters to balance granularity with context preservation. The 512-character chunk size with 50-character overlap ensures sufficient context for semantic understanding whilst maintaining computational tractability. Hierarchical separators prioritise natural text boundaries, starting with paragraph breaks and progressively falling back to sentences, punctuation, and finally individual words when necessary. The hybrid approach demonstrates superior performance compared to individual methods by leveraging complementary strengths. Dense retrieval captures nuanced semantic relationships crucial for academic content, whilst sparse retrieval ensures critical exact matches are not missed. Neural reranking provides final optimisation, resulting in high-quality evidence selection for citation verification. ### 3.4 Citation Processing Pipeline The processing pipeline transforms raw academic citations into verifiable claims whilst preserving essential factual content. This preprocessing stage is critical for accurate verification as raw citations often contain attribution elements that obscure core factual assertions. Preprocessing Operations systematically transform citations through several key steps. Reference markers such as brackets, parentheses, and citation numbers are removed to isolate factual content from bibliographic formatting. Author attributions are eliminated whilst preserving substantive claims, converting statements like “Smith et al. found that X” to “X was found” using passive voice constructions. Numerical values, statistical measures, and specific metrics are carefully preserved as these often represent the most verifiable aspects of citations. The process ensures all claims become independently verifiable statements that can be assessed against source material without requiring external attribution context. Structured Output Parsing employs LLM-based analysis with explicit schema requirements to ensure consistent and comprehensive results. The system generates structured JSON outputs containing classification labels, detailed reasoning with both summary and point-by-point analysis, confidence scores, and ranked evidence snippets with relevance scores and source locations. This structured approach enables programmatic processing whilst maintaining human interpretability. The verification prompt carefully incorporates processed claims, top-ranked evidence snippets with relevance scores, and optional metadata for enhanced contextual understanding. Prompts explicitly instruct models to assess factual alignment between citations and source material whilst considering broader study context when metadata is available. Instructions emphasise objective evaluation based on textual evidence rather than general knowledge or assumptions. Evidence Snippet Ranking applies systematic relevance scoring to prioritise the most pertinent supporting material. Each evidence snippet receives numerical relevance scores indicating how well it addresses the citation claim, enabling users to focus on the most important supporting or contradicting evidence. The system tracks comprehensive metadata including processing timestamps, reference file identifiers, chunk locations, and system version information for complete result provenance and reproducibility analysis. This systematic approach enables transparent verification where users understand classification rationale through extracted evidence and detailed reasoning, supporting both automated analysis and human review processes. The pipeline’s modular design allows for component-level optimisation and adaptation to different document types and verification requirements whilst maintaining consistent output quality. ### 3.5 Web Interface and Deployment <details> <summary>User_Interface.png Details</summary>  ### Visual Description ## Screenshot: SemanticCite AI-Powered Citation Verification Interface ### Overview The image shows a web-based interface for SemanticCite, an AI tool for verifying citations against reference documents. The layout is divided into two primary sections: a left sidebar for model configuration and a right main area for inputting citation text and reference documents. --- ### Components/Axes #### Left Sidebar (Model Configurations) 1. **LLM Configuration** - **LLM Provider**: Dropdown with value "local" - **Dual Model Configuration** - **Preprocessing Model**: Input field with value "ollama/SemanticCite-Refiner-Qwen3-1l" - **Classification Model**: Input field with value "ollama/SemanticCite-Checker-Qwen3-4" - **Model Endpoint URL**: Input field with value "http://localhost:11434" - **Embedding Configuration** - **Embedding Provider**: Dropdown with value "local" - **Embedding Model Name**: Input field with value "Qwen/Qwen3-Embedding-0.6B" 2. **Disclaimers** - **Privacy**: "All processing occurs locally" (highlighted in blue) - **AI-Assisted Analysis**: "Verify results independently" #### Main Content Area 1. **Header** - **Title**: "AI-powered semantic citation verification" - **Subtitle**: "Analyse whether your citations are accurately supported by reference documents." 2. **Input Fields** - **Citation Text**: Textarea labeled "Enter citation text" - **Reference Document** - **Upload Options**: - "Upload File" (blue button) - "Download from URL" (gray button) - **Drag-and-Drop Zone**: "Drag and drop file here" with file type restrictions (TXT, PDF, MD, 200MB limit) - **Reference Metadata (Optional)**: Textarea with placeholder "Enter title, abstract, and other reference metadata... Supports plain text and markdown formatting." 3. **Action Button** - **Check Citation**: Blue button with magnifying glass icon --- ### Detailed Analysis #### Left Sidebar - **LLM Configuration**: Uses local models for language processing. - **Dual Model Setup**: Separates preprocessing (refinement) and classification tasks using Qwen3-based models. - **Embedding Configuration**: Local Qwen3 embedding model for vector representation. - **URL Endpoint**: Localhost server at port 11434 for model communication. #### Main Content Area - **Input Workflow**: 1. Users enter citation text. 2. Upload or link reference documents. 3. Optional metadata enhances context for verification. - **Privacy Emphasis**: Explicit disclaimer about local processing to address data security concerns. --- ### Key Observations 1. **Model Specialization**: The system uses distinct models for preprocessing (refinement) and classification (verification). 2. **Local Processing**: All operations occur on the user's device, avoiding cloud dependency. 3. **File Flexibility**: Supports multiple document formats and direct URL integration. 4. **Metadata Guidance**: Optional metadata field encourages users to provide contextual information for better verification accuracy. --- ### Interpretation This interface demonstrates a privacy-focused, locally hosted AI tool for academic citation verification. The dual-model architecture suggests a two-stage process: first refining citations (preprocessing) and then validating them against references (classification). The emphasis on local processing and file format flexibility indicates a design prioritizing user control and data security. The optional metadata field implies that richer context improves verification outcomes, though the system remains functional without it. The clear separation of configuration and input areas enhances usability for technical users familiar with model customization. </details> Figure 3: SemanticCite web interface showing citation input, reference document upload options (file upload or URL download), optional metadata entry, and configurable model parameters. The interface supports multiple LLM providers and embedding models, enabling flexible deployment across different resource constraints and institutional requirements. To facilitate widespread adoption and ensure accessibility for researchers without programming expertise, we developed a web-based interface using Streamlit (Figure 3). The interface provides intuitive controls for citation verification, allowing users to input citation text, upload reference documents (PDF, TXT, or Markdown formats), or download references directly from URLs. Users can configure both LLM providers (OpenAI, Anthropic Claude, Google Gemini, or local Ollama models) and embedding models (local SentenceTransformers, OpenAI embeddings, or custom endpoints) through an interactive sidebar, enabling flexible deployment across different institutional resource constraints and privacy requirements. The system displays results including classification labels with color coding, confidence scores, detailed reasoning explanations, and ranked evidence snippets extracted from the reference document. All analysis results can be exported to Markdown format for documentation and integration into research workflows. This implementation demonstrates the practical deployment of the SemanticCite framework whilst maintaining full analytical capabilities and transparency in the verification process. ## 4 Model Development and Evaluation This section details the development and evaluation of fine-tuned large language models for automated citation verification within the SemanticCite framework. We present our complete experimental pipeline for training specialized citation analysis models, beginning with the systematic construction of a balanced, multi-disciplinary dataset spanning eight academic fields (§4.1). We describe our automated training data generation methodology that leverages LLM-based filtering and annotation to create high-quality datasets for both citation preprocessing and classification tasks (§4.2). The fine-tuning process employs QLoRA techniques with Qwen3 models [qwen2024] of varying scales to develop efficient, specialized systems for citation verification (§4.3). We introduce a weighted evaluation framework specifically designed to assess the ordinal nature of citation support categories (§4.4), and conclude with comprehensive experimental results demonstrating model performance across different scales and computational requirements (§4.5). This experimental framework establishes the effectiveness of fine-tuned language models for citation verification while providing insights into the trade-offs between model complexity, computational efficiency, and task-specific performance in academic text analysis. ### 4.1 Data Selection We constructed a high-quality training dataset using a multi-stage approach designed to ensure representative coverage across academic disciplines while maintaining rigorous quality standards for bibliometric metadata and full-text content. The resulting corpus targets balanced representation across citation impact levels, disciplinary boundaries, and functional citation categories, providing a robust empirical foundation for training and evaluating citation analysis models. An overview of the data selection process is presented in Figure 4. <details> <summary>data_selection_diagram.png Details</summary>  ### Visual Description ## Flowchart: Research Methodology for Citation Analysis ### Overview The flowchart outlines a four-stage research methodology for citation analysis, progressing from paper selection to dataset creation. It includes functional classification of citations, reference document acquisition, and a balanced training dataset summary. ### Components/Axes 1. **Stages** (Vertical Flow): - Stage 1: Stratified Paper Selection & Metadata Acquisition - Stage 2: Full-Text Acquisition - Stage 3: Reference Document Acquisition - Balanced Training Dataset (Bottom Section) 2. **Sub-Components**: - Functional Citation Classification (Left Box) - Citation Selection (Right Box) 3. **Dataset Metrics** (Bottom Section): - Citations Analyzed: 1,111 - Academic Fields: 8 - Citation Types: 6 - Numerical Citations: 27% ### Detailed Analysis #### Stage 1: Stratified Paper Selection & Metadata Acquisition - Uses Semantic Scholar Graph with quality filters - Citation impact stratification applied - Metadata collected across 8 academic fields: - Computer Science - Medicine - Chemistry - Biology - Materials Science - Physics - Geology - Psychology #### Stage 2: Full-Text Acquisition - Systematic PDF download with integrity verification - Full-text extraction performed #### Functional Citation Classification - LLM-based categorization into 6 types: - BACKGROUND - METHOD - COMPARISON - MOTIVATION - EXTENSION - FUTURE #### Citation Selection - Two-stage filtering and prioritization: - Single-reference contexts - Factual assertions - Quantitative claims - High verification potential #### Stage 3: Reference Document Acquisition - Acquires cited reference documents for verification - Processes: - Full-text - Abstracts - Multi-format processing #### Balanced Training Dataset - **Quantitative Metrics**: - 1,111 citations analyzed - 8 academic fields represented - 6 citation types included - 27% numerical citations - **Qualitative Features**: - Equal distribution across citation impact levels - Comprehensive metadata - Citation claims - Alignment classification - Reasoning - Reference text ### Key Observations 1. **Structured Progression**: Methodology follows a logical flow from paper selection to dataset creation 2. **LLM Integration**: Machine learning used for citation classification 3. **Field Diversity**: Covers 8 distinct academic disciplines 4. **Citation Balance**: 27% numerical citations suggest focus on quantitative analysis 5. **Multi-format Processing**: Accommodates various document types in reference acquisition ### Interpretation This methodology demonstrates a systematic approach to building a citation analysis dataset: 1. **Data Curation**: Stages 1-3 ensure high-quality, verified data collection through multiple filters (citation impact, verification potential) 2. **LLM Utilization**: Functional classification leverages machine learning for efficient categorization 3. **Dataset Balance**: The 27% numerical citations and equal distribution across impact levels suggest careful balancing for model training 4. **Interdisciplinary Scope**: Coverage of 8 fields indicates broad applicability potential The methodology appears designed to create a robust training dataset for citation analysis models, with particular emphasis on quantitative claims and verification potential. The inclusion of multi-format processing and comprehensive metadata suggests preparation for complex NLP tasks. </details> Figure 4: Overview of the data selection and processing pipeline for semantic citation verification #### 4.1.1 Data Collection and Processing Pipeline The training dataset was constructed through a systematic three-stage pipeline designed to collect academic papers with diverse citation characteristics across multiple disciplines, targeting a corpus of 4,000 papers with balanced representation across citation impact levels. Stage 1: Stratified Paper Selection The collection process employed a stratified sampling approach across eight academic fields (Computer Science, Medicine, Chemistry, Biology, Materials Science, Physics, Geology, and Psychology) using the Semantic Scholar Graph API. This selection ensures comprehensive coverage of distinct disciplinary citation practices and scholarly communication patterns. Papers were filtered according to stringent criteria including publication year (2019-2023), minimum citation count ( $\geq 5$ ), minimum reference count ( $\geq 10$ ), and open access availability. However, open-access designation did not guarantee that all papers were actually available for download. The temporal constraint ensures currency whilst providing sufficient time for citation accumulation, and the citation thresholds guarantee papers with meaningful scholarly impact and comprehensive reference networks. To ensure balanced representation across citation impact levels, papers were categorised into three citation rate bins (Low, Mid, High) based on field-specific empirically-derived thresholds that account for citations-per-year rates rather than absolute counts. This normalisation approach addresses the substantial variation in citation velocity across academic disciplines. These thresholds varied considerably across disciplines, reflecting domain-specific citation practices—for instance, Medicine exhibited substantially higher thresholds (Mid: 14, High: 31 citations/year) compared to Geology (Mid: 4, High: 7 citations/year). This stratification ensures that the dataset captures papers across the full spectrum of scholarly impact within each discipline, avoiding bias towards highly-cited outliers whilst maintaining representation of influential work. Stage 2: Metadata Acquisition For each selected paper, extensive metadata was retrieved including bibliographic information, abstracts, author details, venue information, external identifiers, and complete citation and reference networks. This metadata collection provides essential context for understanding citation patterns and enables sophisticated analysis of citation behaviour across different paper characteristics. Quality assurance measures excluded papers lacking valid open access PDF URLs or failing PDF format validation, ensuring accessibility of full-text content essential for detailed citation analysis. Stage 3: Full-Text Document Acquisition The final stage involved systematic download of PDF documents using validated open access URLs. Each download was verified for successful completion and proper PDF format before inclusion in the dataset, with verification including file size validation, PDF structure integrity checks, and preliminary text extraction testing. This methodology yielded a balanced, high-quality corpus of approximately 500 papers per academic field, with equal distribution across citation impact categories. The approach ensures representative coverage across academic disciplines and citation patterns whilst maintaining access to both structured bibliometric data and full-text content, providing a robust foundation for training citation analysis models. #### 4.1.2 Citation Function Classification Methodology Classification Framework The citation classification system employs a systematic approach to categorise citation functions within academic texts, building upon the established taxonomy of Jurgens et al. (2018) [jurgens2018measuring]. The methodology implements a six-category schema designed to capture the primary functional roles of citations in scholarly discourse. This approach balances thorough coverage of citation purposes with tractable computational requirements for automated classification. - BACKGROUND citations establish the general field, provide contextual information, or reference foundational knowledge essential for understanding the current work. - METHOD citations reference specific algorithms, techniques, datasets, software, or experimental procedures employed or adapted in the current work, representing instrumental citations that directly inform methodological choices. - RESULT_COMPARISON citations directly compare or contrast findings, results, or conclusions with those of the cited work, often appearing in results and discussion sections where authors evaluate their findings against existing evidence. - MOTIVATION citations indicate that the cited work directly inspired the research problem or provided primary motivation for the current study, representing foundational influences on research direction. - EXTENSION citations signal that the current work builds upon, extends, improves, or adapts methods or ideas from the cited work, indicating direct intellectual lineage. - FUTURE_WORK citations mention prior work specifically in the context of suggested research directions or open problems, typically appearing in concluding sections where authors outline potential developments. Dual-Dimensional Classification Approach The classification system operates along two complementary dimensions to capture both functional and quantitative aspects of citation usage. The primary dimension assigns each citation context to one of the six functional categories through systematic analysis of the surrounding textual context, considering both immediate sentence-level context and broader paragraph-level discourse patterns. The secondary dimension employs binary classification to identify whether the citation context explicitly involves numerical or quantitative information—including percentages, specific values, metrics, performance scores, or statistical data—directly related to the claim or comparison involving the cited work. This dual-dimensional approach recognises that quantitative citations often require different verification strategies compared to qualitative references, as numerical claims demand precise fact-checking against source documents whilst qualitative citations may require more nuanced semantic analysis. The quantitative dimension proves particularly valuable for identifying citations that make specific empirical claims requiring direct verification against source data. Computational Implementation The classification process uses large language models (LLMs) with structured prompting to ensure consistency and reproducibility across the entire dataset. Citation contexts are processed in configurable batches to optimise computational efficiency whilst maintaining classification accuracy. Each classification decision is accompanied by a concise rationale explaining the categorical assignment, providing transparency and enabling quality assessment of the automated classification process. The methodology incorporates error handling and validation mechanisms, including verification of output format compliance, categorical validity checks, and fallback procedures for problematic cases. #### 4.1.3 Citation Selection Methodology To generate high-quality training data for fine-tuning large language models on citation analysis tasks, we implemented a two-stage automated selection pipeline using LLM-based filtering and ranking designed to identify citation instances most suitable for verification analysis. The first stage applied strict filtering criteria to identify valid citation instances, requiring that: (1) the context contained only a single reference rather than multiple citations, ensuring unambiguous attribution and avoiding complications from citation clustering; (2) the context represented an actual citation sentence rather than a bibliography entry, maintaining focus on in-text scholarly discourse; and (3) the citation text was properly embedded within the contextual passage, ensuring sufficient surrounding context for meaningful analysis. From the filtered candidates, a second LLM stage selected the top 5 citations per paper that warranted verification analysis. The selection prioritised citations containing specific claims, numerical statements, factual assertions, and substantial content that could be meaningfully verified against their source references, rather than brief mentions or general acknowledgments. This prioritisation ensures that the resulting training data focuses on citations with sufficient semantic content to support meaningful verification tasks. Files with fewer than 3 citations were excluded to maintain dataset quality and ensure sufficient citation diversity within each source document. This systematic approach ensured the selection of citation instances most suitable for training models to assess citation-reference alignment and factual accuracy. ### 4.2 Training Data Generation Training datasets are generated for two distinct model types addressing different aspects of citation analysis. The Citation Preprocessing Model transforms raw citation statements into cleaned claims through systematic processing, creating input-output pairs mapping original citations to their standardised, processed forms. This preprocessing removes reference markers, author attributions, and publication identifiers whilst preserving numerical values and factual assertions, converting author-centric statements to fact-centric representations suitable for verification. The Classification Model generates training examples for citation support classification through a more complex pipeline involving: (1) preparing vector stores from reference documents using hybrid retrieval methods, (2) extracting semantically relevant text chunks through embedding-based similarity matching using Qwen3-Embedding-0.6B, and (3) generating structured outputs containing the four classification labels (SUPPORTED, PARTIALLY SUPPORTED, UNSUPPORTED, UNCERTAIN), detailed reasoning, and confidence scores. Ground truth annotations were generated using GPT-4.1 (gpt-4.1-2025-04-14) with carefully designed prompts ensuring consistent and reliable classifications. The citation preprocessing employed prompts instructing the model to remove reference markers, convert author-centred statements to fact-centred assertions, maintain numerical values, and ensure claims stand alone as verifiable statements. For classification analysis, GPT-4 was provided with processed citations, reference document metadata, and relevant text snippets, then instructed to classify citations using our 4-class taxonomy. The structured output schema required classification labels, reasoning dictionaries containing summary and detailed analysis, and numerical confidence ratings (0.0-1.0). ### 4.3 Model Fine-tuning The fine-tuning methodology employed QLoRA (Quantised Low-Rank Adaptation) techniques with Qwen3 language models to develop specialised systems for citation verification. Model Architecture and Quantisation The system used Qwen3 models in three configurations (1.7B, 4B, and 8B parameters) with 4-bit quantisation via BitsAndBytes to enable efficient training on consumer hardware. The QLoRA configuration employed a rank (r) of 16 with an alpha value of 32, targeting attention projection layers (q_proj, k_proj, v_proj, o_proj) and feed-forward network components (gate_proj, up_proj, down_proj). Gradient checkpointing through Unsloth optimisation reduced memory consumption whilst maintaining training stability, enabling deployment across diverse computational environments. Training Data Preparation and Configuration Three distinct training datasets were prepared for different citation analysis tasks. The Preprocessing Dataset contained 1,111 examples for text simplification and paraphrasing tasks, with input-output pairs averaging 49.2 and 38.8 tokens respectively. The Classification Dataset comprised 1,111 examples for citation verification tasks, featuring significantly longer sequences (average 810.9 input tokens, 178.5 output tokens) due to inclusion of reference snippets and metadata. The training protocol implemented a conservative approach with 2 epochs, a learning rate of $2\times 10^{-4}$ , and AdamW optimisation with 8-bit precision. Batch sizes were dynamically adjusted based on GPU memory (1-4 per device) with gradient accumulation steps (2-8) to maintain effective batch sizes. The system employed linear learning rate scheduling with 5% warmup ratio and 1% weight decay for regularisation. Task-Specific Training Approaches Text Preprocessing Models were trained with sequence lengths of 1024 tokens while the Citation Classification Models used an extended sequence lengths (4608 tokens). Structured Output Models generated JSON-formatted responses containing classification labels, detailed reasoning, and confidence scores (0.0-1.0), enabling programmatic integration and uncertainty quantification. Training data underwent systematic preprocessing whereby complex nested JSON structures were flattened into coherent text sequences, with task-specific instructions integrated using Alpaca formatting templates to create standardised prompt structures. Models were exported in multiple formats including LoRA adapters, merged weights, and GGUF quantised versions for deployment flexibility. The system maintained reproducibility through fixed random seeds and deterministic training procedures, with data splitting following a 90:10 train-test partition using stratified sampling. ### 4.4 Evaluation Methodology Our classification task involves predicting four categories with natural ordinal structure: SUPPORTED (3), PARTIALLY SUPPORTED (2), UNSUPPORTED (1), and UNCERTAIN (0). Unlike traditional multi-class evaluation that treats all misclassifications equally, we implement a weighted evaluation framework that penalizes errors proportionally to their semantic distance. #### 4.4.1 Weighted Evaluation Framework We define a weight matrix where misclassification penalty equals the absolute difference between ordinal values: $W(i,j)=|v(i)-v(j)|$ . This ensures that extreme errors (e.g., SUPPORTED $\to$ UNCERTAIN, weight = 3) receive higher penalties than adjacent-level misclassifications (e.g., SUPPORTED $\to$ PARTIALLY SUPPORTED, weight = 1). Weighted accuracy is calculated as: $$ \text{ACC}_{\text{weighted}}=1-\frac{\sum_{i,j}W(i,j)\times\text{CM}(i,j)}{N\times\max(W)} \tag{1} $$ where CM $(i,j)$ represents confusion matrix entries, $N$ is total predictions, and $\max(W)=3$ . We also calculate ordinal Mean Absolute Error (MAE) treating classes as ordered variables, providing intuitive 0-1 scale performance assessment. Our evaluation combines standard metrics (accuracy, F1-scores, Cohen’s $\kappa$ ) with ordinal-aware measures (weighted accuracy, ordinal MAE) to capture both traditional classification performance and task-specific semantic alignment. #### 4.4.2 Text Generation Quality Metrics Beyond classification accuracy, we assess text generation quality using character-level similarity and length calibration metrics. Character similarity is calculated using Jaccard similarity at the character level, measuring the overlap of unique characters between predictions and ground truth texts. Specifically, we convert both texts to lowercase, remove spaces, create sets of unique characters, and calculate Jaccard similarity as the intersection divided by the union of character sets. This metric captures lexical overlap and vocabulary consistency regardless of word order, providing insight into how well models preserve semantic content during text generation. Length difference measures the deviation in word count between predictions and ground truth, indicating model calibration in output verbosity. ### 4.5 Results #### 4.5.1 Model Performance Overview We evaluated three fine-tuned Qwen3 models (1.7B, 4B, 8B parameters) using QLoRA techniques on a stratified test set of 112 examples. Models were assessed across both preprocessing and classification tasks, with preprocessing involving citation text cleaning and standardization, while classification represents the primary citation verification task. #### 4.5.2 Preprocessing Task Performance Table 1: Preprocessing Performance Results | Qwen3 1.7B Qwen3 4B Qwen3 8B | 112 112 112 | 112 (100%) 112 (100%) 112 (100%) | 93.32% 94.15% 94.30% | 26.7 words 26.7 words 26.4 words | 26.3 words 26.3 words 26.3 words | +0.4 words +0.4 words +0.1 words | | --- | --- | --- | --- | --- | --- | --- | All models achieved exceptional performance on the preprocessing task, demonstrating 100% valid prediction rates with high character similarity ( $>93\$ ) and excellent length calibration. The preprocessing task involves cleaning citation text by removing reference markers, author attributions, and standardizing claims into fact-centric statements suitable for verification. The consistently high performance across all model sizes indicates that text preprocessing represents a simpler task compared to classification, where models can reliably learn pattern-based transformations. The 8B model achieved marginally superior character similarity (94.30%) and best length calibration (+0.1 words), though all models performed comparably, suggesting that even the 1.7B model is sufficient for preprocessing tasks in resource-constrained deployments. #### 4.5.3 Classification Task Performance Table 2: Classification Performance Results | Qwen3 1.7B Qwen3 4B Qwen3 8B | 50.00% 66.36% 66.07% | 75.15% 83.64% 83.93% | 33.50% 49.15% 50.13% | 46.61% 65.31% 65.73% | 88.07% 90.01% 89.33% | +4.7 words -0.7 words +3.6 words | | --- | --- | --- | --- | --- | --- | --- | #### 4.5.4 Key Findings 1. Weighted vs. Standard Performance: The two evaluation approaches reveal different aspects of model performance. Standard accuracy shows clear scaling with model size (1.7B: 50.00%, 4B: 66.36%, 8B: 66.07%), while weighted accuracy demonstrates that all models achieve substantially higher performance when accounting for ordinal relationships (1.7B: 75.15%, 4B: 83.64%, 8B: 83.93%). This indicates that models predominantly make semantically reasonable “near-miss” errors rather than extreme misclassifications. 2. Text Generation Quality and Calibration: The 4B model achieves superior text generation performance with the highest character similarity (90.01%) and best length calibration (-0.7 words from ground truth). The 8B model shows slightly lower character similarity (89.33%) and moderate length deviation (+3.6 words), while the 1.7B model exhibits the lowest character similarity (88.07%) and largest length deviation (+4.7 words). This suggests the 4B model achieves optimal balance between accuracy and text generation fidelity. 3. Model Scaling Effects: Performance scaling is non-linear across different metrics. While weighted accuracy scales consistently with model size, the 4B model outperforms the 8B model in standard accuracy, character similarity, and length calibration. All models maintain high character similarity ( $>88\$ ), indicating robust text generation capabilities across different scales. 4. Deployment Considerations: The 4B model emerges as the optimal choice, achieving near-best weighted accuracy (83.64% vs. 83.93% for 8B) while excelling in text generation quality and computational efficiency. The 1.7B model, despite lower absolute performance, maintains meaningful weighted accuracy (75.15%) and reasonable text generation quality (88.07% character similarity) suitable for resource-constrained applications. #### 4.5.5 Implications Comparison with base models reveals the critical importance of fine-tuning for this task: all base Qwen3 models demonstrated substantially lower accuracy, generated significantly longer outputs that deviated from expected length targets, and consistently failed to conform to required JSON output structures for structured classification tasks. The evaluation results demonstrate the complementary insights provided by standard and weighted evaluation approaches for ordinal classification tasks, while revealing important patterns in text generation quality across model scales. The 4B model emerges as optimally calibrated, achieving competitive weighted accuracy (83.64%) while excelling in character similarity (90.01%) and length precision (-0.7 words). This multi-dimensional superiority suggests that intermediate-scale models may achieve better overall performance than larger alternatives, possibly due to reduced over-parameterization effects. The 1.7B model’s 75.15% weighted accuracy alongside 88.07% character similarity demonstrates that even lightweight models can achieve meaningful performance for resource-constrained deployments where understanding ordinal error patterns and maintaining text generation quality are crucial for practical utility. ## 5 Discussion ### 5.1 Novel Contributions This work advances the field of automated citation verification through several key innovations. First, we introduce the first full-text citation verification system that moves beyond abstract-level analysis to examine complete source documents, addressing a critical limitation in existing approaches [wadden2020fact]. Our hybrid retrieval architecture, combining dense semantic search with sparse BM25 matching and neural reranking, demonstrates superior evidence selection compared to individual methods, particularly important for academic texts with specialised terminology. The 4-class classification taxonomy provides nuanced assessment beyond traditional binary schemes. Unlike SciFact’s SUPPORTS/REFUTES framework [wadden2020fact], our taxonomy captures intermediate states requiring different remedial actions. The inclusion of confidence scoring and actionable user guidance transforms abstract classifications into concrete editorial recommendations, bridging automated analysis with human decision-making. We introduce evidence-based transparency through detailed reasoning explanations and ranked evidence snippets that enable users to understand and verify system decisions. This addresses a critical gap in existing citation verification tools that provide classifications without supporting evidence, making our system interpretable and trustworthy for academic applications where understanding the basis for verification decisions is essential. Our framework incorporates confidence calibration and uncertainty quantification, enabling practical deployment scenarios where low-confidence predictions can be flagged for human review. This bridges the gap between fully automated processing and human oversight, addressing real-world deployment needs where system reliability must be transparent and actionable. Our demonstration that fine-tuned lightweight models achieve competitive performance with commercial LLMs whilst requiring significantly reduced computational resources addresses accessibility concerns in academic technology deployment. The Qwen3 4B model’s 90.01% character similarity and superior length calibration (-0.7 words deviation) demonstrate effective performance-efficiency trade-offs for practical deployment. To maximise research impact and enable widespread adoption, we commit to open-sourcing both the fine-tuned models and the complete SemanticCite framework. This open-source approach will facilitate reproducible research, enable customisation for domain-specific applications, and reduce barriers to implementation across diverse institutional contexts. The modular architecture supports easy integration with existing research workflows and citation management systems. ### 5.2 Limitations and Future Work Several limitations warrant acknowledgement and suggest directions for future research. The evaluation framework relies exclusively on OpenAI GPT-4.1 ground truth annotations, potentially introducing systematic biases inherent in commercial language models. Future work should incorporate human expert annotations and inter-annotator agreement studies to validate automated ground truth generation and assess potential model-specific biases. The current dataset, whilst comprehensive across eight academic disciplines, remains limited to English-language publications and open-access materials. Expanding to multilingual contexts and proprietary publications would enhance generalisability. Additionally, the temporal focus on 2019-2023 publications may not capture citation patterns across different research paradigms or historical periods. The framework currently processes multi-reference citations by evaluating each referenced source individually rather than assessing their collective evidential support. This approach can result in citations being classified as PARTIALLY SUPPORTED when individual references each provide partial evidence, even though their combined evidence may constitute full support for the assertion. Future development should implement advanced reasoning mechanisms that evaluate the collective strength of evidence across multiple references, incorporating logical frameworks to assess whether partial support from individual sources aggregates to comprehensive claim validation. Such systems would require sophisticated evidence synthesis algorithms capable of identifying complementary, reinforcing, and contradictory information patterns across reference sets to provide more nuanced and accurate verification assessments. A critical limitation affecting system accuracy is the availability of full-text reference documents for verification. While the framework implements fallback mechanisms to use reference abstracts when full texts are unavailable, this significantly constrains verification depth and accuracy. Many academic publications remain behind paywalls or lack open-access versions, forcing the system to rely on incomplete abstract-level information that may miss crucial supporting or contradictory evidence present in complete documents. This limitation represents the primary impediment to achieving higher accuracy rates, as comprehensive citation verification inherently requires access to the complete source material being referenced. The framework’s applicability extends beyond academic citation verification to broader fact-checking applications for AI-generated content. The same 4-class taxonomy (SUPPORTED, PARTIALLY SUPPORTED, UNSUPPORTED, UNCERTAIN) can be directly applied to verify claims in AI-generated reports, articles, and summaries against their purported sources. This extension addresses the critical challenge of AI hallucinations in content generation, providing systematic verification capabilities for automated content production systems. Future development should focus on AI-assisted improvement suggestions that go beyond classification to provide specific, actionable recommendations for citation enhancement. Such systems could suggest alternative phrasings, identify missing contextual information, recommend additional supporting sources, and propose precise modifications to transform PARTIALLY SUPPORTED citations into SUPPORTED ones. This evolution from detection to correction would create comprehensive AI-powered writing assistance tools. The system’s reliance on text-based analysis limits applicability to multimodal citations involving figures, tables, or mathematical expressions common in scientific literature. Extending the framework to incorporate visual and mathematical content represents a significant technical challenge with substantial potential impact. Deployment scalability remains a practical consideration. Whilst fine-tuned models reduce computational requirements compared to commercial APIs, processing large document corpora still requires substantial resources. Future work should explore efficient inference techniques and distributed processing architectures for institutional-scale implementation. ### 5.3 Implications for Research Quality and Broader Applications The demonstrated effectiveness of SemanticCite has broader implications for research quality and scholarly communication. Automated citation verification tools can significantly enhance peer review efficiency by identifying problematic citations before human review, allowing reviewers to focus on higher-level scientific assessment rather than time-consuming fact-checking. The system’s ability to detect subtle citation issues through the PARTIALLY SUPPORTED category addresses a critical gap in existing verification approaches. These cases often require minor corrections that substantially improve citation accuracy without major manuscript revisions. The framework advances contextual citation understanding by providing detailed reasoning and ranked evidence that illuminate the nuanced relationships between claims and their supporting sources. This enables researchers to better comprehend how individual citations fit within broader scholarly discourse, identify gaps in supporting evidence, and understand the strength of connections between claims and references across different academic contexts and disciplinary conventions. SemanticCite enables systematic retrospective analysis of published work, allowing institutions to assess citation quality in their research output and identify patterns that may require additional training or support. This capability supports institutional research integrity initiatives and provides objective metrics for evaluating citation practices across departments and disciplines. The framework’s direct applicability to AI-generated content verification represents a significant advancement in addressing widespread concerns about AI hallucinations. While recent improvements in models like GPT-5 have reduced hallucination rates to levels that ’seem acceptable to users’ for most cases, eliminating hallucinations entirely may prove impossible due to the fundamental statistical nature of how large language models operate [gibney2025hallucination]. As AI writing assistants become prevalent in content creation, the same classification taxonomy can verify claims in automatically generated reports, summaries, and articles against their sources, providing essential quality control for AI-powered publishing workflows. ## 6 Conclusion This paper introduces SemanticCite, an AI-driven framework for automated citation verification that addresses fundamental limitations in existing approaches. Our 4-class classification taxonomy enables nuanced assessment beyond traditional binary schemes, whilst the hybrid retrieval architecture supports effective full-text analysis for accurate evidence extraction. The demonstration that fine-tuned Qwen3 models achieve competitive performance with significantly reduced computational requirements establishes the feasibility of deploying sophisticated citation verification in resource-constrained environments. The optimal performance of the 4B model (83.64% weighted accuracy, 90.01% character similarity) demonstrates effective practical trade-offs for diverse deployment scenarios. The framework’s practical implications extend beyond academic manuscript review. By providing actionable guidance through detailed reasoning and evidence extraction, SemanticCite bridges the gap between automated analysis and human editorial decision-making, supporting iterative manuscript improvement rather than binary acceptance/rejection outcomes. The same 4-class taxonomy applies directly to AI-generated content verification, addressing critical quality control challenges in automated reports, summaries, and AI-powered publishing workflows. Future research directions include expanding multimodal capabilities, incorporating multilingual support, and developing AI-assisted improvement features that shift the system from detecting citation issues to actively supporting authors with specific recommendations for enhancement and source integration. These extensions will strengthen the framework’s applicability to fact-checking in automated content production systems. SemanticCite represents a significant step towards research quality tools that augment rather than replace human expertise, providing scalable, open-source solutions for maintaining citation accuracy in an era of rapidly expanding scientific output and AI-generated content.