# : Citation Verification with AI-Powered Full-Text Analysis and Evidence-Based Reasoning
**Authors**:
- Sebastian Haan (The University of Sydney, Australia)
Abstract
Effective scientific communication depends on accurate citations that validate sources and guide readers to supporting evidence. Yet academic literature faces mounting challenges: semantic citation errors that misrepresent sources, AI-generated hallucinated references, and traditional citation formats that point to entire papers without indicating which sections substantiate specific claims. We introduce SemanticCite, an AI-powered system that verifies citation accuracy through full-text source analysis while providing rich contextual information via detailed reasoning and relevant text snippets. Our approach combines multiple retrieval methods with a four-class classification system (Supported, Partially Supported, Unsupported, Uncertain) that captures nuanced claim-source relationships and enables appropriate remedial actions for different error types. Our experiments show that fine-tuned lightweight language models achieve performance comparable to large commercial systems with significantly lower computational requirements, making large-scale citation verification practically feasible. The system provides transparent, evidence-based explanations that support user understanding and trust. We contribute a comprehensive dataset of over 1,000 citations with detailed alignments, functional classifications, semantic annotations, and bibliometric metadata across eight disciplines, alongside fine-tuned models and the complete verification framework as open-source software. SemanticCite addresses critical challenges in research integrity through scalable citation verification, streamlined peer review, and quality control for AI-generated content, providing an open-source foundation for maintaining citation accuracy at scale. footnotetext: Email: sebastian.haan@sydney.edu.au footnotetext: Software Repository: https://github.com/sebhaan/semanticcite
<details>
<summary>x2.png Details</summary>
### Visual Description
Icon/Small Image (24x23)
</details>
Repository
<details>
<summary>hf_icon.png Details</summary>
### Visual Description
## Emoji: Hugging Face
### Overview
The image is of a "Hugging Face" emoji. It features a yellow face with a wide smile, closed eyes, and two hands reaching out as if to give a hug.
### Components/Axes
* **Face:** Yellow circular face.
* **Eyes:** Two small, closed, curved shapes.
* **Mouth:** Wide, black smile with a red tongue.
* **Hands:** Two yellow hands with fingers spread, positioned on either side of the face.
* **Outline:** A thin white outline surrounds the entire emoji.
### Detailed Analysis or ### Content Details
The emoji is designed to convey warmth, affection, and a sense of offering a hug. The yellow color is consistent with typical emoji design. The hands are positioned to suggest an embrace.
### Key Observations
The emoji is simple and easily recognizable. The expression is friendly and inviting.
### Interpretation
The "Hugging Face" emoji is used to express positive emotions such as love, support, gratitude, or excitement. It can be used in various contexts to convey a sense of closeness and connection.
</details>
Model & Data
Keywords: citation verification, semantic analysis, natural language processing, research integrity, automated fact-checking
1 Introduction
1.1 Problem Statement
Citations serve as fundamental channels through which scientific knowledge is shared and developed within the academic ecosystem. Despite their critical importance, scholarly literature suffers from a notable prevalence of citation errors that compromise research integrity and quality.
These errors manifest in several forms: citations supporting claims not made in referenced works, citations omitting critical qualifications present in sources, citations attributing findings to incorrect studies, and selective citation of results whilst ignoring contradictory evidence. Beyond these semantic errors, typographical mistakes in bibliographic information also occur, though digital citation management tools have reduced their frequency. Analysis of prestigious science journals reveals that semantic citation errors remain pervasive, with error rates of 25% on average [smith2020quotation].
The implications for research quality are profound. Incorrect citations inadvertently propagate erroneous ideas, contribute to the acceptance of scientific myths, and ultimately erode the credibility of academic scholarship [gotzsche2022citation]. Such inaccuracies can lead to unintended plagiarism, diminished academic standing, and misallocation of valuable research resources.
Manual citation verification during peer review faces substantial limitations. Reviewers are increasingly burdened by overwhelming workloads due to rapid expansion in academic publishing, making it difficult to dedicate sufficient time to meticulous citation evaluation. The lack of standardised review criteria and training contributes to inconsistent evaluations, whilst personal biases can subtly influence outcomes [smith2006peer, tennant2020limitations, mishra2025challenges, sizo2025defining].
A pressing new challenge has emerged with AI-generated content, where advanced language models produce convincing but non-existent “fake citations” that are extraordinarily difficult to detect manually at scale. Recent studies demonstrate that even advanced models like GPT-5 fabricate citations 39% of the time when operating without internet access, highlighting the fundamental nature of this problem [gibney2025hallucination]. These hallucinations arise from the statistical nature of large language models, which make predictions by generalising from learned associations, leading to plausible but sometimes incorrect outputs [gibney2025hallucination]. As AI-assisted writing becomes prevalent in academic contexts, traditional verification approaches designed for human authors prove insufficient for detecting AI-generated content that can produce highly plausible but entirely fabricated citations at scale.
Traditional citation presentation provides no mechanism for readers to quickly understand cited work context without manually retrieving and reading entire source documents. Citations merely point to entire papers without indicating specific supporting sections, forcing readers to interrupt their flow, locate sources, search through lengthy documents for relevant passages, and return to the original text. This lack of precise contextual guidance impedes efficient knowledge discovery and creates barriers to thorough fact-checking during both research and peer review processes.
Current automated citation tools remain severely limited across multiple dimensions. Reference management software (EndNote, Zotero, Mendeley) excel at bibliographic formatting but provide no semantic verification capabilities. Existing automated verification systems validate citations against authority databases (Web of Science, PubMed, CrossRef), detecting incorrect metadata or non-existent references, but cannot verify whether citation claims accurately reflect source document content. A reference may exist with correct metadata whilst the citing author mischaracterises the source’s actual findings. Similarly, format-checking systems verify style compliance but do not analyse semantic alignment between claims and sources.
Even systems attempting content-level verification typically rely on abstract-only analysis rather than examining complete source documents. This approach misses critical supporting evidence, methodological caveats, limitations discussions, and contradictory findings present in full papers. Important contextual information such as experimental conditions, population constraints, statistical significance details, and author-acknowledged limitations rarely appears in abstracts yet proves essential for accurate citation verification.
Furthermore, traditional verification approaches employ simple binary schemes (supported/unsupported) that fail to capture partial support, where core assertions align with sources but contextual elements are missing. These situations require different remedial actions than completely unsupported claims, yet binary systems cannot distinguish between them. Additionally, automated systems provide classifications without explaining reasoning or presenting supporting evidence, preventing independent validation.
These converging challenges create an urgent need for comprehensive citation verification systems. Pervasive semantic errors, limited verification tools, opaque automated systems, AI-generated hallucinations, and manual verification friction demand solutions that detect errors, explain reasoning, provide actionable guidance, and scale efficiently whilst maintaining the transparency essential for scholarly work.
1.2 Research Objectives
This work addresses these fundamental limitations through four primary objectives. First, we develop an automated full-text citation verification system that transcends surface-level bibliographic checks to perform deep semantic analysis of citation content against complete source documents. Second, we create a nuanced 4-class classification scheme (Supported, Partially Supported, Unsupported, Uncertain) that captures the complexity of citation-reference relationships beyond traditional binary approaches, enabling appropriate remedial actions for different types of citation errors. Third, we demonstrate that fine-tuned lightweight models can achieve competitive performance with large commercial language models whilst requiring significantly reduced computational resources. Fourth, we implement transparent evidence-based reasoning that enables users to understand and independently validate system decisions through relevant text snippets extracted from source documents and detailed explanations of classification rationale. This transparency is essential for maintaining the verifiability and trust required in academic contexts, addressing the opacity that has plagued previous automated verification systems.
1.3 Contributions
This paper introduces SemanticCite, the first comprehensive AI-powered framework for automated full-text citation verification. Unlike existing approaches that rely on abstract-level analysis or binary classification schemes, our system performs deep semantic analysis of complete source documents using a 4-class taxonomy that captures the nuanced relationships between citations and their references. SemanticCite bridges the gap between automated detection and actionable remediation by providing detailed evidence-based reasoning that enables authors and reviewers to understand and address citation inaccuracies systematically. This represents a fundamental advance from simple error detection to comprehensive citation quality assessment with practical guidance for manuscript improvement. By combining hybrid retrieval and reranking techniques with fine-tuned language models, SemanticCite enables thorough citation analysis while maintaining computational efficiency, making sophisticated verification accessible to individual researchers and institutions regardless of computational resources.
We contribute three key resources spanning data, models, and implementation. Our dataset comprises 1,111 analysed citations across 8 academic disciplines, including extracted claims, 4-class semantic classifications, reasoning explanations, confidence scores, and ranked evidence snippets. We release fine-tuned Qwen3 models (1.7B and 4B parameters) trained for both citation preprocessing and semantic classification tasks that achieve competitive performance with commercial systems while requiring significantly fewer computational resources. We open-source the SemanticCite framework code, including claim extraction pipelines, hybrid retrieval systems, and classification method.
2 Related Work
2.1 Citation Analysis and Verification
Traditional citation analysis has primarily focused on bibliometric approaches using citation counts as indicators of research impact, failing to capture semantic relationships between citing statements and referenced sources. The field of automated fact-checking has provided important foundations for citation verification. Wadden et al. (2020) [wadden2020fact] introduced scientific claim verification with their SciFact dataset comprising 1.4K expert-written claims paired with abstracts annotated for veracity, demonstrating that domain adaptation significantly improves performance versus general-purpose fact-checking models.
However, existing systems face significant limitations. Most rely on abstract-level analysis rather than full-text examination, potentially missing crucial evidence within complete documents. Binary classification schemes (SUPPORTS/REFUTES) fail to capture nuanced citation-reference relationships [wadden2020fact]. The emergence of AI-generated content has introduced new challenges, with advanced language models producing convincing but non-existent citations. Whilst tools like CiteSure [citesure2024] address this specific problem and systems like Morressier’s Integrity Manager detect various misconduct indicators, these specialised tools focus on specific error types rather than comprehensive citation verification [sae2024research].
Some existing platforms analyse citation usage within citing documents, extracting surrounding context and classifying whether the citing statement supports or contradicts the reference generally [nicholson2021scite]. However, these systems examine the directionality in reverse, i.e., they assess whether the citation supports the reference rather than whether the source document supports the specific citation claim. They cannot verify if the referenced source actually contains the evidence the citing author asserts, leaving the fundamental question of claim accuracy unaddressed.
2.2 Multi-class Citation Classification in NLP
The evolution from binary to multi-class classification reflects growing recognition that real-world relationships exist on spectra rather than simple dichotomies. Multi-class approaches offer significant advantages by capturing degrees of alignment rather than forcing binary decisions. Systems employing classifications such as “supported,” “partially supported,” “unsupported,” and “uncertain” provide granular assessments that better reflect scientific discourse complexity [haan2025developing].
The Jurgens et al. (2018) taxonomy [jurgens2018measuring] represents a notable contribution, defining six categories (Background, Extends, Uses, Motivation, Compare/Contrast, Future work) for citation intent classification using the ACL-ARC dataset of 1,941 expert-annotated instances. Subsequent research expanded available data with the SciCite dataset, over five times larger than ACL-ARC, simplifying taxonomy to three categories (Background, Method, Result) whilst achieving superior performance with pre-trained language models like XLNet (88.93% F1 score) [mercier2020impactcite].
The diversity of taxonomies across datasets indicates lack of universal agreement on citation function definitions. This fragmentation, coupled with recognition that citations may serve multiple purposes, suggests current single-label classification may oversimplify relationships, necessitating flexible frameworks supporting customisable taxonomies and multi-label classification [abu-jbara2013, lauscher2021multicite].
2.3 Retrieval-Augmented Generation
Retrieval-augmented generation (RAG) has emerged as a powerful paradigm for fact-checking applications, combining information retrieval with language model generation capabilities [lewis2020retrieval, gao2023retrieval]. In citation verification contexts, RAG enables systems to retrieve relevant evidence from source documents and generate informed assessments of citation accuracy. Traditional RAG approaches face limitations when applied to academic texts due to their complexity and specialised terminology [li2024budget].
Recent advances in reverse RAG methodologies show promise for citation verification by first extracting or summarising information, then linking each data point back to source documents through systematic fact verification. This approach ensures every claim is traceable to original sources, addressing hallucination problems in standard generative models [rajpurkar2024claimify, metropolitansky2025towards].
Hybrid retrieval approaches combining dense semantic search with sparse keyword matching have demonstrated superior performance over individual methods [arivazhagan2023, sawarkar2024blended, sultania2024domain]. Dense retrieval using vector embeddings captures semantic relationships, whilst sparse methods like BM25 (Robertson & Zaragoza, 2009) ensure exact term correspondence. Neural reranking using cross-encoders further optimises candidate ordering based on query-document relevance, providing robust foundations for evidence-based citation verification systems.
3 Methodology
3.1 System Architecture
SemanticCite employs a four-stage pipeline that combines hybrid retrieval-augmented generation with large language models for automated citation verification. The system operates through sequential stages: text preprocessing, hybrid retrieval, neural reranking, and LLM-based analysis, implemented using the LangChain framework with support for multiple providers. Figure 1 provides an overview of the complete citation verification pipeline, illustrating the flow from input processing through to structured output generation.
<details>
<summary>pipeline_diagram.png Details</summary>
### Visual Description
## Diagram: Semantic Citation Pipeline
### Overview
The image is a flowchart illustrating a semantic citation pipeline. It outlines the steps involved in processing citation text and reference documents to produce structured output, including classification, evidence, reasoning, and metadata. The pipeline consists of several stages: Input, Processing, Embedding, Hybrid Retrieval, Reranking, LLM Analysis, and Output.
### Components/Axes
The diagram consists of rectangular boxes representing different stages of the pipeline, connected by arrows indicating the flow of information. Each box contains a title and a brief description of the processes involved. The boxes are color-coded to visually distinguish the different stages.
* **INPUT** (Light Blue): Contains "Citation Text", "Reference Document", and "Metadata (optional)".
* **PROCESSING** (Yellow): Contains "Document Processing" (PDF extraction, Text chunking) and "Citation Processing" (Remove references, Extract core claim).
* **EMBEDDING** (Light Red): Contains "Vector Store" and "Embeddings".
* **HYBRID RETRIEVAL** (Light Blue): Contains "Dense Retrieval" (Vector similarity) and "Sparse Retrieval" (BM25 keyword).
* **RERANKING** (Purple): Contains "Neural Reranking" and "FlashRank".
* **LLM ANALYSIS** (Light Green): Contains "Citation Verification" (Citation vs Evidence, Metadata context, Structured output).
* **OUTPUT** (Light Gray): Contains "Classification" (4 categories + confidence), "Evidence" (Top 3 chunks + scores), "Reasoning" (Detailed explanation), and "Metadata" (Timestamps + metrics).
### Detailed Analysis or ### Content Details
1. **INPUT:**
* Citation Text: The text of the citation being analyzed.
* Reference Document: The document being cited.
* Metadata (optional): Additional information about the citation or document.
2. **PROCESSING:**
* Document Processing:
* PDF extraction: Extracting text from PDF documents.
* Text chunking: Dividing the text into smaller segments.
* Citation Processing:
* Remove references: Removing references from the citation text.
* Extract core claim: Identifying the main assertion made by the citation.
3. **EMBEDDING:**
* Vector Store: A database for storing vector embeddings.
* Embeddings: Numerical representations of the text.
4. **HYBRID RETRIEVAL:**
* Dense Retrieval:
* Vector similarity: Finding similar vectors.
* Sparse Retrieval:
* BM25 keyword: Using BM25 to find relevant keywords.
5. **RERANKING:**
* Neural Reranking:
* FlashRank: A neural reranking model.
6. **LLM ANALYSIS:**
* Citation Verification:
* Citation vs Evidence: Comparing the citation to the evidence.
* Metadata context: Using metadata to provide context.
* Structured output: Generating structured output.
7. **OUTPUT:**
* Classification:
* 4 categories + confidence: Classifying the citation into one of four categories, along with a confidence score.
* Evidence:
* Top 3 chunks + scores: Providing the top three chunks of evidence, along with their scores.
* Reasoning:
* Detailed explanation: Providing a detailed explanation of the reasoning behind the classification.
* Metadata:
* Timestamps + metrics: Providing timestamps and other metrics.
### Key Observations
* The pipeline starts with inputting citation text and reference documents.
* The processing stage involves document and citation processing.
* Embedding creates vector representations of the text.
* Hybrid retrieval combines dense and sparse retrieval methods.
* Reranking uses neural models to improve the ranking of results.
* LLM analysis verifies the citation and generates structured output.
* The output includes classification, evidence, reasoning, and metadata.
### Interpretation
The diagram illustrates a comprehensive pipeline for semantic citation analysis. It combines various techniques, including document processing, embedding, retrieval, reranking, and LLM analysis, to extract meaningful information from citations and generate structured output. The pipeline aims to provide a more nuanced understanding of citations by considering both the text and the context in which they appear. The use of hybrid retrieval and neural reranking suggests an effort to improve the accuracy and relevance of the results. The final output, including classification, evidence, reasoning, and metadata, provides a rich set of information that can be used for various purposes, such as citation verification, knowledge discovery, and information retrieval.
</details>
Figure 1: Semantic Citation Verification Pipeline: A multi-stage automated system for citation verification combining document processing (PDF extraction and text chunking), vector embedding storage, hybrid retrieval using both dense semantic similarity and sparse BM25 keyword matching, neural reranking with FlashRank, and LLM-based analysis. The pipeline outputs a classification result, supporting evidence, detailed reasoning, and confidence score for each citation verification task.
The citation verification system takes three inputs: citation text, reference document, and optional metadata. Then it produces a classification label, confidence score, supporting evidence, and detailed reasoning. The system processes reference documents through text extraction and chunking, whilst citations undergo preprocessing to remove attribution markers and extract core factual claims.
Document processing begins with text extraction using PyMuPDF PyMuPDF Documentation: https://github.com/pymupdf/PyMuPDF for PDF files, followed by recursive text splitting into 512-character chunks with 50-character overlap. The chunking strategy employs hierarchical separators including paragraph breaks, sentences, punctuation, and whitespace to maintain semantic coherence whilst ensuring manageable sizes for retrieval operations.
Citation preprocessing transforms raw citations into standardised claims by removing reference markers, author attributions, and publication identifiers whilst preserving numerical values and factual assertions. This transformation converts author-centric statements to fact-centric assertions suitable for independent verification. For example, “Smith et al. (2020) found that exercise reduces cardiovascular risk by 30%” becomes “Exercise reduces cardiovascular risk by 30%”.
The system supports flexible configuration of LLM providers including OpenAI, Anthropic Claude, Google Gemini, and local models via LiteLLM LiteLLM Documentation: https://docs.litellm.ai/docs/. Similarly, embedding models can be configured across OpenAI, local SentenceTransformers, or custom endpoints, enabling deployment across diverse computational environments whilst maintaining consistent performance. Temperature settings are controlled for reproducible outputs, with deterministic citation processing using zero temperature.
3.2 Classification Scheme
Our 4-class taxonomy represents a fundamental contribution to citation verification methodology, providing nuanced assessment beyond traditional binary approaches. The classification scheme follows a logical progression from fully supported to unsupported citations, with an uncertainty class capturing ambiguous cases where clear determination cannot be made. Figure 2 illustrates the complete classification framework with actionable guidance for each category.
The development of this taxonomy addresses critical limitations in existing binary classification systems. Traditional SUPPORTS/REFUTES schemes, whilst computationally simpler, fail to capture the complexity of citation-reference relationships commonly encountered in academic literature. Many citations exist in intermediate states requiring different remedial actions, making binary classification insufficient for practical citation verification applications.
<details>
<summary>classification_scheme_diagram.png Details</summary>
### Visual Description
## Diagram: Classification Scheme
### Overview
The image presents a classification scheme, categorizing information or claims into four distinct categories: Supported, Partially Supported, Unsupported, and Uncertain. Each category is defined by a set of indicators and suggested actions. The diagram uses a top-down flow, with each category box leading to an "Actions" box via a downward-pointing arrow.
### Components/Axes
* **Title:** Classification Scheme (located at the top-center of the image)
* **Categories (Top Row):**
* **SUPPORTED (Fully Aligned):** Green box
* **PARTIALLY SUPPORTED (Partially Aligned):** Yellow box
* **UNSUPPORTED (Misaligned):** Red box
* **UNCERTAIN (Indeterminate):** Gray box
* **Indicators (Within each category box):** A bulleted list of indicators that define the category.
* **Actions (Bottom Row):** A box below each category box, containing a bulleted list of suggested actions.
* **Arrows:** Downward-pointing arrows connecting each category box to its corresponding "Actions" box.
### Detailed Analysis or ### Content Details
**1. SUPPORTED (Fully Aligned) - Green Box (Top-Left)**
* **Indicators:**
* Complete alignment
* Proper contextual usage
* Comprehensive representation
* **Actions:**
* No changes needed
* Consider adding context
* Citation is accurate
**2. PARTIALLY SUPPORTED (Partially Aligned) - Yellow Box (Top-Middle-Left)**
* **Indicators:**
* Core claim supported
* Missing nuances/context
* Simplified representation
* **Actions:**
* Review evidence snippets
* Revise to include key details
* Clarify/expand citation
**3. UNSUPPORTED (Misaligned) - Red Box (Top-Middle-Right)**
* **Indicators:**
* Claim absent from source
* Contradiction with source
* Misrepresentation
* **Actions:**
* Remove/replace claim
* Search alternative references
* Review reasoning provided
**4. UNCERTAIN (Indeterminate) - Gray Box (Top-Right)**
* **Indicators:**
* Ambiguous content
* Insufficient information
* Inconclusive relationship
* **Actions:**
* Review evidence carefully
* Revise for clarity
* Consider removing citation
### Key Observations
* The classification scheme provides a structured approach to evaluating the validity and reliability of information or claims.
* Each category has specific indicators that help in determining its classification.
* The suggested actions provide guidance on how to address issues identified in each category.
* The color-coding (green, yellow, red, gray) visually represents the degree of support or certainty, with green indicating the highest level of support and red indicating the lowest.
### Interpretation
The classification scheme is designed to aid in the critical evaluation of information. It provides a framework for assessing the alignment, context, and representation of claims, and suggests appropriate actions based on the classification. The scheme highlights the importance of verifying sources, addressing ambiguities, and ensuring accurate citations. The progression from "Supported" to "Uncertain" suggests a decreasing level of confidence in the validity of the information, prompting increasingly cautious and thorough review processes.
</details>
Figure 2: Four-Category Classification Scheme for Source-Claim Alignment Assessment
3.2.1 Detailed Classification Categories
SUPPORTED (Fully Aligned): Citations demonstrate complete alignment between claims and source material, maintaining proper context and comprehensive representation. These citations accurately convey the referenced content without oversimplification, missing context, or distortion of meaning. The core indicators include complete factual alignment, appropriate contextual usage, and comprehensive representation of source nuances. For instance, a citation stating “The human genome contains approximately 20,000 protein-coding genes” fully aligns with source material stating “Recent genomic studies have confirmed that the human genome encodes about 20,000 protein-coding genes.” Such citations receive high confidence scores and require no modifications, though users may optionally add additional context from provided evidence snippets.
PARTIALLY SUPPORTED (Partially Aligned): Core claims are present in source material but lack critical nuances, contextual information, or important qualifications, leading to incomplete representation. These citations contain accurate fundamental assertions but oversimplify complex findings or omit essential caveats present in the source. Key indicators include core claim support with missing minor nuances, simplified representation, or incomplete contextual information. An example would be citing “Exercise guarantees a 50% reduction in cardiovascular disease risk” when the source states “Regular exercise has been shown to reduce cardiovascular disease risk by up to 50% in some studies, though results can vary.” The citation captures the core finding but overstates certainty and omits variability. Such citations receive medium confidence scores and require minor revisions to include qualifying details and expand scope for complete accuracy.
UNSUPPORTED (Misaligned): Claims are absent from, contradicted by, or significantly misrepresent source material. These citations demonstrate fundamental disconnection between stated claims and referenced content, ranging from complete absence of supporting evidence to direct contradiction of source findings. Critical indicators include claims absent from source material, direct contradiction between claims and evidence, or significant misrepresentation of source content. For example, claiming “Drinking coffee cures cancer” when the source states “Some studies suggest compounds in coffee may lower risk of certain cancers, but evidence is not conclusive” represents serious misrepresentation. Such citations receive low confidence scores and require major revision, removal, or alternative reference identification.
UNCERTAIN (Indeterminate): Ambiguous relationships exist where insufficient evidence, unclear claim-reference relationships, or inadequate verification context prevent definitive classification. These cases often involve vague claims, complex source material with multiple interpretations, or insufficient contextual information for reliable assessment. Key indicators include ambiguous claim or source content, insufficient contextual information, or inconclusive claim-reference relationships. An example might be citing “Vitamin D supplementation prevents all cases of osteoporosis” when the source discusses “varying degrees of effectiveness depending on factors such as age, dosage, and baseline deficiency levels.” The broad generalisation in the claim combined with nuanced source findings creates ambiguity requiring additional clarification. Such citations warrant careful review of evidence snippets to identify ambiguity sources.
3.2.2 Actionable User Guidance Framework
The classification scheme provides specific, actionable guidance aligned with potential user responses:
- SUPPORTED citations receive confirmation that no changes are needed, with optional suggestions for additional context enhancement.
- PARTIALLY SUPPORTED citations receive specific guidance on reviewing evidence snippets to identify missing nuances and revising content to include qualifying details.
- UNSUPPORTED citations prompt users to remove inaccurate claims, replace them with evidence-based statements, or seek alternative supporting references.
- UNCERTAIN classifications guide users to carefully review evidence snippets, revise citations to include necessary contextual details, or consider citation removal if relationships remain unclear.
This structured guidance framework transforms abstract classification labels into concrete editorial actions, bridging the gap between automated analysis and practical manuscript improvement. The approach recognises that different classification outcomes require different remedial strategies, moving beyond simple accuracy scores to provide constructive improvement pathways.
3.2.3 Advantages Over Binary Classification
The multi-class approach offers several critical advantages over binary schemes. Binary classification forces complex citation-reference relationships into oversimplified dichotomies, failing to distinguish between minor revisions and major corrections. Our taxonomy captures nuanced relationships requiring different remedial actions, providing more precise feedback for authors and reviewers.
Furthermore, the inclusion of an UNCERTAIN category acknowledges inherent limitations in automated analysis, providing transparent handling of ambiguous cases rather than forcing potentially incorrect binary decisions. This approach enhances system trustworthiness by explicitly flagging cases requiring human judgment whilst maintaining high confidence in clear-cut classifications.
The taxonomy also aligns with real-world editorial workflows where citations often require incremental improvements rather than complete acceptance or rejection. By providing graduated classification levels, the system supports iterative manuscript refinement and targeted quality improvement efforts.
3.3 Hybrid Retrieval System
The retrieval architecture combines dense semantic search with sparse keyword matching to maximise evidence recall whilst maintaining precision. This hybrid approach addresses individual method limitations: dense retrieval captures semantic relationships but may miss exact term matches, whilst sparse retrieval ensures keyword correspondence but lacks semantic understanding.
Dense Retrieval employs vector embeddings stored in ChromaDB using cosine similarity matching. Document chunks are mapped into high-dimensional vector spaces where semantically similar content clusters together. When processing citation claims, the system calculates similarity scores between query vectors and stored document vectors, retrieving the 15 most semantically relevant chunks. This approach excels at capturing conceptual relationships and paraphrased content that might use different terminology whilst conveying similar meaning.
Sparse Retrieval utilises BM25 algorithm for exact term matching, ensuring that citations requiring specific keyword correspondence are properly identified. BM25 considers term frequency, document length normalisation, and inverse document frequency to rank passages based on lexical overlap with citation claims. This method retrieves an additional 15 chunks focusing on exact term matches, complementing semantic retrieval by capturing precise terminology and specific numerical values.
Neural Reranking applies FlashRank cross-encoder optimisation to the combined retrieval results. After deduplication, the hybrid results undergo reranking based on query-document relevance scores computed by BERT-based cross-encoders. This process optimises candidate ordering and selects the top 3 most relevant passages for downstream analysis, ensuring focus on the most pertinent evidence whilst maintaining computational efficiency.
Document Chunking Strategy employs recursive character splitting with carefully chosen parameters to balance granularity with context preservation. The 512-character chunk size with 50-character overlap ensures sufficient context for semantic understanding whilst maintaining computational tractability. Hierarchical separators prioritise natural text boundaries, starting with paragraph breaks and progressively falling back to sentences, punctuation, and finally individual words when necessary.
The hybrid approach demonstrates superior performance compared to individual methods by leveraging complementary strengths. Dense retrieval captures nuanced semantic relationships crucial for academic content, whilst sparse retrieval ensures critical exact matches are not missed. Neural reranking provides final optimisation, resulting in high-quality evidence selection for citation verification.
3.4 Citation Processing Pipeline
The processing pipeline transforms raw academic citations into verifiable claims whilst preserving essential factual content. This preprocessing stage is critical for accurate verification as raw citations often contain attribution elements that obscure core factual assertions.
Preprocessing Operations systematically transform citations through several key steps. Reference markers such as brackets, parentheses, and citation numbers are removed to isolate factual content from bibliographic formatting. Author attributions are eliminated whilst preserving substantive claims, converting statements like “Smith et al. found that X” to “X was found” using passive voice constructions. Numerical values, statistical measures, and specific metrics are carefully preserved as these often represent the most verifiable aspects of citations. The process ensures all claims become independently verifiable statements that can be assessed against source material without requiring external attribution context.
Structured Output Parsing employs LLM-based analysis with explicit schema requirements to ensure consistent and comprehensive results. The system generates structured JSON outputs containing classification labels, detailed reasoning with both summary and point-by-point analysis, confidence scores, and ranked evidence snippets with relevance scores and source locations. This structured approach enables programmatic processing whilst maintaining human interpretability.
The verification prompt carefully incorporates processed claims, top-ranked evidence snippets with relevance scores, and optional metadata for enhanced contextual understanding. Prompts explicitly instruct models to assess factual alignment between citations and source material whilst considering broader study context when metadata is available. Instructions emphasise objective evaluation based on textual evidence rather than general knowledge or assumptions.
Evidence Snippet Ranking applies systematic relevance scoring to prioritise the most pertinent supporting material. Each evidence snippet receives numerical relevance scores indicating how well it addresses the citation claim, enabling users to focus on the most important supporting or contradicting evidence. The system tracks comprehensive metadata including processing timestamps, reference file identifiers, chunk locations, and system version information for complete result provenance and reproducibility analysis.
This systematic approach enables transparent verification where users understand classification rationale through extracted evidence and detailed reasoning, supporting both automated analysis and human review processes. The pipeline’s modular design allows for component-level optimisation and adaptation to different document types and verification requirements whilst maintaining consistent output quality.
3.5 Web Interface and Deployment
<details>
<summary>User_Interface.png Details</summary>
### Visual Description
## UI Screenshot: SemanticCite Configuration and Input Form
### Overview
The image is a screenshot of a user interface for a tool called SemanticCite, which is designed for AI-powered semantic citation verification. The UI includes configuration options for the underlying language models, input fields for citation text and reference documents, and a section for optional reference metadata.
### Components/Axes
**Left Panel: Model Configurations**
* **Model Configurations:** A heading for the left panel.
* **LLM Configuration:** A sub-heading.
* **LLM Provider:** Label for a dropdown menu.
* **local:** The selected option in the dropdown.
* **Dual Model Configuration:** A sub-heading.
* **Preprocessing Model:** Label for a text input field.
* **ollama/SemanticCite-Refiner-Qwen3-11:** The value in the text input field.
* **Classification Model:** Label for a text input field.
* **ollama/SemanticCite-Checker-Qwen3-4:** The value in the text input field.
* **Model endpoint URL:** Label for a text input field.
* **http://localhost:11434:** The value in the text input field.
* **Embedding Configuration:** A sub-heading.
* **Embedding Provider:** Label for a dropdown menu.
* **local:** The selected option in the dropdown.
* **Embedding Model Name:** Label for a text input field.
* **Qwen/Qwen3-Embedding-0.6B:** The value in the text input field.
* **Disclaimers:** A sub-heading.
* **Privacy: All processing occurs locally:** A statement about privacy.
* **AI-assisted analysis - verify results independently:** A statement about the analysis process.
**Right Panel: SemanticCite Input Form**
* **Header:**
* **SemanticCite Logo:** A logo with the text "SemanticCite" and an AI icon.
* **AI-powered semantic citation verification:** A title.
* **Analyse whether your citations are accurately supported by reference documents.:** A description of the tool's purpose.
* **Citation Text:** A section for entering citation text.
* **Citation Text:** Section title.
* **Enter citation text:** Placeholder text in the input field.
* **Reference Document:** A section for uploading or linking to a reference document.
* **Reference Document:** Section title.
* **Upload File:** A link to upload a file.
* **Download from URL:** A link to download from a URL.
* **Upload reference document:** Label above the drag and drop area.
* **Drag and drop file here:** Text inside the drag and drop area.
* **Limit 200MB per file • TXT, PDF, MD:** File size and format limitations.
* **Browse files:** Button to browse files.
* **Reference Metadata (Optional):** A section for entering optional reference metadata.
* **Reference Metadata (Optional):** Section title.
* **Enter reference metadata:** Placeholder text in the input field.
* **Optional: Enter title, abstract, and other reference metadata... Supports plain text and markdown formatting.:** Description of the input field.
* **Check Citation:** A button to initiate the citation check.
### Detailed Analysis or ### Content Details
The UI is divided into two main sections: configuration and input. The configuration section on the left allows the user to select the language models and embedding models to be used for the citation verification process. The input section on the right allows the user to enter the citation text, upload a reference document, and enter optional reference metadata.
The configuration options include:
* LLM Provider: Set to "local".
* Preprocessing Model: Set to "ollama/SemanticCite-Refiner-Qwen3-11".
* Classification Model: Set to "ollama/SemanticCite-Checker-Qwen3-4".
* Model endpoint URL: Set to "http://localhost:11434".
* Embedding Provider: Set to "local".
* Embedding Model Name: Set to "Qwen/Qwen3-Embedding-0.6B".
The input fields include:
* Citation Text: An empty text field for entering the citation text.
* Reference Document: A file upload area with a drag-and-drop feature and a "Browse files" button. The file size limit is 200MB, and the supported file formats are TXT, PDF, and MD.
* Reference Metadata (Optional): An empty text field for entering optional reference metadata.
### Key Observations
* The UI is designed to be user-friendly, with clear labels and instructions.
* The configuration options allow the user to customize the citation verification process.
* The input fields are designed to accommodate different types of input, including text, files, and metadata.
* The privacy disclaimer indicates that all processing occurs locally.
### Interpretation
The SemanticCite UI provides a comprehensive set of tools for verifying citations using AI. The configuration options allow the user to fine-tune the verification process, while the input fields allow the user to provide the necessary information for the verification to be performed. The privacy disclaimer suggests that the tool is designed to be used in a secure environment. The tool appears to be designed for researchers, writers, and anyone who needs to ensure the accuracy of their citations.
</details>
Figure 3: SemanticCite web interface showing citation input, reference document upload options (file upload or URL download), optional metadata entry, and configurable model parameters. The interface supports multiple LLM providers and embedding models, enabling flexible deployment across different resource constraints and institutional requirements.
To facilitate widespread adoption and ensure accessibility for researchers without programming expertise, we developed a web-based interface using Streamlit (Figure 3). The interface provides intuitive controls for citation verification, allowing users to input citation text, upload reference documents (PDF, TXT, or Markdown formats), or download references directly from URLs. Users can configure both LLM providers (OpenAI, Anthropic Claude, Google Gemini, or local Ollama models) and embedding models (local SentenceTransformers, OpenAI embeddings, or custom endpoints) through an interactive sidebar, enabling flexible deployment across different institutional resource constraints and privacy requirements. The system displays results including classification labels with color coding, confidence scores, detailed reasoning explanations, and ranked evidence snippets extracted from the reference document. All analysis results can be exported to Markdown format for documentation and integration into research workflows. This implementation demonstrates the practical deployment of the SemanticCite framework whilst maintaining full analytical capabilities and transparency in the verification process.
4 Model Development and Evaluation
This section details the development and evaluation of fine-tuned large language models for automated citation verification within the SemanticCite framework. We present our complete experimental pipeline for training specialized citation analysis models, beginning with the systematic construction of a balanced, multi-disciplinary dataset spanning eight academic fields (§4.1). We describe our automated training data generation methodology that leverages LLM-based filtering and annotation to create high-quality datasets for both citation preprocessing and classification tasks (§4.2). The fine-tuning process employs QLoRA techniques with Qwen3 models [qwen2024] of varying scales to develop efficient, specialized systems for citation verification (§4.3). We introduce a weighted evaluation framework specifically designed to assess the ordinal nature of citation support categories (§4.4), and conclude with comprehensive experimental results demonstrating model performance across different scales and computational requirements (§4.5). This experimental framework establishes the effectiveness of fine-tuned language models for citation verification while providing insights into the trade-offs between model complexity, computational efficiency, and task-specific performance in academic text analysis.
4.1 Data Selection
We constructed a high-quality training dataset using a multi-stage approach designed to ensure representative coverage across academic disciplines while maintaining rigorous quality standards for bibliometric metadata and full-text content. The resulting corpus targets balanced representation across citation impact levels, disciplinary boundaries, and functional citation categories, providing a robust empirical foundation for training and evaluating citation analysis models. An overview of the data selection process is presented in Figure 4.
<details>
<summary>data_selection_diagram.png Details</summary>
### Visual Description
## Flowchart: Citation Analysis Workflow
### Overview
The image is a flowchart outlining a multi-stage process for citation analysis, starting from paper selection and metadata acquisition, proceeding through full-text acquisition and citation classification/selection, and culminating in reference document acquisition and the creation of a balanced training dataset.
### Components/Axes
* **Stages:** The flowchart is divided into three main stages, each represented by a rounded rectangle.
* **Flow Direction:** Arrows indicate the flow of the process from one stage to the next.
* **Functional Citation Classification:** A box detailing LLM-based functional categorization.
* **Citation Selection:** A box detailing two-stage filtering and prioritization.
* **Balanced Training Dataset:** A box summarizing the dataset characteristics.
### Detailed Analysis
**Stage 1: Stratified Paper Selection & Metadata Acquisition**
* Description: Semantic Scholar Graph with quality filters, citation impact stratification, and metadata collection.
* Disciplines: Computer Science, Medicine, Chemistry, Biology, Materials Science, Physics, Geology, Psychology.
**Stage 2: Full-Text Acquisition**
* Description: Systematic PDF download with integrity verification and full-text extraction.
**Functional Citation Classification**
* Description: LLM-based functional categorization.
* Categories:
* BACKGROUND
* METHOD
* COMPARISON
* MOTIVATION
* EXTENSION
* FUTURE
**Citation Selection**
* Description: Two-stage filtering and prioritization.
* Criteria:
* Single-reference contexts
* Factual assertions
* Quantitative claims
* High verification potential
**Stage 3: Reference Document Acquisition**
* Description: Acquisition of cited reference documents for verification.
* Formats: Full-text, Abstracts, Multi-format processing.
**Balanced Training Dataset**
* Citations Analysed: 1,111
* Academic Fields: 8
* Citation Types: 6
* Numerical Citations: 27%
* Additional Information: Equal distribution across citation impact levels. Comprehensive metadata, citation claims, alignment classification, reasoning, and reference text.
### Key Observations
* The process is sequential, moving from broad paper selection to detailed reference document acquisition.
* LLM-based functional categorization is used for citation classification.
* Citation selection involves a two-stage filtering and prioritization process.
* The final dataset is balanced across citation impact levels.
### Interpretation
The flowchart describes a systematic approach to building a citation analysis dataset. The process begins with a broad selection of papers from Semantic Scholar, filtered by quality and citation impact. Full texts are then acquired and processed. Citations are classified using an LLM-based approach, categorized by function (e.g., background, method, comparison). A two-stage filtering process selects relevant citations based on criteria such as single-reference contexts and factual assertions. Finally, reference documents are acquired for verification, and a balanced training dataset is created. The dataset includes a large number of citations (1,111), spans multiple academic fields (8), and includes various citation types (6). The 27% numerical citations suggests a focus on quantitative research. The equal distribution across citation impact levels indicates an effort to avoid bias in the dataset.
</details>
Figure 4: Overview of the data selection and processing pipeline for semantic citation verification
4.1.1 Data Collection and Processing Pipeline
The training dataset was constructed through a systematic three-stage pipeline designed to collect academic papers with diverse citation characteristics across multiple disciplines, targeting a corpus of 4,000 papers with balanced representation across citation impact levels.
Stage 1: Stratified Paper Selection
The collection process employed a stratified sampling approach across eight academic fields (Computer Science, Medicine, Chemistry, Biology, Materials Science, Physics, Geology, and Psychology) using the Semantic Scholar Graph API. This selection ensures comprehensive coverage of distinct disciplinary citation practices and scholarly communication patterns. Papers were filtered according to stringent criteria including publication year (2019-2023), minimum citation count ( $≥ 5$ ), minimum reference count ( $≥ 10$ ), and open access availability. However, open-access designation did not guarantee that all papers were actually available for download. The temporal constraint ensures currency whilst providing sufficient time for citation accumulation, and the citation thresholds guarantee papers with meaningful scholarly impact and comprehensive reference networks.
To ensure balanced representation across citation impact levels, papers were categorised into three citation rate bins (Low, Mid, High) based on field-specific empirically-derived thresholds that account for citations-per-year rates rather than absolute counts. This normalisation approach addresses the substantial variation in citation velocity across academic disciplines. These thresholds varied considerably across disciplines, reflecting domain-specific citation practices—for instance, Medicine exhibited substantially higher thresholds (Mid: 14, High: 31 citations/year) compared to Geology (Mid: 4, High: 7 citations/year). This stratification ensures that the dataset captures papers across the full spectrum of scholarly impact within each discipline, avoiding bias towards highly-cited outliers whilst maintaining representation of influential work.
Stage 2: Metadata Acquisition
For each selected paper, extensive metadata was retrieved including bibliographic information, abstracts, author details, venue information, external identifiers, and complete citation and reference networks. This metadata collection provides essential context for understanding citation patterns and enables sophisticated analysis of citation behaviour across different paper characteristics. Quality assurance measures excluded papers lacking valid open access PDF URLs or failing PDF format validation, ensuring accessibility of full-text content essential for detailed citation analysis.
Stage 3: Full-Text Document Acquisition
The final stage involved systematic download of PDF documents using validated open access URLs. Each download was verified for successful completion and proper PDF format before inclusion in the dataset, with verification including file size validation, PDF structure integrity checks, and preliminary text extraction testing. This methodology yielded a balanced, high-quality corpus of approximately 500 papers per academic field, with equal distribution across citation impact categories. The approach ensures representative coverage across academic disciplines and citation patterns whilst maintaining access to both structured bibliometric data and full-text content, providing a robust foundation for training citation analysis models.
4.1.2 Citation Function Classification Methodology
Classification Framework
The citation classification system employs a systematic approach to categorise citation functions within academic texts, building upon the established taxonomy of Jurgens et al. (2018) [jurgens2018measuring]. The methodology implements a six-category schema designed to capture the primary functional roles of citations in scholarly discourse. This approach balances thorough coverage of citation purposes with tractable computational requirements for automated classification.
- BACKGROUND citations establish the general field, provide contextual information, or reference foundational knowledge essential for understanding the current work.
- METHOD citations reference specific algorithms, techniques, datasets, software, or experimental procedures employed or adapted in the current work, representing instrumental citations that directly inform methodological choices.
- RESULT_COMPARISON citations directly compare or contrast findings, results, or conclusions with those of the cited work, often appearing in results and discussion sections where authors evaluate their findings against existing evidence.
- MOTIVATION citations indicate that the cited work directly inspired the research problem or provided primary motivation for the current study, representing foundational influences on research direction.
- EXTENSION citations signal that the current work builds upon, extends, improves, or adapts methods or ideas from the cited work, indicating direct intellectual lineage.
- FUTURE_WORK citations mention prior work specifically in the context of suggested research directions or open problems, typically appearing in concluding sections where authors outline potential developments.
Dual-Dimensional Classification Approach
The classification system operates along two complementary dimensions to capture both functional and quantitative aspects of citation usage. The primary dimension assigns each citation context to one of the six functional categories through systematic analysis of the surrounding textual context, considering both immediate sentence-level context and broader paragraph-level discourse patterns. The secondary dimension employs binary classification to identify whether the citation context explicitly involves numerical or quantitative information—including percentages, specific values, metrics, performance scores, or statistical data—directly related to the claim or comparison involving the cited work. This dual-dimensional approach recognises that quantitative citations often require different verification strategies compared to qualitative references, as numerical claims demand precise fact-checking against source documents whilst qualitative citations may require more nuanced semantic analysis. The quantitative dimension proves particularly valuable for identifying citations that make specific empirical claims requiring direct verification against source data.
Computational Implementation
The classification process uses large language models (LLMs) with structured prompting to ensure consistency and reproducibility across the entire dataset. Citation contexts are processed in configurable batches to optimise computational efficiency whilst maintaining classification accuracy. Each classification decision is accompanied by a concise rationale explaining the categorical assignment, providing transparency and enabling quality assessment of the automated classification process. The methodology incorporates error handling and validation mechanisms, including verification of output format compliance, categorical validity checks, and fallback procedures for problematic cases.
4.1.3 Citation Selection Methodology
To generate high-quality training data for fine-tuning large language models on citation analysis tasks, we implemented a two-stage automated selection pipeline using LLM-based filtering and ranking designed to identify citation instances most suitable for verification analysis.
The first stage applied strict filtering criteria to identify valid citation instances, requiring that: (1) the context contained only a single reference rather than multiple citations, ensuring unambiguous attribution and avoiding complications from citation clustering; (2) the context represented an actual citation sentence rather than a bibliography entry, maintaining focus on in-text scholarly discourse; and (3) the citation text was properly embedded within the contextual passage, ensuring sufficient surrounding context for meaningful analysis.
From the filtered candidates, a second LLM stage selected the top 5 citations per paper that warranted verification analysis. The selection prioritised citations containing specific claims, numerical statements, factual assertions, and substantial content that could be meaningfully verified against their source references, rather than brief mentions or general acknowledgments. This prioritisation ensures that the resulting training data focuses on citations with sufficient semantic content to support meaningful verification tasks. Files with fewer than 3 citations were excluded to maintain dataset quality and ensure sufficient citation diversity within each source document. This systematic approach ensured the selection of citation instances most suitable for training models to assess citation-reference alignment and factual accuracy.
4.2 Training Data Generation
Training datasets are generated for two distinct model types addressing different aspects of citation analysis. The Citation Preprocessing Model transforms raw citation statements into cleaned claims through systematic processing, creating input-output pairs mapping original citations to their standardised, processed forms. This preprocessing removes reference markers, author attributions, and publication identifiers whilst preserving numerical values and factual assertions, converting author-centric statements to fact-centric representations suitable for verification.
The Classification Model generates training examples for citation support classification through a more complex pipeline involving: (1) preparing vector stores from reference documents using hybrid retrieval methods, (2) extracting semantically relevant text chunks through embedding-based similarity matching using Qwen3-Embedding-0.6B, and (3) generating structured outputs containing the four classification labels (SUPPORTED, PARTIALLY SUPPORTED, UNSUPPORTED, UNCERTAIN), detailed reasoning, and confidence scores.
Ground truth annotations were generated using GPT-4.1 (gpt-4.1-2025-04-14) with carefully designed prompts ensuring consistent and reliable classifications. The citation preprocessing employed prompts instructing the model to remove reference markers, convert author-centred statements to fact-centred assertions, maintain numerical values, and ensure claims stand alone as verifiable statements.
For classification analysis, GPT-4 was provided with processed citations, reference document metadata, and relevant text snippets, then instructed to classify citations using our 4-class taxonomy. The structured output schema required classification labels, reasoning dictionaries containing summary and detailed analysis, and numerical confidence ratings (0.0-1.0).
4.3 Model Fine-tuning
The fine-tuning methodology employed QLoRA (Quantised Low-Rank Adaptation) techniques with Qwen3 language models to develop specialised systems for citation verification.
Model Architecture and Quantisation
The system used Qwen3 models in three configurations (1.7B, 4B, and 8B parameters) with 4-bit quantisation via BitsAndBytes to enable efficient training on consumer hardware. The QLoRA configuration employed a rank (r) of 16 with an alpha value of 32, targeting attention projection layers (q_proj, k_proj, v_proj, o_proj) and feed-forward network components (gate_proj, up_proj, down_proj). Gradient checkpointing through Unsloth optimisation reduced memory consumption whilst maintaining training stability, enabling deployment across diverse computational environments.
Training Data Preparation and Configuration
Three distinct training datasets were prepared for different citation analysis tasks. The Preprocessing Dataset contained 1,111 examples for text simplification and paraphrasing tasks, with input-output pairs averaging 49.2 and 38.8 tokens respectively. The Classification Dataset comprised 1,111 examples for citation verification tasks, featuring significantly longer sequences (average 810.9 input tokens, 178.5 output tokens) due to inclusion of reference snippets and metadata.
The training protocol implemented a conservative approach with 2 epochs, a learning rate of $2Ă— 10^{-4}$ , and AdamW optimisation with 8-bit precision. Batch sizes were dynamically adjusted based on GPU memory (1-4 per device) with gradient accumulation steps (2-8) to maintain effective batch sizes. The system employed linear learning rate scheduling with 5% warmup ratio and 1% weight decay for regularisation.
Task-Specific Training Approaches
Text Preprocessing Models were trained with sequence lengths of 1024 tokens while the Citation Classification Models used an extended sequence lengths (4608 tokens). Structured Output Models generated JSON-formatted responses containing classification labels, detailed reasoning, and confidence scores (0.0-1.0), enabling programmatic integration and uncertainty quantification. Training data underwent systematic preprocessing whereby complex nested JSON structures were flattened into coherent text sequences, with task-specific instructions integrated using Alpaca formatting templates to create standardised prompt structures.
Models were exported in multiple formats including LoRA adapters, merged weights, and GGUF quantised versions for deployment flexibility. The system maintained reproducibility through fixed random seeds and deterministic training procedures, with data splitting following a 90:10 train-test partition using stratified sampling.
4.4 Evaluation Methodology
Our classification task involves predicting four categories with natural ordinal structure: SUPPORTED (3), PARTIALLY SUPPORTED (2), UNSUPPORTED (1), and UNCERTAIN (0). Unlike traditional multi-class evaluation that treats all misclassifications equally, we implement a weighted evaluation framework that penalizes errors proportionally to their semantic distance.
4.4.1 Weighted Evaluation Framework
We define a weight matrix where misclassification penalty equals the absolute difference between ordinal values: $W(i,j)=|v(i)-v(j)|$ . This ensures that extreme errors (e.g., SUPPORTED $→$ UNCERTAIN, weight = 3) receive higher penalties than adjacent-level misclassifications (e.g., SUPPORTED $→$ PARTIALLY SUPPORTED, weight = 1).
Weighted accuracy is calculated as:
$$
\text{ACC}_{\text{weighted}}=1-\frac{\sum_{i,j}W(i,j)\times\text{CM}(i,j)}{N\times\max(W)} \tag{1}
$$
where CM $(i,j)$ represents confusion matrix entries, $N$ is total predictions, and $\max(W)=3$ . We also calculate ordinal Mean Absolute Error (MAE) treating classes as ordered variables, providing intuitive 0-1 scale performance assessment.
Our evaluation combines standard metrics (accuracy, F1-scores, Cohen’s $\kappa$ ) with ordinal-aware measures (weighted accuracy, ordinal MAE) to capture both traditional classification performance and task-specific semantic alignment.
4.4.2 Text Generation Quality Metrics
Beyond classification accuracy, we assess text generation quality using character-level similarity and length calibration metrics. Character similarity is calculated using Jaccard similarity at the character level, measuring the overlap of unique characters between predictions and ground truth texts. Specifically, we convert both texts to lowercase, remove spaces, create sets of unique characters, and calculate Jaccard similarity as the intersection divided by the union of character sets. This metric captures lexical overlap and vocabulary consistency regardless of word order, providing insight into how well models preserve semantic content during text generation. Length difference measures the deviation in word count between predictions and ground truth, indicating model calibration in output verbosity.
4.5 Results
4.5.1 Model Performance Overview
We evaluated three fine-tuned Qwen3 models (1.7B, 4B, 8B parameters) using QLoRA techniques on a stratified test set of 112 examples. Models were assessed across both preprocessing and classification tasks, with preprocessing involving citation text cleaning and standardization, while classification represents the primary citation verification task.
4.5.2 Preprocessing Task Performance
Table 1: Preprocessing Performance Results
| Qwen3 1.7B Qwen3 4B Qwen3 8B | 112 112 112 | 112 (100%) 112 (100%) 112 (100%) | 93.32% 94.15% 94.30% | 26.7 words 26.7 words 26.4 words | 26.3 words 26.3 words 26.3 words | +0.4 words +0.4 words +0.1 words |
| --- | --- | --- | --- | --- | --- | --- |
All models achieved exceptional performance on the preprocessing task, demonstrating 100% valid prediction rates with high character similarity ( $>93\%$ ) and excellent length calibration. The preprocessing task involves cleaning citation text by removing reference markers, author attributions, and standardizing claims into fact-centric statements suitable for verification. The consistently high performance across all model sizes indicates that text preprocessing represents a simpler task compared to classification, where models can reliably learn pattern-based transformations. The 8B model achieved marginally superior character similarity (94.30%) and best length calibration (+0.1 words), though all models performed comparably, suggesting that even the 1.7B model is sufficient for preprocessing tasks in resource-constrained deployments.
4.5.3 Classification Task Performance
Table 2: Classification Performance Results
| Qwen3 1.7B Qwen3 4B Qwen3 8B | 50.00% 66.36% 66.07% | 75.15% 83.64% 83.93% | 33.50% 49.15% 50.13% | 46.61% 65.31% 65.73% | 88.07% 90.01% 89.33% | +4.7 words -0.7 words +3.6 words |
| --- | --- | --- | --- | --- | --- | --- |
4.5.4 Key Findings
1. Weighted vs. Standard Performance: The two evaluation approaches reveal different aspects of model performance. Standard accuracy shows clear scaling with model size (1.7B: 50.00%, 4B: 66.36%, 8B: 66.07%), while weighted accuracy demonstrates that all models achieve substantially higher performance when accounting for ordinal relationships (1.7B: 75.15%, 4B: 83.64%, 8B: 83.93%). This indicates that models predominantly make semantically reasonable “near-miss” errors rather than extreme misclassifications.
2. Text Generation Quality and Calibration: The 4B model achieves superior text generation performance with the highest character similarity (90.01%) and best length calibration (-0.7 words from ground truth). The 8B model shows slightly lower character similarity (89.33%) and moderate length deviation (+3.6 words), while the 1.7B model exhibits the lowest character similarity (88.07%) and largest length deviation (+4.7 words). This suggests the 4B model achieves optimal balance between accuracy and text generation fidelity.
3. Model Scaling Effects: Performance scaling is non-linear across different metrics. While weighted accuracy scales consistently with model size, the 4B model outperforms the 8B model in standard accuracy, character similarity, and length calibration. All models maintain high character similarity ( $>88\%$ ), indicating robust text generation capabilities across different scales.
4. Deployment Considerations: The 4B model emerges as the optimal choice, achieving near-best weighted accuracy (83.64% vs. 83.93% for 8B) while excelling in text generation quality and computational efficiency. The 1.7B model, despite lower absolute performance, maintains meaningful weighted accuracy (75.15%) and reasonable text generation quality (88.07% character similarity) suitable for resource-constrained applications.
4.5.5 Implications
Comparison with base models reveals the critical importance of fine-tuning for this task: all base Qwen3 models demonstrated substantially lower accuracy, generated significantly longer outputs that deviated from expected length targets, and consistently failed to conform to required JSON output structures for structured classification tasks. The evaluation results demonstrate the complementary insights provided by standard and weighted evaluation approaches for ordinal classification tasks, while revealing important patterns in text generation quality across model scales. The 4B model emerges as optimally calibrated, achieving competitive weighted accuracy (83.64%) while excelling in character similarity (90.01%) and length precision (-0.7 words). This multi-dimensional superiority suggests that intermediate-scale models may achieve better overall performance than larger alternatives, possibly due to reduced over-parameterization effects. The 1.7B model’s 75.15% weighted accuracy alongside 88.07% character similarity demonstrates that even lightweight models can achieve meaningful performance for resource-constrained deployments where understanding ordinal error patterns and maintaining text generation quality are crucial for practical utility.
5 Discussion
5.1 Novel Contributions
This work advances the field of automated citation verification through several key innovations. First, we introduce the first full-text citation verification system that moves beyond abstract-level analysis to examine complete source documents, addressing a critical limitation in existing approaches [wadden2020fact]. Our hybrid retrieval architecture, combining dense semantic search with sparse BM25 matching and neural reranking, demonstrates superior evidence selection compared to individual methods, particularly important for academic texts with specialised terminology.
The 4-class classification taxonomy provides nuanced assessment beyond traditional binary schemes. Unlike SciFact’s SUPPORTS/REFUTES framework [wadden2020fact], our taxonomy captures intermediate states requiring different remedial actions. The inclusion of confidence scoring and actionable user guidance transforms abstract classifications into concrete editorial recommendations, bridging automated analysis with human decision-making.
We introduce evidence-based transparency through detailed reasoning explanations and ranked evidence snippets that enable users to understand and verify system decisions. This addresses a critical gap in existing citation verification tools that provide classifications without supporting evidence, making our system interpretable and trustworthy for academic applications where understanding the basis for verification decisions is essential.
Our framework incorporates confidence calibration and uncertainty quantification, enabling practical deployment scenarios where low-confidence predictions can be flagged for human review. This bridges the gap between fully automated processing and human oversight, addressing real-world deployment needs where system reliability must be transparent and actionable.
Our demonstration that fine-tuned lightweight models achieve competitive performance with commercial LLMs whilst requiring significantly reduced computational resources addresses accessibility concerns in academic technology deployment. The Qwen3 4B model’s 90.01% character similarity and superior length calibration (-0.7 words deviation) demonstrate effective performance-efficiency trade-offs for practical deployment.
To maximise research impact and enable widespread adoption, we commit to open-sourcing both the fine-tuned models and the complete SemanticCite framework. This open-source approach will facilitate reproducible research, enable customisation for domain-specific applications, and reduce barriers to implementation across diverse institutional contexts. The modular architecture supports easy integration with existing research workflows and citation management systems.
5.2 Limitations and Future Work
Several limitations warrant acknowledgement and suggest directions for future research. The evaluation framework relies exclusively on OpenAI GPT-4.1 ground truth annotations, potentially introducing systematic biases inherent in commercial language models. Future work should incorporate human expert annotations and inter-annotator agreement studies to validate automated ground truth generation and assess potential model-specific biases.
The current dataset, whilst comprehensive across eight academic disciplines, remains limited to English-language publications and open-access materials. Expanding to multilingual contexts and proprietary publications would enhance generalisability. Additionally, the temporal focus on 2019-2023 publications may not capture citation patterns across different research paradigms or historical periods.
The framework currently processes multi-reference citations by evaluating each referenced source individually rather than assessing their collective evidential support. This approach can result in citations being classified as PARTIALLY SUPPORTED when individual references each provide partial evidence, even though their combined evidence may constitute full support for the assertion. Future development should implement advanced reasoning mechanisms that evaluate the collective strength of evidence across multiple references, incorporating logical frameworks to assess whether partial support from individual sources aggregates to comprehensive claim validation. Such systems would require sophisticated evidence synthesis algorithms capable of identifying complementary, reinforcing, and contradictory information patterns across reference sets to provide more nuanced and accurate verification assessments.
A critical limitation affecting system accuracy is the availability of full-text reference documents for verification. While the framework implements fallback mechanisms to use reference abstracts when full texts are unavailable, this significantly constrains verification depth and accuracy. Many academic publications remain behind paywalls or lack open-access versions, forcing the system to rely on incomplete abstract-level information that may miss crucial supporting or contradictory evidence present in complete documents. This limitation represents the primary impediment to achieving higher accuracy rates, as comprehensive citation verification inherently requires access to the complete source material being referenced.
The framework’s applicability extends beyond academic citation verification to broader fact-checking applications for AI-generated content. The same 4-class taxonomy (SUPPORTED, PARTIALLY SUPPORTED, UNSUPPORTED, UNCERTAIN) can be directly applied to verify claims in AI-generated reports, articles, and summaries against their purported sources. This extension addresses the critical challenge of AI hallucinations in content generation, providing systematic verification capabilities for automated content production systems.
Future development should focus on AI-assisted improvement suggestions that go beyond classification to provide specific, actionable recommendations for citation enhancement. Such systems could suggest alternative phrasings, identify missing contextual information, recommend additional supporting sources, and propose precise modifications to transform PARTIALLY SUPPORTED citations into SUPPORTED ones. This evolution from detection to correction would create comprehensive AI-powered writing assistance tools.
The system’s reliance on text-based analysis limits applicability to multimodal citations involving figures, tables, or mathematical expressions common in scientific literature. Extending the framework to incorporate visual and mathematical content represents a significant technical challenge with substantial potential impact.
Deployment scalability remains a practical consideration. Whilst fine-tuned models reduce computational requirements compared to commercial APIs, processing large document corpora still requires substantial resources. Future work should explore efficient inference techniques and distributed processing architectures for institutional-scale implementation.
5.3 Implications for Research Quality and Broader Applications
The demonstrated effectiveness of SemanticCite has broader implications for research quality and scholarly communication. Automated citation verification tools can significantly enhance peer review efficiency by identifying problematic citations before human review, allowing reviewers to focus on higher-level scientific assessment rather than time-consuming fact-checking.
The system’s ability to detect subtle citation issues through the PARTIALLY SUPPORTED category addresses a critical gap in existing verification approaches. These cases often require minor corrections that substantially improve citation accuracy without major manuscript revisions.
The framework advances contextual citation understanding by providing detailed reasoning and ranked evidence that illuminate the nuanced relationships between claims and their supporting sources. This enables researchers to better comprehend how individual citations fit within broader scholarly discourse, identify gaps in supporting evidence, and understand the strength of connections between claims and references across different academic contexts and disciplinary conventions.
SemanticCite enables systematic retrospective analysis of published work, allowing institutions to assess citation quality in their research output and identify patterns that may require additional training or support. This capability supports institutional research integrity initiatives and provides objective metrics for evaluating citation practices across departments and disciplines.
The framework’s direct applicability to AI-generated content verification represents a significant advancement in addressing widespread concerns about AI hallucinations. While recent improvements in models like GPT-5 have reduced hallucination rates to levels that ’seem acceptable to users’ for most cases, eliminating hallucinations entirely may prove impossible due to the fundamental statistical nature of how large language models operate [gibney2025hallucination]. As AI writing assistants become prevalent in content creation, the same classification taxonomy can verify claims in automatically generated reports, summaries, and articles against their sources, providing essential quality control for AI-powered publishing workflows.
6 Conclusion
This paper introduces SemanticCite, an AI-driven framework for automated citation verification that addresses fundamental limitations in existing approaches. Our 4-class classification taxonomy enables nuanced assessment beyond traditional binary schemes, whilst the hybrid retrieval architecture supports effective full-text analysis for accurate evidence extraction.
The demonstration that fine-tuned Qwen3 models achieve competitive performance with significantly reduced computational requirements establishes the feasibility of deploying sophisticated citation verification in resource-constrained environments. The optimal performance of the 4B model (83.64% weighted accuracy, 90.01% character similarity) demonstrates effective practical trade-offs for diverse deployment scenarios.
The framework’s practical implications extend beyond academic manuscript review. By providing actionable guidance through detailed reasoning and evidence extraction, SemanticCite bridges the gap between automated analysis and human editorial decision-making, supporting iterative manuscript improvement rather than binary acceptance/rejection outcomes. The same 4-class taxonomy applies directly to AI-generated content verification, addressing critical quality control challenges in automated reports, summaries, and AI-powered publishing workflows.
Future research directions include expanding multimodal capabilities, incorporating multilingual support, and developing AI-assisted improvement features that shift the system from detecting citation issues to actively supporting authors with specific recommendations for enhancement and source integration. These extensions will strengthen the framework’s applicability to fact-checking in automated content production systems.
SemanticCite represents a significant step towards research quality tools that augment rather than replace human expertise, providing scalable, open-source solutions for maintaining citation accuracy in an era of rapidly expanding scientific output and AI-generated content.