2604.06170

# Paper Circle: An Open-source Multi-agent Research Discovery and Analysis Framework **Authors**: GitHub: github.com/MAXNORM8650/papercircle, Website: papercircle.vercel.app/ ## Abstract The rapid growth of scientific literature has made it increasingly difficult for researchers to efficiently discover, evaluate, and synthesize relevant work. Recent advances in multi-agent large language models (LLMs) have demonstrated strong potential for understanding user intent and are being trained to utilize various tools. In this paper, we introduce Paper Circle, a multi-agent research discovery and analysis system designed to reduce the effort required to find, assess, organize, and understand academic literature. The system comprises two complementary pipelines: (1) a Discovery Pipeline that integrates offline and online retrieval from multiple sources, multi-criteria scoring, diversity-aware ranking, and structured outputs; and (2) an Analysis Pipeline that transforms individual papers into structured knowledge graphs with typed nodes (e.g., concepts, methods, experiments, and figures) and edges, enabling graph-aware question answering and coverage verification. Both pipelines are implemented within a coder LLM–based multi-agent orchestration framework and produce fully reproducible, synchronized outputs (JSON, CSV, BibTeX, Markdown, and HTML) at each agent step. This paper describes the system architecture, agent roles, retrieval and scoring methods, knowledge graph schema, and evaluation interfaces that together form the Paper Circle research workflow. We benchmark Paper Circle on both paper retrieval and paper review generation, reporting hit rate, MRR, and Recall@K. Results show consistent improvements with stronger agent models. We have publicly released the website and code. <details> <summary>2604.06170v1/x2.png Details</summary>  ### Visual Description ## Circular Diagram: Abstract Geometric Composition ### Overview The image depicts a minimalist circular diagram with a central purple circle surrounded by concentric white segments. No textual labels, legends, or numerical data are visible. The design emphasizes symmetry and color contrast. ### Components/Axes - **Central Element**: A solid purple circle at the center, occupying ~20% of the total diameter. - **Surrounding Segments**: 12 evenly spaced white triangular segments radiating outward, forming a ring around the purple core. - **Color Scheme**: - Purple (#800080) for the central circle. - White (#FFFFFF) for the peripheral segments. - **No Axes, Legends, or Labels**: The diagram lacks textual annotations, scales, or categorical markers. ### Detailed Analysis - **Symmetry**: The 12 white segments are uniformly distributed, suggesting rotational symmetry (30° spacing between segments). - **Proportions**: - Central purple circle: ~20% of the total radius. - White segments: ~80% of the total radius, with uniform width. - **Texture/Shading**: No gradients or shading variations; both elements are flat-colored. ### Key Observations 1. **Color Contrast**: The purple-white dichotomy creates a stark visual hierarchy, emphasizing the central circle. 2. **Segment Uniformity**: All white segments are identical in shape and spacing, indicating intentional design for balance. 3. **Absence of Data**: No numerical values, textual descriptions, or contextual clues (e.g., units, categories) are present. ### Interpretation This diagram likely serves as an abstract representation of a system or concept, such as: - A **logo** or **branding element** (e.g., a target symbol with a central focus). - A **conceptual model** for modular systems (e.g., 12 components arranged around a core). - An **artistic abstraction** prioritizing form over function. The lack of textual information prevents direct interpretation of data trends or relationships. The design’s simplicity suggests it may function as a placeholder, icon, or symbolic representation rather than a data-driven visualization. ## Notes on Missing Elements - No textual content exists in the image. - No spatial grounding of legends or axis markers is possible. - No trends or numerical data can be inferred. </details> Paper Circle: An Open-source Multi-agent Research Discovery and Analysis Framework Komal Kumar 1, Aman Chadha 2, Salman Khan 1, Fahad Shahbaz Khan 1, Hisham Cholakkal 1 1 Mohamed bin Zayed University of Artificial Intelligence 2 AWS Generative AI Innovation Center, Amazon Web Services GitHub: github.com/MAXNORM8650/papercircle Website: papercircle.vercel.app/ <details> <summary>2604.06170v1/x3.png Details</summary>  ### Visual Description ##Diagram: Multi-Agent Code Orchestrator System Architecture ### Overview The diagram illustrates a complex multi-agent system architecture for processing input data through various orchestrators, agents, and tools. It depicts data flow from input to output, with multiple processing pathways and integration points. ### Components/Axes **Key Elements:** 1. **Input** (Yellow box) - Entry point for data 2. **Discover** (Process) - Initial data processing 3. **arXiv Live** (Process) - Community data integration 4. **Community** (Process) - Social data processing 5. **View** (Output) - Visualization component 6. **State** (Database) - System state storage 7. **Paper Mind Orchestrator** (Blue box) - Research paper processing 8. **Paper Graph** (Process) - Paper data structuring 9. **Discovery Orchestrator** (Blue box) - Data discovery management 10. **Web Agent** (Light blue box) - Web interaction component 11. **Multi-agent Code Orchestrator Tracker** (Central process) - Core coordination system 12. **Query/Search/Sorting/Analysis/Export Agents** (Light blue boxes) - Specialized processing agents 13. **Tools** (Gray boxes) - System utilities including Web Tool, Web Visit, Query Intent, Paper Tool, Export, Sorting, Analysis, Structure **Color Coding:** - Yellow: Input/Output components - Blue: Orchestrators - Light Blue: Agents - Pink: Processes - Purple: State component - Gray: Tools ### Detailed Analysis **Flow Pathways:** 1. **Primary Input Path:** - Input → Discover → arXiv Live → Community → Discovery Orchestrator → Web Agent → State 2. **Secondary Input Path:** - Input → Circles → Paper Mind Orchestrator → Paper Graph 3. **Agent Coordination:** - Multi-agent Code Orchestrator Tracker connects to all agents (Query/Search/Sorting/Analysis/Export) 4. **Tool Integration:** - Tools section connects to all agents, providing specialized functionality **Spatial Relationships:** - Input/Output components (yellow) positioned at top-left and top-right - Orchestrators (blue) form central processing nodes - Agents (light blue) arranged in horizontal row beneath tracker - Tools (gray) at bottom, connected to all agents - State (purple) and View (yellow) at right side ### Key Observations 1. **Parallel Processing:** Multiple input paths suggest concurrent data processing capabilities 2. **Agent Specialization:** Five distinct agents handle specific tasks (query, search, sorting, analysis, export) 3. **Orchestrator Hierarchy:** Two levels of orchestrators (Paper Mind and Discovery) manage different data types 4. **Tool Integration:** System incorporates external resources through dedicated tools 5. **State Management:** Persistent storage component (State) maintains system status 6. **Visualization:** View component suggests output presentation capabilities ### Interpretation This architecture demonstrates a sophisticated multi-agent system designed for academic research data processing. The system employs: - **Distributed Processing:** Specialized agents handle different aspects of data manipulation - **Orchestration Layers:** Two orchestrator levels manage paper data and general discovery - **Tool Integration:** External resources are accessed through dedicated tools - **State Persistence:** System maintains continuous state tracking - **Output Flexibility:** Multiple output pathways (State database and View visualization) The design emphasizes modularity and scalability, allowing new agents or tools to be added without disrupting core functionality. The parallel processing paths suggest efficient handling of large data volumes, while the agent specialization enables deep analysis of specific data types. The integration of web tools and community data (arXiv Live) indicates a focus on real-time, collaborative research processing. </details> Figure 1: Overview of the Paper Circle pipeline. Given a user query, Paper Circle builds a paper set from multiple sources (e.g., paper graph, community, and arXiv live) via the Paper Mind for analysis and Discovery Orchestrators for search of the paper. A multi-agent layer (query, search, sorting, analysis, export) is coordinated by the Tracker, which maintains a shared state that is persisted to a backing store and displayed to the user through interface. ## 1 Introduction The pace of scientific publication has accelerated exponentially, creating a significant burden on researchers attempting to stay abreast of new developments Reddy and Shojaee (2025); Pramanick et al. (2023). Traditional search engines and recommendation systems often struggle to provide the depth and context required for rigorous literature reviews, leading to fragmented discovery workflows. Recently, the advent of Large Language Models (LLMs) has catalyzed a shift towards "AI Scientists", autonomous multi-agent systems (MAS) capable of generating hypotheses, conducting experiments, and even writing papers Chen et al. (2025b); Naumov et al. (2025). While these systems demonstrate the potential of agentic workflows, there remains a critical gap between fully autonomous simulations and the practical, collaborative needs of human research communities. Paper Circle addresses (as shown in the Figure 1) this gap by introducing a comprehensive Multi-Agent Research Platform that supports the entire lifecycle of literature engagement: from discovery and analysis to critique and synthesis. In the Table 1, we compared to existing multi-agent architectures for scientific literature tasks. Paper Circle offers a unique combination of capabilities that no existing system jointly provides. Specifically, it is designed to reduce the effort required to find, assess, organize, and understand academic literature. Unlike purely autonomous systems that aim to replace the researcher, Paper Circle is designed as a collaborative workbench that augments human intelligence through three integrated subsystems: Table 1: Comparison of Paper Circle against prior literature systems. Green indicates supported, orange indicates partial support, and red indicates unsupported. Favorable Partial Unfavorable 1. Discovery Pipeline: A multi-agent retrieval system that goes beyond simple keyword matching. It employs a multi-dimensional scoring framework to surface high-value research. Crucially, this pipeline is deterministic and produces structured artifacts (JSON, linear logs) at every step. 1. Paper Mind Graph: To facilitate deep understanding, Paper Circle constructs a dynamic Knowledge Graph from retrieved literature. This "Paper Mind" enables researchers to query the collective intelligence of a reading list, identifying latent connections between disparate works and supporting complex Question-Answering workflows that are grounded in specific citation sub-graphs. 1. Review Agents: This platform features a team of specialized review agents that generate detailed critiques and scores, consistently highlighting strengths and weaknesses to guide human reading priorities Naumov et al. (2025). By integrating these capabilities into a shared "Reading Circle" environment, Paper Circle transforms literature review from a solitary task into a community-driven, AI-augmented operation. ## 2 Related Work ### 2.1 Autonomous Scientific Discovery The emerging field of AI-Scientists aim to automate the entire research lifecycle. Systems like DORA AI agent Naumov et al. (2025) and EvoResearch Gajjar (2025) demonstrate end-to-end capabilities, from hypothesis generation to report writing. Similarly, O-Researcher Li et al. (2025), MARS Chen et al. (2025a), and AlphaResearch Yu et al. (2025c) treat research as a multi-step optimization problem, often using reinforcement learning to refine discovery strategies. Specialized agents have also been proposed for causal discovery, such as CausalSteward Wang et al. (2025) and other multi-agent frameworks Le et al. (2025). While these systems push the boundaries of autonomy, Paper Circle prioritizes curation and reproducibility over full automation. Instead of replacing the researcher, Paper Circle acts as a force multiplier for human teams, ensuring that the discovery process remains transparent and verifiable. ### 2.2 MAS in Specialized Domains MAS have shown remarkable success in specific scientific verticals. In chemistry and materials science, frameworks like ChemThinker Ju et al. (2025), MOOSE-Chem Yang et al. (2025), and ChemBOMAS Han et al. (2025a) leverage LLMs to discover new molecules and optimize experiments Kumbhar et al. (2025). In biology and healthcare, agents facilitate single-cell analysis (CellAgent Xiao et al. (2024)), phenotype discovery (PhenoGraph Niyakan and Qian (2025)), and clinical data analysis Spieser et al. (2025). Other applications range from drug discovery Fehlis et al. (2025) and psychiatry diagnosis Xiao et al. (2025) to financial forecasting, where systems like ASTRAFIN Singh and Kumar (2025) and other stock analysis agents Chandrashekar et al. (2025); Wawer and Chudziak (2025) predict market trends. Paper Circle complements these domain-specific tools by providing a general-purpose discovery pipeline that can be adapted to any discipline, serving as the foundational layer for literature review and knowledge management. ### 2.3 Community Simulation and Collaboration A distinct line of research focuses on simulating or facilitating the social aspects of science. ResearchTown Yu et al. (2025a, b) models the research community using agents to understand how ideas propagate. Other works explore collaborative dynamics through automated negotiation (NegoLog Doğru et al. (2024), NEGOTIATOR Keskin et al. (2024)) and cohesive dialogue generation Chu et al. (2024). Frameworks like PiFlow Pu et al. (2025), REDEREF Yuan and Xie (2025), and blackboard systems Salemi et al. (2025) propose mechanisms for agent collaboration in information discovery. Paper Circle distinguishes itself by moving beyond simulation; it provides a real-world platform for human-AI collaboration. It does not just model how researchers might interact, but actively facilitates those interactions through shared reading lists, discussion threads, and collaborative ranking. ## 3 Methodology ### 3.1 Background Multi-Agent Systems (MAS) represent a paradigm where autonomous entities interact to solve complex problems distributedly. In the context of scientific discovery, MAS allows for the decomposition of intricate research tasks,such as literature search, reading, and reasoning,into manageable sub-routines handled by specialized agents Wooldridge (2002). Unlike monolithic LLM approaches, agentic workflows can maintain distinct personas (e.g., "The Skeptic", "The Creative") and leverage external tools, reducing hallucination and improving reasoning depth through inter-agent dialogue Reddy and Shojaee (2025). The baseline for our orchestration layer is the smolagents (Roucher et al., 2025) library. The pipeline uses a CodeAgent (CoA) as the central orchestrator, which can attend parallel agent calls and toll calls and multiple ToolCallingAgent (ToCA) instances, each attached to specific capabilities (e.g., arXiv retrival, PDF parsing). The baseline responsibilities include (i) tool invocation, (ii) multi-step planning via the orchestrator, and (iii) delegation to specialized agents. PaperCircle extends this foundation by adding structured outputs, offline search capabilities, and rigorous evaluation metrics. We preserve the baseline tool interface, where each tool receives explicit parameters and returns a formatted string response, allowing the orchestrator to chain steps while maintaining high readability and traceability. ### 3.2 System Architecture Figure 1 illustrates the overall architecture of Paper Circle. The system consists of two complementary multi-agent pipelines: the Discovery Pipeline for finding relevant papers, and the Analysis Pipeline for deep understanding of individual papers. ### 3.3 Paper Discovery Agent Design The main diagram of the discovery subsystem is shown in Figure 2, which is composed of multiple agents, each bound to a small, explicit tool interface. It is inspired by the TTD-DR (Han et al., 2025b) for iteratively updating the updated version at each agentic step. The core agents are: <details> <summary>2604.06170v1/x4.png Details</summary>  ### Visual Description ## Flowchart: Draft Creation Process with Orchestrated Agents ### Overview The diagram illustrates a multi-step workflow for creating a final draft through iterative refinement and agent collaboration. It begins with an empty draft and progresses through "Noised Draft" and "Denoised Draft" stages, ultimately reaching a "Final Draft" after `k` steps. The process is managed by an **Orchestrator Agent** that coordinates specialized agents and integrates user input. --- ### Components/Axes 1. **Top Section (Inputs/Outputs):** - **Papers**: Formats listed (json, bib, md, csv) - **Links**, **Statistics**, **Summary**: Output categories 2. **Central Workflow:** - **Empty Draft** → **Noised Draft** (steps 0) → **Denoised Draft** (steps k) → **Final Draft** - **Orchestrator Agent**: Central node coordinating agents - **User Query**: Input trigger - **Website Visit**: Clarification feedback loop 3. **Agent Subsystem:** - **Query Decompositor** - **Paper Search** - **Ranking & Sorting** - **Analysis & Insights** - **Export Agent** --- ### Detailed Analysis 1. **Draft Evolution:** - The process starts with an **Empty Draft** (step 0). - Iterative refinement occurs between **Noised Draft** and **Denoised Draft** stages, with `k` steps indicating repeated cycles of noise injection and denoising. - Final output is a **Final Draft**, suggesting convergence after sufficient iterations. 2. **Agent Roles:** - **Orchestrator Agent**: Manages state updates and coordinates agents. - **Query Decompositor**: Breaks down user queries into actionable components. - **Paper Search**: Retrieves relevant documents (JSON, BIB, MD, CSV formats). - **Ranking & Sorting**: Prioritizes search results. - **Analysis & Insights**: Extracts key information from papers. - **Export Agent**: Finalizes output (e.g., summary, statistics). 3. **Feedback Loops:** - **Clarification**: Website visits allow user input to refine the process. - **State Updates**: The Orchestrator Agent dynamically adjusts the workflow based on agent outputs. --- ### Key Observations - **Iterative Refinement**: The "Noised Draft" → "Denoised Draft" loop emphasizes error correction and improvement. - **Modular Design**: Specialized agents handle distinct tasks (search, analysis, export), enabling scalability. - **User Integration**: The "Website Visit" node ensures human oversight and adaptability. --- ### Interpretation This flowchart represents a **human-in-the-loop AI system** for academic or technical writing. The Orchestrator Agent acts as a workflow manager, while specialized agents handle domain-specific tasks. The iterative "Noised/Denoised" steps mirror machine learning techniques (e.g., denoising autoencoders), suggesting the system uses probabilistic or statistical methods to refine drafts. The inclusion of **Website Visit** highlights the importance of human validation, ensuring outputs align with user intent. The process balances automation (via agents) with human oversight, making it suitable for complex, knowledge-intensive tasks like research paper drafting or report generation. </details> Figure 2: The main iterative diagram for the paper discovery framework. The system maintains an explicit, evolving discovery state (papers, links, statistics, and summaries) that is iteratively updated through agentic steps. Starting from an empty draft, the orchestrator agent alternates between noising and denoising operations over multiple steps, progressively refining the draft into a final result. When necessary, a web search agent is invoked for clarification or recent information. Intent Classification Agent. Parses user text into search mode (offline, online, or both), conference filters, year range, and ranking preferences. Most importantly, it uses a web agent in the pipeline for any unclear queries or recent knowledge. Paper Search Agent. Executes offline or online retrieval based on intent, merges results, performs deduplication, and updates state and outputs. Sorting Agent. Reorders papers using recency, citations, similarity, novelty, BM25 Chen and Wiseman (2023), or combined weights; or applies a cross-encoder reranker Wang et al. (2020). Analysis Agent. Computes aggregate statistics and insights, including source distribution, year trends, and top authors. Export Agent. Produces synchronized exports and provides a consistent interface for downstreaming. Web Search Agent. Provides auxiliary access to web search tools when online lookups are required. ### 3.4 Paper Analysis Agent While the discovery pipeline addresses the challenge of finding relevant papers, researchers also need to understand and synthesize the content of individual papers deeply Korat (2025). Paper Circle addresses this with a complementary Paper Analysis Agent that transforms research papers into structured, queryable knowledge graphs with full traceability to the original text. <details> <summary>2604.06170v1/x5.png Details</summary>  ### Visual Description ## Flowchart: Academic Paper Analysis Workflow ### Overview The image depicts a hierarchical workflow for analyzing academic papers, structured as a flowchart with four main stages: **Paper Analysis (Orchestrator)**, **Concept Extractor**, **Method Extractor**, **Experiment Extractor**, and **Linkage Agent**. Each stage feeds into four downstream processes: **INGESTION**, **CHUNKING**, **GRAPH BUILDER**, and **QnA**, which are further decomposed into granular components. ### Components/Axes - **Top Level (Orchestrator)**: - **Paper Analysis (Orchestrator)** (central node at the top). - **First Tier (Extractors)**: - **Concept Extractor** (leftmost). - **Method Extractor** (second from left). - **Experiment Extractor** (second from right). - **Linkage Agent** (rightmost). - **Second Tier (Processes)**: - **INGESTION** (leftmost). - **CHUNKING** (second from left). - **GRAPH BUILDER** (second from right). - **QnA** (rightmost). - **Third Tier (Components)**: - **INGESTION**: PDF Parse, Sections, Figures, Tables. - **CHUNKING**: Paragraph, Figures, Tables, Equations. - **GRAPH BUILDER**: Concepts, Methods, Experiments, Links. - **QnA**: RAG, Locate, Export, Verify. ### Detailed Analysis - **Flow Direction**: - Arrows flow **top-down** from the Orchestrator to the Extractors, then to the Processes, and finally to the Components. - No bidirectional or cyclical connections are present. - **Component Relationships**: - The Orchestrator distributes tasks to the four Extractors, which in turn route data to the four Processes. - Each Process is specialized: - **INGESTION** handles raw data parsing (PDFs, sections, figures, tables). - **CHUNKING** breaks content into smaller units (paragraphs, equations). - **GRAPH BUILDER** organizes extracted data into conceptual, methodological, and experimental relationships. - **QnA** focuses on retrieval-augmented generation (RAG) and verification. ### Key Observations 1. **Hierarchical Structure**: The workflow is strictly layered, with no overlap between tiers. 2. **Specialization**: Each Process and Component has a distinct, non-overlapping role. 3. **No Numerical Data**: The flowchart lacks quantitative metrics, focusing solely on structural relationships. ### Interpretation This flowchart represents a **modular pipeline for academic paper analysis**, emphasizing decomposition and specialization. The Orchestrator acts as a central coordinator, ensuring tasks are distributed to domain-specific Extractors. The Processes then transform raw data into structured components, enabling downstream tasks like knowledge graph construction (GRAPH BUILDER) and question-answering (QnA). The absence of numerical data suggests the diagram prioritizes **process clarity over performance metrics**, likely serving as a conceptual blueprint for a larger system. The workflow aligns with principles of **modular design** and **pipeline optimization**, where each stage is isolated to simplify debugging and scalability. For example, separating INGESTION from CHUNKING ensures that parsing errors do not propagate to later stages. Similarly, the Linkage Agent’s role in connecting concepts, methods, and experiments implies a focus on interdisciplinary analysis. ### Limitations - **No Metrics**: The lack of quantitative data (e.g., processing time, accuracy) limits insights into efficiency. - **Static Representation**: The flowchart does not depict dynamic interactions or feedback loops, which may exist in a real-world implementation. </details> Figure 3: A paper analysis orchestrator agents for concepts, methods, experiments, and cross-entity linkages. The pipeline consists of four main stages: ingestion, which parses PDFs into structured elements (sections, figures, tables, equations); semantic chunking, which produces structure-aware text units; graph construction, which builds a typed knowledge graph of concepts, methods, experiments, and their relations with full traceability to source text; and a Q&A layer that enables graph-aware retrieval, verification, and export. The Paper Analysis Agent operates as a multi-stage pipeline with four specialized components as shown in the figure: (1) Ingestion Layer, (2) Graph Builder, (3) Q&A System, and Verification Layer. #### PDF Ingestion and Chunking. The ingestion pipeline uses PyMuPDF for robust PDF parsing Adhikari and Agarwal (2024). The PDFParser class extracts: Metadata: Title, authors, abstract, arXiv ID, venue, and page count. Sections: Hierarchical section structure with parent-child relationships, identified via numbering patterns (e.g., “1.2 Background”). Figures and Tables: Caption text, page locations, and nearby context for linkage. Equations: Numbered equations with surrounding context. Unlike token-based chunking, the SemanticChunker Qu et al. (2025) creates chunks aligned with document structure. Paragraphs within sections are grouped up to a configurable limit (default 1500 characters), while figures, tables, and equations are preserved as distinct chunks with their captions and context. #### Knowledge Graph Schema. The mind graph follows a typed schema with nodes Zhang et al. (2025a) for papers, sections, concepts, methods, experiments, datasets, and visual elements (figures, tables, equations), and edges encoding structural and semantic relations (e.g., hierarchy, definition, proposal, usage, evaluation, illustration, dependency). All nodes and edges carry provenance metadata—including source chunk IDs, page numbers, verification status, confidence scores, and timestamps—ensuring full traceability to the original PDF. ### 3.5 Multi-Agent Extraction The GraphBuilder (Zhu et al., 2024b) orchestrates four specialized CoA -based extractors. The Concept Extractor identifies and classifies key concepts by type and importance; the Method Extractor extracts algorithms and techniques from method sections; the Experiment Extractor recovers experimental setups, datasets, metrics, and results; and the Linkage Agent connects figures and tables to the concepts or methods they illustrate. Extraction proceeds in staged phases—concepts, methods, experiments, visual linkage, and inter-concept relations—each incrementally updating the shared MindGraph. #### Graph-Aware Q&A. The Q&A module combines vector retrieval with graph traversal. An EmbeddingStore indexes text chunks and node descriptions, while the GraphRetriever retrieves top- $k$ relevant nodes and chunks and expands context via 1-hop neighbors. The PaperQA agent generates answers grounded in retrieved text, graph relations, and linked figures or tables, and returns supporting evidence with confidence estimates. A locate function enables precise localization of concepts, figures, or tables by page and context. #### Coverage Verification. To prevent silent omissions, a CoverageChecker evaluates figure, table, section, and equation coverage, producing an overall coverage score and identifying unlinked or missing elements with actionable diagnostics. This provides a lightweight quality assurance step prior to downstream use. <details> <summary>2604.06170v1/x6.png Details</summary>  ### Visual Description ## Flowchart: Academic Paper Analysis System ### Overview The flowchart depicts a multi-agent system for analyzing academic papers, starting from a PDF/URL input and culminating in a final report. The system employs hierarchical processing through specialized agents, with feedback loops and cross-agent dependencies. ### Components/Axes 1. **Input Layer** - Paper (pdf/url) → Orchestrator Agent - Orchestrator Agent → PDF Processor (feedback loop) 2. **Core Processing** - Orchestrator Agent → Paper Meta Data - Orchestrator Agent → Paper Agent context 3. **Specialized Agents** - **Deep Analyzer** (inputs: Paper Meta Data) - **Critic Agent** (inputs: Paper Meta Data) - **Literature Expert Agent** (inputs: Paper Meta Data) - **Knowledge Graph Agent** (inputs: Paper Meta Data) - **Contribution Analyzer Agent** (inputs: Deep Analyzer) - **Reproducibility Checker Agent** (inputs: Critic Agent, Literature Expert Agent) - **Summarizer Agent** (inputs: Knowledge Graph Agent) 4. **Sub-Components** - ArXiv Search (inputs: Reproducibility Checker Agent) - Locate text (inputs: Reproducibility Checker Agent) - Semantic Scholar (inputs: Reproducibility Checker Agent) - Knowledge Graph (inputs: Summarizer Agent) 5. **Output** - Generate Final report (inputs: All specialized agents) ### Detailed Analysis - **Color Coding**: - Light pink boxes represent input/output nodes (Paper, PDF Processor, Generate Final report) - Medium pink boxes denote agent components - Dark pink boxes indicate sub-components - **Flow Direction**: - Top-to-bottom hierarchy with left-to-right branching - Feedback loop from Orchestrator Agent to PDF Processor - **Dependencies**: - Reproducibility Checker Agent integrates inputs from two distinct paths - Knowledge Graph Agent feeds directly into Summarizer Agent - Final report aggregates outputs from all specialized agents ### Key Observations 1. **Hierarchical Structure**: - Orchestrator Agent acts as central coordinator - Specialized agents operate in parallel but feed into common sub-components 2. **Cross-Functional Integration**: - Reproducibility Checker Agent combines Critic Agent and Literature Expert Agent outputs - Knowledge Graph Agent serves as both input and output node 3. **Feedback Mechanism**: - PDF Processor receives input from Orchestrator Agent while also providing initial processing ### Interpretation This system demonstrates a modular approach to academic paper analysis, where: 1. **Specialization Drives Efficiency**: Each agent focuses on specific aspects (e.g., contribution analysis, reproducibility) while sharing common resources like Paper Meta Data 2. **Knowledge Graph Centrality**: The Knowledge Graph Agent acts as both processor and repository, suggesting its critical role in synthesizing information 3. **Reproducibility Emphasis**: The dedicated Reproducibility Checker Agent highlights the system's focus on methodological rigor 4. **Iterative Processing**: The feedback loop between Orchestrator Agent and PDF Processor implies continuous refinement of input processing The architecture suggests a balance between depth (through specialized agents) and breadth (through cross-agent integration), with the Knowledge Graph serving as the system's memory and synthesis engine. </details> Figure 4: Multi-agent paper analysis and review architecture. Given a paper specified by a PDF or URL, an orchestrator agent coordinates PDF processing and maintains shared paper metadata and agent context. Specialized agents operate in parallel to perform deep technical analysis, contribution extraction, critical review, literature linking, reproducibility checking, summarization, and knowledge graph construction. External tools such as arXiv search, Semantic Scholar, and targeted text localization are invoked as needed. The orchestrator aggregates agent outputs into a unified, structured final report, enabling comprehensive, reviewer-style analysis with modular extensibility. ### 3.6 Research Review Framework In Sec. 3.4, we describe the paper analysis of agentic capabilities, which we further extend for automated peer-review-style assessment. Unlike AgentReview Jin et al. (2024); D’Arcy et al. (2024), we follow the paper analysis perspective, which not only provides the review but also builds a strong graph between the concepts. #### Architecture. The system is built upon a multi-agent orchestration framework (Figure 4) that coordinates the execution of seven specialized roles. Each agent is instantiated as a ToCA or CoA Roucher et al. (2025). #### Deep Analyzer. Focuses on the technical core of the paper. It breaks down the mathematical foundations, identifies specific methodology components, and extracts primary experimental findings. #### Critic. Emulates a senior conference reviewer (e.g., NeurIPS, ICML). It provides a rigorous assessment of strengths and weaknesses, generates author-facing questions, and assigns scores for novelty, clarity, and significance. #### Literature Expert. Interfaces with external academic databases including Semantic Scholar and arXiv. It maps the paper’s position within the existing research landscape and verifies citation accuracy. #### Contribution Analyzer. Separates explicit author claims from verified technical contributions, identifying potential overclaiming or missing baseline comparisons. #### Reproducibility Checker: Quantifies the transparency of the research by assessing the availability of source code, hyperparameter specifications, dataset accessibility, and compute requirement disclosures. #### Summarizer. Generates multi-fidelity summaries across different abstraction levels, ranging from concise executive summaries to deep technical precis. #### Orchestration and Pipeline Execution The Multi Agent Orchestrator manages the lifecycle of these agents through a multi-stage pipeline. The system supports parallel execution using a ThreadPoolExecutor. ## 4 Experiments ### 4.1 Experimental setup All the experiments are done with open-source model with $4× 40$ GB Nvidia GPUs. We used the Ollama https://ollama.com/ platform with the fastllm library (Gong et al., 2025). #### Database Curation. We curated a diverse corpus, as shown in Table 2 of research papers from leading CS and ML conferences, primarily sourced from OpenReview https://openreview.net/ and augmented with metadata and peer-review information. | Count | 12 | 39 | 13 | 13 | 25 | 25 | 5 | 5 | 7 | 4 | 144 | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | Table 2: The Database corpus across major conferences. The “Other” category includes venues such as AISTATS, RSS, SIGGRAPH, and WACV. Count indicates the number of the most recent conference venue included. #### Evaluation. Paper Circle provides built-in evaluation metrics. When a ground-truth paper title or identifier is provided, the system computes Mean Reciprocal Rank (MRR), Recall@K, Precision@K, and hit rates. These metrics are computed per step and stored in the JSON file for longitudinal tracking. For batch evaluation, a parallel benchmarking utility executes multiple queries concurrently and aggregates mean metrics and timing statistics. This supports lightweight comparisons between search configurations (offline vs. online, BM25 Chen and Wiseman (2023) vs. semantic (all-MiniLM-L6-v2 Wang et al. (2020)), with or without Qwen3-Reranker-0.6B (Zhang et al., 2025b)) without requiring external tooling. #### Baseline Agent. This framework is developed using the Smolagent multi-agent tool, calling the (ToCA) agent and the code agent (CoA), with tools utilized being manually developed. #### Architecture. We evaluate multiple retrieval baselines: bm25, bm25+reranker (BM25 Chen and Wiseman (2023) & cross-encoder Zhang et al. (2025b)), reranker Zhang et al. (2025b), semantic Wang et al. (2020), and hybrid (BM25 combined with semantic retrieval). We also compare pipeline structures with different agent compositions: full includes all five agents (intent, search, sort, analysis, export), minimal uses only the search agent, search_sort uses search and sort, search_analysis uses search and analysis, and no_intent is a full pipeline with no intent. ### 4.2 Results #### Natural Text-based retrieval. We evaluate our multi-agent paper retrieval system across multiple LLMs and retrieval baselines. We did two query type experiments, one a research assistant-based natural queries generated by running gpt-oss-20B models (called RAbench), and randomly sampling one paper record from the database, extracting a concise “topic" phrase from its title, keywords or abstract, then picking a natural-language template and optional prefix to turn that topic into a realistic search query. We also randomly chose a scope (conference/year/range/none) to add corresponding text to the query and to emit matching structured filters. This query we referred to as SemanticBench. All experiments were conducted on a 50 query benchmark, measuring the success rate, the hit rate, the mean reciprocal rank (MRR), and the recall. #### Model Comparison. Table 3 presents comprehensive evaluation results comparing agent-based models with retrieval baselines. The results reveal a clear performance hierarchy across methods and scales. Two agent models achieve the highest retrieval effectiveness with an 80% hit rate, qwen3-coder-30b-Q3KM (quantized) and qwen3-coder:30b —with qwen3-coder-30b-Q3KM also delivering the best ranking quality (MRR = 0.627) while requiring less memory for smolagent multi-step reasoning. These top-performing models are also the fastest, taking approx. 21–22 seconds per query, indicating no latency penalty for improved accuracy. The BM25 baseline remains highly competitive (78% HR), outperforming most agent-based approaches and highlighting the continued strength of lexical matching in academic retrieval. Finally, RA-Bench results show higher performance than SemanticBench, suggesting that LLM-perturbed queries may be easier for multi-agent retrieval, though this requires further investigation. Table 3: Combined benchmark results for agent-based models and retrieval baselines. Best results are shown in bold. All the results are calculated using semantic benchmarks. Only the last (blue) is evaluated on 500 RAbench queries, which shows syntetically written query is easier to retrieve compared to the random template following. #### Paper analysis visualization. In the Figure 3, we provide various output visualizations, including concept built graph (A), concept definition chart (B), interactive Q&A with precise information (C), markdown analysis output (D), and finally flow chart connecting the concepts of blocks (E). All of this analysis togather provides the complete understanding of the paper. <details> <summary>2604.06170v1/x7.png Details</summary>  ### Visual Description ## Diagram: Any4D: Unified Field - Forward Me ### Overview The image depicts a technical flowchart diagram titled "Any4D: Unified Field - Forward Me," accompanied by explanatory text in a sidebar, a question-answer section, and UI elements. The flowchart uses color-coded nodes to represent components of a multi-view reconstruction system, with a central blue node as the core concept. A sidebar explains "manifold-constrained hyper-connections" (mHC) and their technical context. UI tabs (Flowchart, Concepts, Methods, Experiments, Interactive Graph) suggest an interactive technical interface. ### Components/Axes **Flowchart Diagram**: - **Central Node**: Blue circle labeled "Any4D: Unified Field - Forward Me" (core concept). - **Categorized Nodes**: - **Top/Left**: - Yellow nodes: "Metric-Scale Re", "Lack of Large-S", "Efficient Infer", "Predictive Cost", "Dynamic Video S", "Scene Flow DPT", "Multi-view Trail", "Training Dataset", "Score Flow Sup." - Black nodes: "40 Waymo-DriveTrack", "4D Reconstruct", "Fragmented Sub-Metric", "Dynamic 4D Video S". - **Right**: - Purple nodes: "Simultaneous Lo", "Dynamic Scene IT", "Dynamic Scene F", "Allocentric Coord", "Structure-from-Init", "Structure-from-Img". - Green nodes: "Feed-Forward Wu", "Optical Flow", "Simultaneous Lo", "Any4D", "40 Waymo-DriveTrack". - **Bottom**: - Orange nodes: "8 40 Waymo-DriveTrack", "9 5000 Dynamic Dataset", "10 3D DeepStereo". - **Bottom-Right**: - Red nodes: "Scene Flow DPT", "3 5000 Scene Flow", "7 725 CoarseNet". **Sidebar Text**: - **Concepts**: - **Feed-Forward Multi-view Inference**: "A reconstruction method that processes multi-view data without iterative optimization." - **BlendochWS**: "A loss function for 4D monocular video reconstruction." - **DINOv2 Image Encoder**: "A vision encoder trained on DINOv2 with 8% dropout rate." - **Methods**: - **Scene-Flow DPT Decoder**: "A decoder trained on scene flow data with specific architecture." - **Multi-view Training**: "Training using multi-view inference." - **Benchmarking Setup**: "Evaluation methodology for datasets like TAPVID-3D, DriveTrack, and Waymo Open Dataset." **Question-Answer Section**: - **Question**: "What is manifold-constrained hyper-connections (mHC), and what is Hyper-Connections (HC)?" - **Answer**: - **Hyper-Connections (HC)**: Extends residual connections by expanding stream width and diversifying connectivity patterns, improving performance but causing training instability. - **Manifold-Constrained HC (mHC)**: Addresses instability via manifold constraints while retaining benefits of expanded streams. - **Figures**: Figure 1 (mHC architecture), Figure 5 (training dynamics). **UI Elements**: - **Tabs**: Flowchart (active), Concepts, Methods, Experiments, Interactive Graph. - **Technical Terms**: - "mHC/HC Expansion Rat" - "mHC/HC Gating Factor" - "Sinkhorn-Knopp" - "Grouped Query Attention" - "Multi-Head Latent Attention" ### Detailed Analysis 1. **Flowchart Structure**: - The central node "Any4D" connects to three main categories: - **1 Introduction** (left): Focuses on foundational challenges (e.g., lack of large-scale datasets). - **2 Existing 4D Recorder** (top-right): Details components like Scene Flow DPT and Optical Flow. - **3 Dynamic Scene R** (bottom-right): Highlights challenges like fragmented sub-metrics. - **4 40 Waymo-DriveTrack** (bottom-left): Represents the benchmarking dataset. 2. **Sidebar Technical Content**: - Describes mHC as an improved version of HC, addressing training instability through manifold constraints. - Explains DINOv2 Image Encoder and Scene-Flow DPT Decoder as specialized components. - Multi-view Training and Benchmarking Setup emphasize evaluation methodologies. 3. **Question-Answer Section**: - Clarifies the distinction between HC (general residual extension) and mHC (manifold-constrained improvement). - References Figure 1 and 5 for architectural and training dynamics. ### Key Observations - The flowchart emphasizes **multi-view reconstruction** with a focus on scene flow, dynamic 4D video, and dataset-specific challenges. - The sidebar links mHC to technical improvements in training stability and reconstruction quality. - UI tabs suggest an interactive platform for exploring technical components (e.g., "Interactive Graph" likely visualizes data flow). ### Interpretation The diagram illustrates the **Any4D framework**, which integrates multi-view reconstruction techniques with hyper-connection optimizations. The central node ("Any4D") serves as the unifying concept, while connected nodes break down into components like scene flow, dynamic scenes, and dataset-specific challenges. The sidebar contextualizes mHC as a technical solution to training instability, positioning it within broader methods like DINOv2 encoding and benchmarking setups. The UI elements (tabs, technical terms) imply an interactive environment where users can explore the flowchart’s nodes, access detailed methods, or visualize data flow. The question-answer section highlights the importance of mHC in balancing performance and stability, suggesting it is a critical innovation in the framework. **Critical Uncertainty**: - Color meanings for flowchart nodes are not explicitly defined in the legend, leaving their relationships to the core concept (Any4D) speculative. - The exact numerical values for metrics like "3 5000 Scene Flow" or "7 725 CoarseNet" are ambiguous and require further context from figures (e.g., Figure 5). **Notable Trends**: - The flowchart prioritizes **scene flow** and **dynamic 4D video** as key research areas, with repeated references to Waymo datasets and Scene Flow DPT. - The sidebar’s emphasis on mHC suggests it is a foundational component addressing prior limitations in hyper-connection methods. This diagram positions Any4D as a system designed to improve multi-view 4D reconstruction through manifold-constrained hyper-connections, leveraging specialized decoders, training strategies, and benchmarking datasets. </details> Figure 5: The main outputs of the analysis agent for a representative paper. (A) Interactive concept graph constructed from the paper, where nodes correspond to extracted concepts and edges denote semantic relationships. (B) Automatically generated concept explanations, each linked to the originating paper sections and pages. (C) Graph-aware question answering interface, providing answers grounded in extracted content along with supporting figures and references. (D) Structured Markdown exports summarizing all extracted concepts and methods for downstream use. (E) Flowchart view illustrating the high-level organization and relationships among concepts, methods, and experimental components of the paper. #### Paper review analysis To evaluate our multi-agent review system, we conducted a study using the released ICLR 2024 reviews. We randomly selected 50 papers spanning diverse rating levels, and report the results in Figure 6. We observe that the code-oriented agent (qwen3-coder-30B) often struggles to sustain a coherent review workflow, whereas chat-style LLMs (e.g., gpt-oss) produce stronger and more consistent reviews. Overall, review quality improves with larger models, suggesting that capacity and instruction-following are particularly important for end-to-end reviewing. <details> <summary>2604.06170v1/figures/simple_dashboard.png Details</summary>  ### Visual Description ## Chart/Diagram Type: Multi-subplot Visualization ### Overview The image contains four subplots (A-D) comparing model performance metrics across four AI models: GPT-120B, Qwen-30B-Q3, GPT-20B, and Qwen-30B. Each subplot uses distinct visualizations (bar charts, line charts) and color-coded legends to represent statistical measures. --- ### Components/Axes #### Subplot (A): Average MSE (with 95% CI) - **Y-axis**: Mean Squared Error (0–5) - **X-axis**: Model names (GPT-120B, Qwen-30B-Q3, GPT-20B, Qwen-30B) - **Legend**: None (colors directly map to models) - **Error Bars**: Vertical lines indicating 95% confidence intervals #### Subplot (B): Average Correlation - **Y-axis**: Correlation Coefficient (-0.15–0.25) - **X-axis**: Model names (same as Subplot A) - **Legend**: - Pearson (green, striped) - Spearman (blue, striped) - **Error Bars**: Smaller vertical lines for confidence intervals #### Subplot (C): Accuracy at Different Thresholds - **X-axis**: Error Threshold (-0.5, 1.0, 1.5) - **Y-axis**: Average Accuracy (%) (0–100) - **Legend**: - GPT-120B (green) - Qwen-30B-Q3 (red) - GPT-20B (blue) - Qwen-30B (orange) - **Lines**: Solid with markers at threshold points #### Subplot (D): Model Success Rate - **Y-axis**: Success Rate (%) (0–100) - **X-axis**: Model names (same as Subplot A) - **Legend**: None (colors match Subplot A) - **Bar Labels**: Success rate percentages (e.g., 96%, 94%) and counts (e.g., 48/50) --- ### Detailed Analysis #### Subplot (A): Average MSE - **Values**: - GPT-120B: 1.42 (±0.25) - Qwen-30B-Q3: 3.79 (±0.85) - GPT-20B: 1.34 (±0.20) - Qwen-30B: 3.44 (±0.75) - **Trend**: Qwen-30B-Q3 has the highest MSE, while GPT-20B has the lowest. #### Subplot (B): Average Correlation - **Pearson**: - GPT-120B: 0.05 (±0.03) - Qwen-30B-Q3: 0.01 (±0.02) - GPT-20B: 0.04 (±0.02) - Qwen-30B: -0.14 (±0.03) - **Spearman**: - GPT-120B: 0.09 (±0.04) - Qwen-30B-Q3: -0.02 (±0.03) - GPT-20B: 0.04 (±0.02) - Qwen-30B: -0.15 (±0.04) - **Trend**: Negative correlations for Qwen-30B in both metrics. #### Subplot (C): Accuracy at Thresholds - **Trends**: - All models improve accuracy as error thresholds increase. - GPT-120B and GPT-20B consistently outperform others. - Qwen-30B-Q3 shows the steepest improvement (18% gain from -0.5 to 1.5). #### Subplot (D): Model Success Rate - **Values**: - GPT-120B: 96% (48/50) - Qwen-30B-Q3: 94% (47/50) - GPT-20B: 84% (42/50) - Qwen-30B: 76% (38/50) - **Trend**: Success rates decrease from GPT-120B to Qwen-30B. --- ### Key Observations 1. **MSE vs. Success Rate**: Models with lower MSE (GPT-120B, GPT-20B) have higher success rates. 2. **Correlation Anomalies**: Qwen-30B shows negative correlations, suggesting potential inverse relationships. 3. **Accuracy Thresholds**: GPT-120B maintains >80% accuracy across all thresholds, while Qwen-30B drops below 60% at 1.5. 4. **Confidence Intervals**: Qwen-30B-Q3 has the widest error bars in MSE, indicating higher uncertainty. --- ### Interpretation - **Model Performance**: GPT-120B and Qwen-30B-Q3 dominate in MSE, correlation, and success rate, suggesting superior robustness. - **Negative Correlations**: Qwen-30B’s negative Spearman/Pearson values may indicate flawed metric alignment or data distribution issues. - **Threshold Sensitivity**: GPT-120B’s consistent accuracy across thresholds implies better generalization, while Qwen-30B struggles with higher errors. - **Success Rate Alignment**: Success rates in (D) directly correlate with MSE and correlation results, reinforcing the reliability of these metrics. This visualization highlights trade-offs between model complexity (e.g., GPT-120B vs. Qwen-30B) and performance, with GPT-based models generally outperforming Qwen variants. </details> Figure 6: Paper review results analysis. This study was conducted on 50 randomly selected ICLR 2024 reviews. #### Qualitative assessment We evaluated PaperCircle through 81 real-world discovery sessions (78 unique queries) conducted by researchers across diverse topics. The analysis of the results is shwon in the Table 4 and in Table 5. The 81 sessions span 9 research domains including world models, LLM training, neural architectures, multi-agent systems, healthcare AI (11%), model efficiency (10%), domain-specific applications (10%), computer vision (7%), and scientific reasoning (6%), demonstrating domain-agnostic applicability. The table below compares measurable discovery outcomes against the capabilities of standard single-source search tools. Table 4: Comparison of source coverage and export-related functionality across literature discovery systems. Percentages are computed with respect to the PaperCircle paper set. $†$ Fraction of PaperCircle’s 21,115 papers not retrievable from that single source alone. $‡$ Estimated based on the natural query. | Metric | arXiv | Semantic Scholar | Google Scholar | PaperCircle | | --- | --- | --- | --- | --- | | Sources queried per run | 1 | 1 | 1 | 8.7 avg. | | Papers not retrievable $†$ | 70.9% | 80.4% | 36.9% $‡$ | 9.0% | | PDF availability | ~90% | ~60% | Variable | 62.5% | | Supported export formats | 0 | 1–2 | 1 | 5 | | Bulk export support | ✗ | ✗ | ✗ | ✓ | | Process-level logs | ✗ | ✗ | ✗ | ✓ | Table 5: Summary statistics of Paper Circle usage and outputs. | Sessions | 81 | Observed user sessions | | --- | --- | --- | | Papers | 21,115 | Total papers processed | | arXiv miss | 70.9% | Fraction not retrievable from arXiv | | Semantic Scholar miss | 80.4% | Fraction not retrievable from Semantic Scholar alone | | Duplicates removed | 18,613 (43.5%) | Duplicate entries removed during processing | | Median time | 2.3 min | Median runtime per session | | Export formats | 5 / session | Number of supported export formats per session | Preliminary user feedback indicates minimal cognitive load when using PaperCircle. NASA-TLX Colligan et al. (2015) assessment yields an overall workload of 1.2/7, with five of six dimensions scoring the minimum (1/7) and effort at 2/7. Usability ratings are correspondingly strong: positive items (frequency of use, ease, integration, learnability, confidence) average 7.6/10, while negative items (complexity, support needs, inconsistency, cumbersomeness, learning curve) average 2.6/10. Notably, the participant rated learnability at 8/10 and learning barrier at 1/10, suggesting the system is accessible without prior training. ### 4.3 Ablation Studies We conduct comprehensive ablation studies to understand the contribution of different system components, including retrieval baselines, query configuration, and pipeline structures. #### Full Query utilization To assess the full capability of our system, we conducted an extended evaluation using the qwen3-coder-30b model across 500 queries under various configurations. Results are presented in Table 6. Table 6: Extended benchmark results for the Qooba agent (qwen3-coder-30b) across different configurations. | Default (Full Agent) With Filters & Offline Offline Only | 500 50 50 | 0.9818 0.9600 0.9200 | 0.8824 0.8485 0.6476 | 0.8381 0.7800 0.5600 | 0.9312 0.9000 0.7400 | 21.54 22.76 41.45 | | --- | --- | --- | --- | --- | --- | --- | | No Mentions | 50 | 0.6400 | 0.4316 | 0.3600 | 0.5200 | 38.35 | | Online/Offline Mix | 50 | 0.6200 | 0.4595 | 0.4200 | 0.5000 | 38.50 | #### Observations. The “With Filters & Offline” configuration performs better, suggesting that explicit context (conference/year filters) combined with local database access is highly effective. Notably, the “No Mentions” and “Online/Offline Mix” configurations show significant performance degradation (62–64% hit rate), indicating that specific paper references and structured retrieval chains are critical for accuracy. Overall, configurations exhibit similar latency, indicating stable scaling of the multi-agent pipeline across query settings as well. ### 4.4 Retrieval Baseline Ablations Table 7: Ablation study results comparing retrieval baselines and pipeline structures using qwen3-coder-30b. Full represents the full pipeline structure, minimal represents | BM25 Full BM25 Search Sort BM25 No Intent | bm25 bm25 bm25 | full search_sort no_intent | 0.9600 0.9600 0.9600 | 0.8629 0.8620 0.8554 | 0.8000 0.8000 0.8000 | 0.9200 0.9200 0.9200 | 33.75 33.95 31.47 | | --- | --- | --- | --- | --- | --- | --- | --- | | BM25 Search Analysis | bm25 | search_analysis | 0.9600 | 0.8437 | 0.7800 | 0.9200 | 32.81 | | BM25 Minimal | bm25 | minimal | 0.9600 | 0.8420 | 0.7800 | 0.9200 | 33.34 | | Hybrid Full | hybrid | full | 0.9600 | 0.8620 | 0.8000 | 0.9200 | 31.65 | | BM25 + Reranker | bm25+reranker | full | 0.9600 | 0.8692 | 0.8000 | 0.9400 | 935.07 | | Semantic Full | semantic | full | 0.9400 | 0.7097 | 0.6200 | 0.8800 | 31.28 | #### Retrieval Baseline Impact. BM25-based methods consistently outperform pure semantic retrieval. The semantic baseline shows a significant drop in R@1 (0.62) compared to BM25-based methods (0.80), suggesting that lexical matching remains crucial for precise paper retrieval. The hybrid approach performs on par with BM25, indicating that combining lexical and semantic signals does not provide additional benefits in this setting. #### Reranking Trade-offs. The BM25 + Reranker configuration achieves the highest MRR (0.8692) and R@5 (0.9400), but at a substantial computational cost, approximately 28 $×$ slower than other methods. This presents a clear accuracy-efficiency trade-off that practitioners must consider based on their deployment requirements. #### Pipeline Complexity. Reducing pipeline complexity (Minimal, Search Analysis configurations) leads to slight drops in MRR and R@1 while maintaining high overall hit rates (96%). Interestingly, removing intent analysis (“No Intent” configuration) results in a faster pipeline with competitive performance, suggesting that intent classification may be redundant for well-structured queries. ## 5 Conclusion Paper Circle shows how multi-agent workflows can streamline research literature management. Its discovery pipeline unifies heterogeneous search sources and multi-criteria scoring into a reproducible tool, using a simple agent–tool interface with shared state, deterministic ranking, and synchronized multi-format outputs. Its analysis pipeline converts papers into structured knowledge graphs that enable graph-aware QA, coverage checks, and human-in-the-loop verification. Future work will focus on the optimization of the unification of the pipeline. ## 6 Limitations Our review agent shows weak alignment with human judgments: across models, the correlation with human reviewer scores remains low ( $r<0.25$ ), and several metrics can even exhibit negative correlations, indicating that the system may rank papers in the opposite order of human preference. As a result, even the best-performing configurations do not reliably distinguish strong from weak submissions, and the system should not be used as a trusted mechanism for comparing or ranking papers. Based on our analysis, we found that this review process gets the benefit of a large model, so this problem can be overcome by large open/closed source models. ## References - N. S. Adhikari and S. Agarwal (2024) A comparative study of pdf parsing tools across diverse document categories. arXiv preprint arXiv:2410.09871. Cited by: §3.4. - Prof. Chandrashekar, M. Akram, M. Khan, P. Kumar, and P. Mandal (2025) A survey on stock investment risk analysis using crewai multi- agent system. International Research Journal of Modernization in Engineering Technology and Science. External Links: Document, Link Cited by: §2.2. - G. Chen, Z. Qiao, W. Wang, D. Yu, X. Chen, H. Sun, M. Liao, K. Fan, Y. Jiang, W. X. Zhao, et al. (2025a) MARS: optimizing dual-system deep research via multi-agent reinforcement learning. arXiv preprint arXiv:2510.04935. Cited by: §2.1. - R. Chen, H. Su, S. TANG, Z. Yin, Q. Wu, H. Li, Y. Sun, W. Ouyang, P. Torr, and N. Dong (2025b) AI-driven automation can become the foundation of next-era science of science research. NIPS 2025. External Links: Link Cited by: §1. - X. Chen and S. Wiseman (2023) Bm25 query augmentation learned end-to-end. arXiv preprint arXiv:2305.14087. Cited by: §B.3, §3.3, §4.1, §4.1, Table 3. - K. Chu, Y. Chen, and H. Nakayama (2024) Cohesive conversations: enhancing authenticity in multi-agent simulated dialogues. COLM 2024. External Links: Link Cited by: §2.3. - L. Colligan, H. W. Potts, C. T. Finn, and R. A. Sinkin (2015) Cognitive workload changes for nurses transitioning from a legacy system with paper documentation to a commercial electronic health record. International journal of medical informatics 84 (7), pp. 469–476. Cited by: §4.2. - M. D’Arcy, T. Hope, L. Birnbaum, and D. Downey (2024) Marg: multi-agent review generation for scientific papers. arXiv preprint arXiv:2401.04259. Cited by: §3.6. - M. Das, P. Alphonse, et al. (2023) A comparative study on tf-idf feature weighting method and its analysis using unstructured dataset. arXiv preprint arXiv:2308.04037. Cited by: §B.3, §D.1. - A. Doğru, M. O. Keskin, C. M. Jonker, T. Baarslag, and R. Aydoğan (2024) NegoLog: an integrated python-based automated negotiation framework with enhanced assessment components. IJCAI 2024. External Links: Link Cited by: §2.3. - Y. Fehlis, C. Crain, A. Jensen, M. Watson, J. Juhasz, P. Mandel, B. Liu, S. Mahon, D. Wilson, and N. Lynch-Jonely (2025) Accelerating drug discovery through agentic ai: a multi-agent approach to laboratory automation in the dmta cycle. arXiv.org. External Links: Document, Link Cited by: §2.2. - P. Gajjar (2025) EvoResearch: a multi-agent ai framework for automated paper analysis. International Journal of Innovative Research in Advanced Engineering. External Links: Document, Link Cited by: §2.1. - R. Gong, S. Bai, S. Wu, Y. Fan, Z. Wang, X. Li, H. Yang, and X. Liu (2025) Past-future scheduler for llm serving under sla guarantees. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, pp. 798–813. Cited by: §4.1. - D. Han, Z. Ai, P. Cai, S. Lu, J. Chen, Z. Ye, S. Sun, B. Gao, L. Ge, W. Wang, et al. (2025a) ChemBOMAS: accelerated bo in chemistry with llm-enhanced multi-agent system. arXiv preprint arXiv:2509.08736. Cited by: §2.2. - R. Han, Y. Chen, Z. CuiZhu, L. Miculicich, G. Sun, Y. Bi, W. Wen, H. Wan, C. Wen, S. Maître, et al. (2025b) Deep researcher with test-time diffusion. arXiv preprint arXiv:2507.16075. Cited by: §3.3. - B. Hui, J. Yang, Z. Cui, J. Yang, D. Liu, L. Zhang, T. Liu, J. Zhang, B. Yu, K. Dang, et al. (2024) Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186. Cited by: Table 3, Table 3, Table 3, Table 3. - Y. Jin, Q. Zhao, Y. Wang, H. Chen, K. Zhu, Y. Xiao, and J. Wang (2024) Agentreview: exploring peer review dynamics with llm agents. arXiv preprint arXiv:2406.12708. Cited by: §3.6. - J. Ju, Y. ZHENG, H. Y. Koh, C. Wang, and S. Pan (2025) ChemThinker: thinking like a chemist with multi-agent llms for deep molecular insights. ICLR 2025. External Links: Link Cited by: §2.2. - M. O. Keskin, B. Buzcu, B. Koçyiğit, U. Çakan, A. Doğru, and R. Aydoğan (2024) NEGOTIATOR: a comprehensive framework for human-agent negotiation integrating preferences, interaction, and emotion. IJCAI 2024. External Links: Link Cited by: §2.3. - A. S. Korat (2025) Synergistic minds: a collaborative multi-agent framework for integrated ai tool development using diverse large language models. World Journal of Advanced Research and Reviews. External Links: Document, Link Cited by: §3.4. - S. Kumbhar, V. Mishra, K. Coutinho, D. Handa, A. Iquebal, and C. Baral (2025) Hypothesis generation for materials discovery and design using goal-driven and constraint-guided llm agents. NAACL 2025. External Links: Link Cited by: §2.2. - J. Lála, O. O’Donoghue, A. Shtedritski, S. Cox, S. G. Rodriques, and A. D. White (2023) PaperQA: retrieval-augmented generative agent for scientific research. arXiv preprint arXiv:2312.07559. External Links: Link Cited by: Table 1, Table 1. - H. D. Le, X. Xia, and C. Zhang (2025) Multi-agent causal discovery using large language models. ICLR 2025. External Links: Link Cited by: §2.1. - W. Li, J. Lin, Z. Jiang, J. Cao, X. Liu, J. Zhang, Z. Huang, Q. Chen, W. Sun, Q. Wang, et al. (2025) Chain-of-agents: end-to-end agent foundation models via multi-agent distillation and agentic rl. arXiv preprint arXiv:2508.13167. Cited by: §2.1. - M. Mishra, M. Stallone, G. Zhang, Y. Shen, A. Prasad, A. M. Soria, M. Merler, P. Selvam, S. Surendran, S. Singh, et al. (2024) Granite code models: a family of open foundation models for code intelligence. arXiv preprint arXiv:2405.04324. Cited by: Table 3. - V. Naumov, D. Zagirova, S. Lin, Y. Xie, W. Gou, A. Urban, N. Tikhonova, K. M. Alawi, M. Durymanov, and F. Galkin (2025) DORA ai scientist: multi-agent virtual research team for scientific exploration discovery and automated report generation. bioRxiv. External Links: Document, Link Cited by: item 3, §1, §2.1. - S. Niyakan and X. Qian (2025) PhenoGraph: a multi-agent framework for phenotype-driven discovery in spatial transcriptomics data augmented with knowledge graphs. bioRxiv. External Links: Document, Link Cited by: §2.2. - A. Pramanick, Y. Hou, S. M. Mohammad, and I. Gurevych (2023) A diachronic analysis of paradigm shifts in nlp research: when, how, and why?. EMNLP 2023. External Links: Link Cited by: §1. - Y. Pu, T. Lin, and H. Chen (2025) PiFlow: principle-aware scientific discovery with multi-agent collaboration. arXiv preprint arXiv:2505.15047. Cited by: §2.3. - R. Qu, R. Tu, and F. Bao (2025) Is semantic chunking worth the computational cost?. In Findings of the Association for Computational Linguistics: NAACL 2025, pp. 2155–2177. Cited by: §3.4. - C. K. Reddy and P. Shojaee (2025) Towards scientific discovery with generative ai: progress, opportunities, and challenges. AAAI 2025. External Links: Link Cited by: §1, §3.1. - A. Roucher, A. V. del Moral, T. Wolf, L. von Werra, and E. Kaunismäki (2025) ‘Smolagents‘: a smol library to build great agentic systems.. Note: https://github.com/huggingface/smolagents Cited by: §3.1, §3.6. - A. Salemi, M. Parmar, P. Goyal, Y. Song, J. Yoon, H. Zamani, H. Palangi, and T. Pfister (2025) Llm-based multi-agent blackboard system for information discovery in data science. arXiv preprint arXiv:2510.01285. Cited by: §2.3. - Y. Shao, Y. Jiang, T. Kanell, P. Xu, O. Khattab, and M. Lam (2024) Assisting in writing Wikipedia-like articles from scratch with large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), K. Duh, H. Gomez, and S. Bethard (Eds.), Mexico City, Mexico, pp. 6252–6278. External Links: Link, Document Cited by: Table 1. - X. Shi, Q. Kou, Y. Li, N. Tang, J. Xie, L. Yu, S. Wang, and H. Zhou (2025) Scisage: a multi-agent framework for high-quality scientific survey generation. arXiv preprint arXiv:2506.12689. Cited by: Table 1. - Er. J. Singh and P. Kumar (2025) ASTRAFIN:- ai financial agent. INTERNATIONAL JOURNAL OF SCIENTIFIC RESEARCH IN ENGINEERING AND MANAGEMENT. External Links: Document, Link Cited by: §2.2. - J. Spieser, A. Balapour, J. Meller, K. Patra, and B. Shamsaei (2025) Multi-agent ai systems for biological and clinical data analysis. Preprints.org. External Links: Document, Link Cited by: §2.2. - Q. Team (2025) Qwen3 technical report. External Links: 2505.09388, Link Cited by: Table 3. - W. Wang, F. Wei, L. Dong, H. Bao, N. Yang, and M. Zhou (2020) Minilm: deep self-attention distillation for task-agnostic compression of pre-trained transformers. Advances in neural information processing systems 33, pp. 5776–5788. Cited by: §3.3, §4.1, §4.1, Table 3. - X. Wang, K. Zhou, W. Wu, H. S. Singh, F. Nan, S. Jin, A. Philip, S. Patnaik, H. Zhu, S. Singh, et al. (2025) Causal-copilot: an autonomous causal analysis agent. arXiv preprint arXiv:2504.13263. Cited by: §2.1. - M. Wawer and J. A. Chudziak (2025) Integrating traditional technical analysis with ai: a multi-agent llm-based approach to stock market forecasting. International Conference on Agents and Artificial Intelligence. External Links: Document, Link Cited by: §2.2. - M. Wooldridge (2002) An introduction to multiagent systems. John Wiley & Sons. Cited by: §3.1. - M. Xiao, B. Liu, H. Li, J. Huang, Q. Xie, X. Zong, M. Ye, and M. Peng (2025) MoodAngels: a retrieval-augmented multi-agent framework for psychiatry diagnosis. NIPS 2025. External Links: Link Cited by: §2.2. - Y. Xiao, J. Liu, Y. Zheng, X. Xie, J. Hao, M. Li, R. Wang, F. Ni, Y. Li, J. Luo, et al. (2024) Cellagent: an llm-driven multi-agent framework for automated single-cell data analysis. arXiv preprint arXiv:2407.09811. Cited by: §2.2. - Z. Yang, W. Liu, B. Gao, T. Xie, Y. Li, W. Ouyang, S. Poria, E. Cambria, and D. Zhou (2025) MOOSE-chem: large language models for rediscovering unseen chemistry scientific hypotheses. ICLR 2025. External Links: Link Cited by: §2.2. - H. Yu, Z. Cheng, Z. Hong, K. Zhu, J. Yao, T. Feng, and J. You (2025a) Research town: simulator of research community. ICLR 2025. External Links: Link Cited by: §2.3. - H. Yu, Z. Hong, Z. Cheng, K. Zhu, K. Xuan, J. Yao, T. Feng, and J. You (2025b) ResearchTown: simulator of human research community. ICML 2025. External Links: Link Cited by: §2.3. - Z. Yu, K. Feng, Y. Zhao, S. He, X. Zhang, and A. Cohan (2025c) AlphaResearch: accelerating new algorithm discovery with language models. arXiv preprint arXiv:2511.08522. Cited by: §2.1. - Y. Yuan and T. Xie (2025) Reinforce llm reasoning through multi-agent reflection. arXiv preprint arXiv:2506.08379. Cited by: §2.3. - B. Zhang, Y. He, L. Pintscher, A. M. Peñuela, and E. Simperl (2025a) Schema generation for large knowledge graphs using large language models. arXiv preprint arXiv:2506.04512. Cited by: §3.4. - Y. Zhang, M. Li, D. Long, X. Zhang, H. Lin, B. Yang, P. Xie, A. Yang, D. Liu, J. Lin, F. Huang, and J. Zhou (2025b) Qwen3 embedding: advancing text embedding and reranking through foundation models. arXiv preprint arXiv:2506.05176. Cited by: §4.1, §4.1. - Q. Zhu, D. Guo, Z. Shao, D. Yang, P. Wang, R. Xu, Y. Wu, Y. Li, H. Gao, S. Ma, et al. (2024a) DeepSeek-coder-v2: breaking the barrier of closed-source models in code intelligence. arXiv preprint arXiv:2406.11931. Cited by: Table 3, Table 3. - Y. Zhu, X. Wang, J. Chen, S. Qiao, Y. Ou, Y. Yao, S. Deng, H. Chen, and N. Zhang (2024b) Llms for knowledge graph construction and reasoning: recent capabilities and future opportunities. World Wide Web 27 (5), pp. 58. Cited by: §3.5. ## Appendix A Paper Review Results We evaluate how well large language models can predict human paper-review scores on ICLR submissions. From the ICLR 2024 dataset, we randomly sampled 50 papers to cover a broad range of human-assigned ratings and evaluated four tool-enabled LLMs: gpt-oss:120b, gpt-oss:20b, qwen3-coder-30b, and a quantized qwen3-coder-30b variant. For each paper, the model produces numerical scores for standard review dimensions (overall rating, soundness, presentation, and contribution), which we compare against the corresponding human scores. #### Metrics. We report regression error (MSE, MAE, RMSE), rank/linear association (Pearson, Spearman), and thresholded accuracy (percentage of predictions within $± 0.5$ , $± 1.0$ , and $± 1.5$ of the human score). We also report the mean and standard deviation of signed errors to characterize systematic bias. Due to occasional missing fields or filtering during preprocessing, the number of evaluated papers $N$ can differ slightly across models. #### Key findings. Across categories, gpt-oss:120b achieves the best overall accuracy on rating and contribution (e.g., rating MAE $=1.68$ ; contribution MAE $=0.62$ ), while gpt-oss:20b is competitive and often stronger on more technical sub-scores such as soundness and presentation. Despite moderate absolute errors on several dimensions, correlations with human scores remain weak across models (generally $|r|<0.25$ ), suggesting that models struggle to preserve the relative ranking of papers even when their average deviation is limited. Code-specialized models (Qwen3-Coder) remain viable baselines, but show larger errors on overall rating and contribution in this setting. | oss-120B | RATING | 4.6934 | 1.6844 | 2.1664 | -0.0407 | 0.0571 | 25.00% | 43.75% | 58.33% | 0.2177 | 2.1555 | 48 | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | oss-120B | SOUNDNESS | 0.7316 | 0.6351 | 0.8554 | -0.0054 | 0.0474 | 58.33% | 85.42% | 87.50% | -0.0816 | 0.8515 | 48 | | oss-120B | PRESENTATION | 0.6564 | 0.6038 | 0.8102 | 0.0701 | 0.1259 | 60.42% | 83.33% | 91.67% | -0.0920 | 0.8049 | 48 | | oss-120B | CONTRIBUTION | 0.6349 | 0.6240 | 0.7968 | 0.0717 | 0.0734 | 56.25% | 85.42% | 91.67% | 0.0087 | 0.7967 | 48 | | oss-20 | RATING | 4.7607 | 1.7647 | 2.1819 | 0.0989 | 0.1869 | 21.43% | 40.48% | 52.38% | 1.5980 | 1.4856 | 42 | | oss-20 | SOUNDNESS | 0.4241 | 0.5190 | 0.6512 | -0.0106 | -0.0226 | 59.52% | 92.86% | 97.62% | 0.3294 | 0.5618 | 42 | | oss-20 | PRESENTATION | 0.4271 | 0.5171 | 0.6535 | -0.1270 | -0.1299 | 64.29% | 90.48% | 97.62% | 0.3512 | 0.5511 | 42 | | oss-20 | CONTRIBUTION | 0.6482 | 0.6702 | 0.8051 | 0.2221 | 0.1757 | 50.00% | 83.33% | 97.62% | 0.6250 | 0.5075 | 42 | | qwen30B-code_qk_3 | RATING | 11.8533 | 2.9879 | 3.4429 | -0.2233 | -0.2837 | 8.51% | 17.02% | 29.79% | 2.9085 | 1.8422 | 47 | | qwen30B-code_qk_3 | SOUNDNESS | 1.6941 | 1.1730 | 1.3016 | 0.0113 | -0.0096 | 17.02% | 46.81% | 72.34% | 1.1454 | 0.6182 | 47 | | qwen30B-code_qk_3 | PRESENTATION | 1.4257 | 1.0191 | 1.1940 | 0.0378 | 0.0271 | 27.66% | 59.57% | 78.72% | 0.9787 | 0.6840 | 47 | | qwen30B-code_qk_3 | CONTRIBUTION | 2.2921 | 1.3865 | 1.5140 | 0.0196 | 0.0224 | 12.77% | 34.04% | 65.96% | 1.3865 | 0.6080 | 47 | | Qwen 30B | RATING | 10.2331 | 2.7930 | 3.1989 | -0.1820 | -0.2216 | 7.89% | 13.16% | 26.32% | 2.6930 | 1.7266 | 38 | | Qwen 30B | SOUNDNESS | 1.7172 | 1.2096 | 1.3104 | -0.1157 | -0.1057 | 13.16% | 39.47% | 73.68% | 1.1491 | 0.6298 | 38 | | Qwen 30B | PRESENTATION | 0.9526 | 0.7180 | 0.9760 | -0.1319 | -0.1495 | 55.26% | 73.68% | 81.58% | 0.6522 | 0.7261 | 38 | | Qwen 30B | CONTRIBUTION | 2.5212 | 1.4746 | 1.5878 | -0.2119 | -0.2160 | 13.16% | 26.32% | 55.26% | 1.4640 | 0.6146 | 38 | Table 8: Paper review score prediction on ICLR 2024. We compare four LLMs on predicting human review scores across rating, soundness, presentation, and contribution. We report error metrics (MSE/MAE/RMSE), correlation (Pearson/Spearman), and thresholded accuracy (within $± 0.5$ , $± 1.0$ , $± 1.5$ of the human score). $N$ denotes the number of papers evaluated for each model after preprocessing. ## Appendix B System Overview Paper Circle is a full-stack platform with a web frontend and a Python backend as shown in the Figure LABEL:fig:front_clint. The frontend (React, TypeScript, Vite, TailwindCSS) provides discovery, reading circles, and discussion features. The backend exposes discovery APIs via FastAPI and implements the multi-agent pipelines used by the system. Supabase (PostgreSQL + Auth) provides storage for users, communities, papers, and sessions. The discovery backend includes two major pipelines: (i) a refactored research discovery pipeline focused on deterministic retrieval, scoring, and diversity, and (ii) a multi-agent research pipeline that produces structured step-by-step outputs with offline search support. Both pipelines are accessible through API endpoints and are integrated into the Paper Circle user interface for interactive discovery workflows. Figure LABEL:fig:discovery_front illustrates the overall architecture of Paper Circle. The system consists of two complementary multi-agent pipelines: the Discovery Pipeline for finding relevant papers, and the Analysis Pipeline for deep understanding of individual papers. The discovery pipeline, as shown in the Figure LABEL:fig:discovery_front is composed of six agents: intent classification, paper search, sorting, analysis, export, and web search. The intent classifier parses natural-language queries into structured constraints (search mode, conferences, year range, max results, and ranking preferences). The paper search agent is the primary retrieval worker; it updates the global state and writes outputs after every search step. The sorting and analysis agents operate on the shared paper list to refine ranking and derive insights. The export agent centralizes output access for downstream workflows, while the web search agent supplements the pipeline with external lookup tools when required. All agents are coordinated by the CodeAgent, which enforces a minimal-step policy for efficiency and uses the intent classifier to decide offline versus online search. The analysis pipeline operates on individual papers, transforming PDF documents into structured knowledge graphs. It employs four specialized extraction agents (concept, method, experiment, and linkage) that process paper content in phases, building a typed graph with full traceability to source locations. The resulting graph supports question answering, coverage verification, and multi-format export. ### B.1 State Management and Outputs State is maintained in PipelineState. Each step increments a counter, logs action metadata, and regenerates synchronized artifacts. The outputs include: (i) papers.json with full paper metadata and computed scores, (ii) links.json with structured links and PDFs/DOIs, (iii) stats.json with aggregate statistics and a leaderboard, (iv) summary.json with generated insights and key findings, (v) retrieval_metrics.json when evaluation is enabled, and (vi) human-readable exports (CSV, BibTeX, Markdown) plus a live HTML dashboard. This approach ensures that each agent step is reproducible and auditable. ### B.2 Retrieval The pipeline supports both offline and online retrieval. Offline search loads papers from a local JSON corpus and optionally filters by conference and year. It ranks results using BM25 by default, with optional semantic similarity (sentence transformers) or hybrid scoring when available. An optional cross-encoder reranker can refine the top results; when enabled, it reranks a first-stage candidate set. Online search aggregates results from arXiv, Semantic Scholar, OpenAlex, and DBLP via their public APIs. A query intent classifier detects search mode, conference constraints, year ranges, and ranking preferences, and routes the query to the appropriate retrieval pathway. Deduplication is applied across sources by normalizing titles. ### B.3 Ranking and Scoring After retrieval, papers are scored along multiple axes: recency, similarity to the query (TF–IDF Das et al. (2023) when available), novelty based on title token frequency, and normalized BM25 scores Chen and Wiseman (2023). The system supports sorting by any single criterion or by a weighted combined score. Relevance scores are computed as a weighted mixture of similarity, recency, citation count, and BM25. Final ranks are assigned after sorting, and the updated ordering is reflected in all exported artifacts. When reranker-based sorting is requested, a cross-encoder replaces the default scoring with direct relevance scores. ### B.4 Analysis and Monitoring The pipeline computes aggregate statistics such as source distribution, year distribution, top authors and venues, keyword frequency, and citation summaries. These analytics populate structured summaries and are visualized in an auto-refreshing HTML dashboard. Each agent action is logged with timestamps and paper counts, enabling reproducibility and step-level auditing of the pipeline. The pipeline also maintains a step log that captures the agent name, action, results preview, and parameters used. <details> <summary>2604.06170v1/figures/analysis_frontend.png Details</summary>  ### Visual Description ## Relational Database Schema Diagram: Academic Paper Management System ### Overview The diagram illustrates a relational database schema for an academic paper management system, depicting seven core tables (`users`, `papers`, `communities`, `paper_engagement`, `paper_analysis`, `community_papers`, `paper_discussions`) and their relationships. Tables are connected via labeled lines representing business logic (e.g., "writes," "has," "receives"). --- ### Components/Axes #### Tables and Columns 1. **users** - Columns: `uuid` (PK), `id`, `email` (string), `full_name` (string), `avatar_url` (string) - Primary Key: `uuid` 2. **papers** - Columns: `uuid` (PK), `id`, `title` (string), `abstract` (text), `url` (string), `pdf_url` (string), `arxiv_id` (string), `doi` (string), `authors` (JSON), `year` (int), `venue` (string), `citation_count` (int) - Primary Key: `uuid` 3. **communities** - Columns: `uuid` (PK), `id`, `name` (string), `slug` (string), `description` (string), `image_url` (string) - Primary Key: `uuid` 4. **paper_engagement** - Columns: `uuid` (PK), `user_id` (FK to `users.uuid`), `paper_id` (FK to `papers.uuid`), `engagement_type` (string: like, view, save), `created_at` (timestamp) - Foreign Keys: `user_id`, `paper_id` 5. **paper_analysis** - Columns: `uuid` (PK), `paper_id` (FK to `papers.uuid`), `analysis_data` (JSON: Nodes & Edges), `markdown_summary` (text), `mindmap_mermaid` (text), `flowchart_mermaid` (text), `created_at` (timestamp) - Foreign Keys: `paper_id` 6. **community_papers** - Columns: `uuid` (PK), `community_id` (FK to `communities.uuid`), `paper_id` (FK to `papers.uuid`), `added_at` (timestamp) - Foreign Keys: `community_id`, `paper_id` 7. **paper_discussions** - Columns: `uuid` (PK), `paper_id` (FK to `papers.uuid`) - Foreign Keys: `paper_id` #### Relationships - **users** ↔ **papers**: 1-to-many via "writes" (users write papers) - **papers** ↔ **communities**: 1-to-many via "belongs_to" (papers belong to communities) - **papers** ↔ **paper_engagement**: 1-to-many via "has" (papers have engagement records) - **papers** ↔ **paper_analysis**: 1-to-1 via "has_one" (papers have analysis records) - **communities** ↔ **community_papers**: 1-to-many via "contains" (communities contain papers) - **papers** ↔ **paper_discussions**: 1-to-many via "has" (papers have discussions) --- ### Detailed Analysis - **users**: Stores user metadata (email, name, avatar URL). - **papers**: Central table with metadata (title, abstract, URLs) and structured data (authors as JSON, citation count). - **communities**: Represents academic groups with descriptive fields (name, slug, image URL). - **paper_engagement**: Tracks user interactions (likes, views, saves) with timestamps. - **paper_analysis**: Stores computational analysis outputs (mindmaps, flowcharts) and summaries. - **community_papers**: Links communities to papers with timestamps for inclusion. - **paper_discussions**: Junction table for paper-specific discussions (many-to-many relationship implied). --- ### Key Observations 1. **Normalization**: Tables are normalized to 3NF, with minimal redundancy (e.g., `authors` stored as JSON in `papers`). 2. **Temporal Tracking**: Timestamps (`created_at`, `added_at`) enable audit trails for engagement and community inclusion. 3. **Modular Analysis**: `paper_analysis` decouples raw data from derived insights (e.g., Mermaid diagrams for visualizations). 4. **Community-Paper Relationships**: Many-to-many via `community_papers`, allowing papers to belong to multiple communities. --- ### Interpretation This schema supports a collaborative academic ecosystem where: - **Users** contribute **papers** to **communities**, which curate and discuss them. - **Engagement metrics** (likes, views) and **analysis outputs** (summaries, visualizations) enhance paper discoverability and utility. - **Temporal data** (`created_at`, `added_at`) allows tracking of paper lifecycle and community adoption. - **JSON fields** (`authors`, `analysis_data`) accommodate semi-structured data, balancing flexibility with relational integrity. The design prioritizes scalability for large datasets (e.g., citation counts, community memberships) while enabling complex queries (e.g., "find papers with >100 citations in neuroscience communities"). </details> Figure 7: Paper analysis and database management for fast inference. ## Appendix C Retrieval Pipeline Paper Circle supports both offline and online retrieval to balance coverage, speed, and reproducibility. The choice between retrieval modes is controlled by the intent classification agent, which parses user queries to determine the optimal search strategy. ### C.1 Offline Retrieval The OfflinePaperSearchEngine enables fast (See the Figure 7, reproducible search over a local database of academic papers stored as JSON files. Each database file contains structured paper metadata including title, authors, abstract, venue, year, track, keywords, and DOI. The offline search process: 1. Database Loading: Papers are loaded from the specified database path with optional filtering by conference (e.g., ICLR, NeurIPS, ACL) and year range. 1. Text Preparation: For each paper, searchable text is constructed by concatenating the title, abstract, and keywords. 1. BM25 Indexing: When available, papers are indexed using the Okapi BM25 algorithm via the rank_bm25 library. The index uses tokenized documents for sparse retrieval. 1. Query Execution: User queries are tokenized and scored against the BM25 index, returning a ranked list of candidates. An optional cross-encoder reranker can refine the top- $k$ results from the first-stage retrieval. When enabled via the AdvancedReranker module, the system uses a transformer-based reranker (e.g., Qwen3-Reranker) to compute more precise relevance scores between the query and candidate documents. ### C.2 Online Retrieval For broader or more current searches, Paper Circle aggregates results from multiple academic APIs: - arXiv: Queries the arXiv API for preprints, extracting title, authors, abstract, categories, and PDF links. - Semantic Scholar: Retrieves papers with citation counts, abstracts, and venue information via the Semantic Scholar Academic Graph API. - OpenAlex: Accesses the OpenAlex catalog for open-access metadata and citation networks. - DBLP: Searches the DBLP computer science bibliography for venue-specific results. Each source is queried in parallel using a thread pool executor for efficiency. Results are normalized into the common Paper data structure before merging. ### C.3 Deduplication After retrieval, the pipeline performs two-stage deduplication to eliminate redundant entries: 1. DOI-based deduplication: Papers with matching DOIs are deduplicated, preferring entries with richer metadata (e.g., abstracts, PDF URLs). 1. Title-based deduplication: Titles are normalized by removing punctuation and converting to lowercase. Duplicate titles are merged, again preferring metadata-complete entries. The deduplication step is critical when aggregating results from multiple sources, as the same paper often appears in arXiv, Semantic Scholar, and OpenAlex with varying metadata quality. ### C.4 Query Expansion The query generation agent converts natural-language user input into a structured search specification containing: - Core keywords: Primary search terms extracted from the query. - Required constraints: Mandatory terms that must appear in results. - Related terms: Synonyms or related concepts to expand recall. - Negative keywords: Terms to exclude from results. - Plausible paper titles: Hypothesized titles for targeted retrieval. This structured specification enables consistent query construction across heterogeneous data sources while capturing user intent more precisely than raw keyword matching. ## Appendix D Scoring and Ranking Paper Circle employs a multi-criteria scoring framework designed for research discovery rather than general information retrieval. Each paper receives scores along multiple dimensions, which are combined using mode-specific weights to produce a final ranking. ### D.1 Scoring Dimensions The system computes the following scores for each retrieved paper: #### Similarity Score Relevance to the user query is computed using TF–IDF Das et al. (2023) vectorization and cosine similarity. The query and paper text (concatenated title and abstract) are transformed into TF–IDF vectors using scikit-learn’s TfidfVectorizer. The similarity score is the cosine of the angle between these vectors: $$ similarity(q,p)=\frac{\vec{v}_q·\vec{v}_p}{\|\vec{v}_q\|·\|\vec{v}_p\|} \tag{1} $$ where $\vec{v}_q$ and $\vec{v}_p$ are the TF–IDF vectors for the query and paper, respectively. #### Recency Score Papers are scored by publication year, with more recent papers receiving higher scores. The recency score is normalized relative to the current year: $$ recency(p)=\frac{year(p)-year_\min}{year_\max-year_\min} \tag{2} $$ where $year_\min$ and $year_\max$ are the minimum and maximum years in the corpus. #### Novelty Score Novelty measures how different a paper is from the corpus centroid, computed as the TF–IDF distance from the average document vector. Papers with unusual terminology or unique topic combinations receive higher novelty scores, surfacing potentially overlooked works. #### BM25 Score When the rank_bm25 library is available, the Okapi BM25 algorithm provides an alternative relevance measure that accounts for term frequency saturation and document length normalization. BM25 scores are normalized to the $[0,1]$ range for comparability with other dimensions. #### Citation Count When available from the source API (primarily Semantic Scholar and OpenAlex), citation counts provide a proxy for impact. Citation-based ranking is optional and disabled by default to avoid recency bias against new papers. ### D.2 Combined Score Computation The final combined score is a weighted sum of individual dimensions: $$ combined(p)=w_s·similarity+w_r·recency+w_n·novelty+w_b·bm25 \tag{3} $$ The weights $(w_s,w_r,w_n,w_b)$ are determined by the search mode: - Stable mode: Prioritizes relevance and authority. Weights: $w_s=0.5$ , $w_r=0.2$ , $w_n=0.1$ , $w_b=0.2$ . - Discovery mode: Prioritizes novelty to surface non-obvious results. Weights: $w_s=0.3$ , $w_r=0.1$ , $w_n=0.4$ , $w_b=0.2$ . - Balanced mode: Equal emphasis across dimensions. Weights: $w_s=0.3$ , $w_r=0.2$ , $w_n=0.2$ , $w_b=0.3$ . Users can override these weights at query time via API parameters, enabling custom relevance trade-offs for specific research contexts. ### D.3 Sorting Stage After scoring, the sorting agent reorders papers according to user preferences. Supported sort criteria include: - recency: Most recent papers first. - citations: Highest-cited papers first. - similarity: Most relevant papers first. - novelty: Most unusual papers first. - bm25: Best BM25 matches first. - combined: Weighted combined score (default). ### D.4 Cross-Encoder Reranking For high-precision use cases, the pipeline supports optional cross-encoder reranking. When enabled, a transformer-based reranker (configured via RerankerConfig) processes query-document pairs through a cross-attention model to compute more accurate relevance scores than first-stage retrieval alone. The MultiStageRetriever first retrieves a larger candidate set (e.g., top-200) using BM25, then reranks to produce the final top- $k$ results. This two-stage approach balances efficiency with ranking quality. ## Appendix E Diversity and Postprocessing Relevance-based ranking alone can produce homogeneous results, with multiple papers covering similar topics or methods. Paper Circle addresses this through diversity-aware postprocessing that ensures the top results span a broader range of perspectives. ### E.1 Maximal Marginal Relevance To improve topical coverage, Paper Circle applies Maximal Marginal Relevance (MMR) to the candidate list after initial scoring. MMR iteratively selects papers that maximize a combination of relevance to the query and dissimilarity to already-selected papers: $$ MMR=\arg\max_p∈ R∖ S≤ft[λ·sim(p,q)-(1-λ)·\max_s∈ Ssim(p,s)\right] \tag{4} $$ where $R$ is the candidate set, $S$ is the set of already-selected papers, $q$ is the query, and $λ$ controls the relevance–diversity trade-off. The diversity parameter $λ$ is mode-dependent: - Stable mode: $λ=0.8$ (relevance-focused). - Discovery mode: $λ=0.5$ (diversity-focused). - Balanced mode: $λ=0.65$ . Similarity between papers is computed using TF–IDF cosine similarity over concatenated title and abstract text. This ensures that top results cover distinct subtopics rather than repeating variations of the same idea. ### E.2 Secondary Views The pipeline constructs specialized views over the ranked list to serve different discovery goals: #### Hidden Gems Papers with high novelty scores but moderate relevance scores are surfaced as “hidden gems.” These are papers that may not rank highly on traditional relevance metrics but offer unique perspectives or cover underexplored topics. The hidden gems view is computed by sorting papers by novelty score and filtering for those below rank 20 in the combined ranking. #### Canonical Papers Papers with high citation counts or appearing in top-tier venues are flagged as “canonical” works. This view helps users identify foundational papers in a research area, complementing the recency-focused main ranking. #### Source Distribution The postprocessing stage also reports the distribution of papers across sources (arXiv, Semantic Scholar, etc.), enabling users to assess coverage and identify potential gaps in the retrieval. ### E.3 Statistics and Analytics After ranking, the analysis agent computes aggregate statistics stored in stats.json: - Year distribution: Paper counts by publication year. - Source distribution: Paper counts by retrieval source. - Top authors: Authors appearing most frequently in results. - Top venues: Conferences and journals with highest representation. - Keyword frequency: Most common terms in paper titles. - Citation statistics: Total, average, median, min, and max citation counts. - Score statistics: Average similarity, novelty, recency, and BM25 scores. These analytics are visualized in an auto-refreshing HTML dashboard that updates every 10 seconds during pipeline execution, providing real-time visibility into the discovery process. ### E.4 Insight Generation The pipeline automatically generates human-readable insights from the collected data: - Publication trends: Identifies the year with the most publications. - Primary source: Reports which API contributed the most results. - Prolific authors: Highlights researchers with multiple papers in the collection. - Citation leaders: Identifies the most-cited paper. - Hot topics: Lists the most frequent keywords. - Open access availability: Reports the percentage of papers with direct PDF links. These insights are stored in summary.json and displayed on the dashboard, helping users quickly understand the landscape of retrieved literature. ## Appendix F Outputs and Interfaces The pipeline maintains synchronized structured outputs after every agent step. The primary artifacts include: - papers.json: Full paper metadata and scores. - links.json: Structured links and PDF/DOI entries. - stats.json: Aggregate statistics and leaderboards. - summary.json: Insights and key findings. - retrieval_metrics.json: Step-level evaluation metrics. Additional exports include CSV, BibTeX, Markdown, and an auto-refreshing HTML dashboard. These outputs allow the same discovery session to be used for curation, citation management, and reporting. The system exposes REST APIs via FastAPI. The discovery endpoint accepts a query and mode, returns structured search specifications, and provides the full ranked list with scores. Mode weights can be queried or overridden at runtime, enabling customized relevance/authority/novelty trade-offs. ## Appendix G Evaluation We evaluate Paper Circle along three axes: (i) retrieval effectiveness under different configurations, (ii) stability and reproducibility of rankings across steps, and (iii) the utility of diversity-aware postprocessing for surfacing non-redundant results. Paper Circle provides built-in evaluation metrics but does not enforce a fixed benchmark dataset. When a ground-truth paper title or identifier is provided, the system computes Mean Reciprocal Rank (MRR), Recall@K, Precision@K, and hit rates. These metrics are computed per step and stored in JSON file for longitudinal tracking. As a minimal illustrative scenario, consider a known target paper in the local corpus: the pipeline is run once using offline retrieval and once using online sources. The resulting MRR and Recall@K values allow direct comparison of configuration impact, while repeated runs confirm stable rankings when deterministic scoring is enabled. Although lightweight, this framing aligns evaluation with discovery goals rather than task-specific QA benchmarks. For batch evaluation, a parallel benchmarking utility executes multiple queries concurrently and aggregates mean metrics and timing statistics. This supports lightweight comparisons between search configurations (offline vs. online, BM25 vs. semantic, with or without reranking) without requiring external tooling. #### Knowledge Graph Schema. The mind graph follows a typed schema with nodes for papers, sections, concepts, methods, experiments, datasets, and visual elements (figures, tables, equations), and edges encoding structural and semantic relations such as hierarchy, definition, proposal, usage, evaluation, illustration, and dependency. Each node and edge is annotated with provenance metadata, including source chunk IDs, page numbers, verification status, confidence scores, and timestamps, providing full traceability from any graph element back to the original PDF. ### G.1 Multi-Agent Extraction The GraphBuilder orchestrates four specialized extraction agents, each implemented as a CodeAgent with domain-specific instructions: #### Concept Extractor Identifies key concepts from text chunks, classifying each by type (definition, technique, theory, phenomenon) and importance (core, supporting, background). The agent outputs structured JSON with concept names, descriptions, and classifications. #### Method Extractor Focuses on sections containing method-related keywords (“method”, “approach”, “architecture”, “algorithm”). For each method, it extracts the name, description, category (proposed, baseline, component), and key steps. #### Experiment Extractor Processes experiment sections to extract experimental setups, datasets used, evaluation metrics, and key results. It also identifies dataset nodes for cross-referencing. #### Linkage Agent Connects figures and tables to the concepts and methods they illustrate. Given a figure caption, nearby text, and a list of existing concepts, the agent determines which concepts the figure relates to and the type of relationship (illustrates, summarizes, compares, demonstrates). The extraction proceeds in five phases: (1) concept extraction from body chunks, (2) method extraction from method sections, (3) experiment and dataset extraction, (4) figure and table linkage, and (5) inter-concept relationship discovery. Each phase updates the shared MindGraph data structure. ### G.2 Graph-Aware Q&A The Q&A system combines vector-based retrieval with graph traversal. The EmbeddingStore indexes both text chunks and node descriptions using sentence-transformers (with a simple bag-of-words fallback when unavailable). Given a question, the GraphRetriever: 1. Retrieves the top- $k$ most similar chunks and nodes. 1. Expands context by including 1-hop graph neighbors. 1. Returns chunks, nodes, and connecting edges. The PaperQA agent constructs a prompt with the retrieved context, including text chunks with their section sources, relevant concept descriptions, and graph relationships. The response includes the answer, supporting sections, relevant figures and tables, and a confidence estimate. A locate function allows users to find where specific items are discussed in the paper by searching across nodes, figures, tables, and text chunks, returning page numbers and context snippets. ### G.3 Coverage Verification To ensure nothing is silently dropped during extraction, the CoverageChecker produces a detailed coverage report: - Figure coverage: How many figures are linked to concepts or methods. - Table coverage: How many tables are linked to results or experiments. - Section coverage: How many sections have extracted concepts. - Equation coverage: How many equations are linked to concepts they define. The report includes an overall coverage score (0–100%), lists of unlinked items with suggestions, and critical issues (e.g., “No figures are linked to concepts/methods”). This enables quality assurance before downstream use. ### G.4 Human Verification Workflow The VerificationManager supports human-in-the-loop review: - verify_node: Mark a node as human-verified. - edit_node: Modify node title or description. - add_edge: Create new relationships. - remove_edge: Delete incorrect relationships. - flag_for_review: Flag nodes for review with a reason. Each action is logged with timestamps, maintaining a complete edit history. Nodes carry a verification_status field (auto-generated, human-verified, human-edited, or flagged) that propagates through exports. ### G.5 Export Formats The system exports to multiple formats for different use cases: - JSON: Full graph data including nodes, edges, chunks, and metadata. - Markdown: Structured reading notes with section outlines. - Mermaid: Mind maps and flowcharts for visualization. - HTML: Interactive D3.js-based graph visualization. All exports preserve traceability metadata, enabling users to navigate from any extracted element back to the original source. ## Appendix H Implementation and Deployment The backend is implemented in Python with FastAPI for service endpoints and relies on standard scientific libraries for retrieval and scoring (scikit-learn, NumPy, pandas). The multi-agent pipeline is defined in textttbackend/agents/discovery/pca.py, while the refactored deterministic pipeline is implemented in textttbackend/core/paperfinder.py. Both pipelines expose functionality through API servers, including a fast discovery variant designed for low-latency responses. The frontend is built with React and TypeScript and integrates discovery results through the API. Supabase provides authentication and persistent data storage for user profiles, communities, sessions, and paper metadata. Containerization support is provided via a Dockerfile, and deployment configurations are included for common platforms (Railway, Render, and Vercel). Environment variables control API URLs and database credentials, enabling local development or hosted deployment without code changes.