2311.05232v2

# A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions **Authors**: Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, Ting Liu > Harbin Institute of Technology 800 Dongchuan Road Harbin Heilongjiang China 150001 > Huawei Inc. Bantian Subdistrict Shenzhen Guangdong China 518129 ## Abstract The emergence of large language models (LLMs) has marked a significant breakthrough in natural language processing (NLP), fueling a paradigm shift in information acquisition. Nevertheless, LLMs are prone to hallucination, generating plausible yet nonfactual content. This phenomenon raises significant concerns over the reliability of LLMs in real-world information retrieval (IR) systems and has attracted intensive research to detect and mitigate such hallucinations. Given the open-ended general-purpose attributes inherent to LLMs, LLM hallucinations present distinct challenges that diverge from prior task-specific models. This divergence highlights the urgency for a nuanced understanding and comprehensive overview of recent advances in LLM hallucinations. In this survey, we begin with an innovative taxonomy of hallucination in the era of LLM and then delve into the factors contributing to hallucinations. Subsequently, we present a thorough overview of hallucination detection methods and benchmarks. Our discussion then transfers to representative methodologies for mitigating LLM hallucinations. Additionally, we delve into the current limitations faced by retrieval-augmented LLMs in combating hallucinations, offering insights for developing more robust IR systems. Finally, we highlight the promising research directions on LLM hallucinations, including hallucination in large vision-language models and understanding of knowledge boundaries in LLM hallucinations. Large Language Models, Hallucination, Factuality, Faithfulness copyright: acmlicensed journal: TOIS journalyear: 2024 journalvolume: 1 journalnumber: 1 article: 1 publicationmonth: 1 doi: 10.1145/3703155 ccs: Computing methodologies Natural language generation ccs: General and reference Surveys and overviews ## 1. Introduction Recently, the emergence of large language models (LLMs) (Zhao et al., 2023d), exemplified by LLaMA (Touvron et al., 2023a, b), Claude (Antropic, 2024), Gemini (Anil et al., 2023; Reid et al., 2024) and GPT-4 (OpenAI, 2023), has ushered in a significant paradigm shift in natural language processing (NLP), achieving unprecedented progress in language understanding (Hendrycks et al., 2021; Huang et al., 2023a), generation (Zhang et al., 2023c; Zhu et al., 2023b) and reasoning (Wei et al., 2022; Kojima et al., 2022; Qiao et al., 2022; Yu et al., 2023b; Chu et al., 2023a). Furthermore, the extensive factual knowledge encoded within LLMs has demonstrated considerable advancements in leveraging LLMs for information seeking (Petroni et al., 2019; AlKhamissi et al., 2022), potentially reshaping the landscape of information retrieval systems (Zhu et al., 2023c). Nevertheless, in tandem with these remarkable advancements, concerns have arisen about the tendency of LLMs to generate hallucinations (Bang et al., 2023; Guerreiro et al., 2023a), resulting in seemingly plausible yet factually unsupported content. Further compounding this issue is the capability of LLMs to generate highly convincing and human-like responses (Sadasivan et al., 2023), which makes detecting these hallucinations particularly challenging, thereby complicating the practical deployment of LLMs, especially real-world information retrieval (IR) systems that have integrated into our daily lives like chatbots (OpenAI, 2022; Anthropic, 2023), search engines (Microsoft, 2023; AI, 2023), and recommender systems (Gao et al., 2023c; Li et al., 2023h). Given that the information provided by these systems can directly influence decision-making, any misleading information has the potential to spread false beliefs, or even cause harm. Notably, hallucinations in conventional natural language generation (NLG) tasks have been extensively studied (Huang et al., 2021a; Ji et al., 2023a), with hallucinations defined as generated content that is either nonsensical or unfaithful to the provided source content. These hallucinations are categorized into two types: intrinsic hallucination, where the generated output contradicts the source content, and extrinsic hallucination, where the generated output cannot be verified from the source. However, given their remarkable versatility across tasks (Bubeck et al., 2023; Bang et al., 2023), understanding hallucinations in LLMs presents a unique challenge compared to models tailored for specific tasks. Besides, as LLMs typically function as open-ended systems, the scope of hallucination encompasses a broader concept, predominantly manifesting factual errors. This shift necessitates a reevaluation and adjustment of the existing taxonomy of hallucinations, aiming to enhance its adaptability in the evolving landscape of LLMs. In this survey, we propose a redefined taxonomy of hallucination tailored specifically for applications involving LLMs. We categorize hallucination into two primary types: factuality hallucination and faithfulness hallucination. Factuality hallucination emphasizes the discrepancy between generated content and verifiable real-world facts, typically manifesting as factual inconsistencies. Conversely, faithfulness hallucination captures the divergence of generated content from user input or the lack of self-consistency within the generated content. This category is further subdivided into instruction inconsistency, where the content deviates from the user’s original instruction; context inconsistency, highlighting discrepancies from the provided context; and logical inconsistency, pointing out internal contradictions within the content. Such categorization refines our understanding of hallucinations in LLMs, aligning it closely with their contemporary usage. Delving into the underlying causes of hallucinations in LLMs is essential not merely for enhancing the comprehension of these phenomena but also for informing strategies aimed at alleviating them. Recognizing the multifaceted sources of LLM hallucinations, our survey identifies potential contributors into three main aspects: data, training, and inference stages. This categorization allows us to span a broad spectrum of factors, providing a holistic view of the origins and mechanisms by which hallucinations may arise within LLM systems. Furthermore, we comprehensively outline a variety of effective detection methods specifically devised for detecting hallucinations in LLMs, as well as an exhaustive overview of benchmarks related to LLM hallucinations, serving as appropriate testbeds to assess the extent of hallucinations generated by LLMs and the efficacy of detection methods. Beyond evaluation, significant efforts have been undertaken to mitigate hallucinations of LLMs. These initiatives are comprehensively surveyed in our study, in accordance with the corresponding causes, spanning from data-related, training-related, and inference-related approaches. In addition, the effectiveness of retrieval-augmented generation (RAG) in mitigating hallucinations has garnered tremendous attention within the field. Despite the considerable potential of RAG, current systems inherently face limitations and even suffer from hallucinations. Accordingly, our survey undertakes an in-depth analysis of these challenges, aiming to provide valuable insights aimed at developing more robust RAG systems. We also highlight several promising avenues for future research, such as hallucinations in large vision-language models and understanding of knowledge boundaries in LLM hallucinations, paving the way for forthcoming research in the field. Comparing with Existing Surveys. As hallucination stands out as a major challenge in generative AI, numerous research (Ji et al., 2023a; Rawte et al., 2023; Liu et al., 2023h; Zhang et al., 2023d; Wang et al., 2023e; Tonmoy et al., 2024) has been directed towards hallucinations. While these contributions have explored LLM hallucination from various perspectives and provided valuable insights, our survey seeks to delineate their distinct contributions and the comprehensive scope they encompass. Ji et al. (2023a) primarily shed light on hallucinations in pre-trained models for NLG tasks, leaving LLMs outside their discussion purview. Tonmoy et al. (2024) mainly focused on discussing the mitigation strategies combating LLM hallucinations. Besides, Liu et al. (2023h) took a broader view of LLM trustworthiness without delving into specific hallucination phenomena, whereas Wang et al. (2023e) provided an in-depth look at factuality in LLMs. However, our work narrows down to a critical subset of trustworthiness challenges, specifically addressing factuality and extending the discussion to include faithfulness hallucinations. To the best of our knowledge, Zhang et al. (2023d) presented research closely aligned with ours, detailing LLM hallucination taxonomies, evaluation benchmarks, and mitigation strategies. However, our survey sets itself apart through a unique taxonomy and organizational structure. We present a detailed, layered classification of hallucinations and conduct a more comprehensive analysis of the causes of hallucinations. Crucially, our proposed mitigation strategies are directly tied to these causes, offering a targeted and coherent framework for addressing LLM hallucinations. forked edges, for tree= grow=east, reversed=true, anchor=base west, parent anchor=east, child anchor=west, base=left, font=, rectangle, draw=hidden-draw, rounded corners, align=left, minimum width=4em, edge+=darkgray, line width=1pt, s sep=3pt, inner xsep=2pt, inner ysep=3pt, ver/.style=rotate=90, child anchor=north, parent anchor=south, anchor=center, , where level=1text width=4.5em,font=,, where level=2text width=8em,font=,, where level=3text width=7.5em,font=,, where level=4text width=7em,font=,, [ Hallucinations in Large Language Models, ver, color=carminepink!100, fill=carminepink!15, text=black [ Hallucination Causes (§ 3), color=harvestgold!100, fill=harvestgold!100, text=black [ Hallucination from Data, color=harvestgold!100, fill=harvestgold!60, text=black [ Misinformation and Biases, color=harvestgold!100, fill=harvestgold!40, text=black [ e.g. Lin et al. (2022); Lee et al. (2022a); Bender et al. (2021) , cause_leaf, text width=20em ] ] [ Knowledge Boundary, color=harvestgold!100, fill=harvestgold!40, text=black [ e.g. Singhal et al. (2023); Katz et al. (2023); Onoe et al. (2022) , cause_leaf, text width=20em ] ] [ Inferior Alignment Data, color=harvestgold!100, fill=harvestgold!40, text=black [ e.g. Gekhman et al. (2024); Li et al. (2024c) , cause_leaf, text width=20em ] ] ] [ Hallucination from Training, color=harvestgold!100, fill=harvestgold!60, text=black [ Hallucination from Pre-training, color=harvestgold!100, fill=harvestgold!40, text=black [ e.g. Li et al. (2023i); Liu et al. (2023a); Wang and Sennrich (2020) , cause_leaf, text width=20em ] ] [ Hallucination from SFT, color=harvestgold!100, fill=harvestgold!40, text=black [ e.g. Schulman (2023); Yang et al. (2023a); Zhang et al. (2023a) , cause_leaf, text width=20em ] ] [ Hallucination from RLHF, color=harvestgold!100, fill=harvestgold!40, text=black [ e.g. Cotra (2021); Perez et al. (2023); Wei et al. (2023); Sharma et al. (2023) , cause_leaf, text width=20em ] ] ] [ Hallucination from Inference, color=harvestgold!100, fill=harvestgold!60, text=black [ Imperfect Decoding Strategies, color=harvestgold!100, fill=harvestgold!40, text=black [ e.g. Stahlberg and Byrne (2019); Holtzman et al. (2020) , cause_leaf, text width=20em ] ] [ Over-confidence, color=harvestgold!100, fill=harvestgold!40, text=black [ e.g. Miao et al. (2021); Chen et al. (2022a, 2023a); Liu et al. (2023i) , cause_leaf, text width=20em ] ] [ Softmax Bottleneck, color=harvestgold!100, fill=harvestgold!40, text=black [ e.g. Miao et al. (2021); Chang and McCallum (2022) , cause_leaf, text width=20em ] ] [ Reasoning Failure, color=harvestgold!100, fill=harvestgold!40, text=black [ e.g. Zheng et al. (2023); Berglund et al. (2023) , cause_leaf, text width=20em ] ] ] ] [ Hallucination Detection and Benchmarks(§ 4), color=cyan!100, fill=cyan!100, text=black [ Hallucination Detection, color=cyan!100, fill=cyan!60, text=black [ Factuality Hallucination Detection, color=cyan!100, fill=cyan!30, text=black [ e.g. Min et al. (2023b); Dhuliawala et al. (2023); Manakul et al. (2023) , detect_leaf, text width=20em ] ] [ Faithfulness Hallucination Detection, color=cyan!100, fill=cyan!30, text=black [ e.g. Maynez et al. (2020); Scialom et al. (2021); Fabbri et al. (2022) , detect_leaf, text width=20em ] ] ] [ Hallucination Benchmarks, color=cyan!100, fill=cyan!60, text=black [ Hallucination Evaluation Benchmarks, color=cyan!100, fill=cyan!30, text=black [ e.g. TruthfulQA (Lin et al., 2022), HalluQA (Cheng et al., 2023), HaluEval-2.0 (Li et al., 2024c) , detect_leaf, text width=20em ] ] [ Hallucination Detection Benchmarks, color=cyan!100, fill=cyan!30, text=black [ e.g. SelfCheckGPT-Wikibio (Miao et al., 2023), HaluEval (Li et al., 2023b), FELM (Chen et al., 2023e) , detect_leaf, text width=20em ] ] ] ] [ Hallucination Mitigation (§ 5), color=lightgreen!100, fill=lightgreen!100, text=black [ Mitigating Data-related Hallucinations, color=lightgreen!100, fill=lightgreen!60, text=black [ Data Filtering, color=lightgreen!100, fill=lightgreen!40, text=black [ e.g. Gunasekar et al. (2023); Touvron et al. (2023b); Abbas et al. (2023) , mitigate_leaf, text width=20em ] ] [ Model Editing, color=lightgreen!100, fill=lightgreen!40, text=black [ e.g. Mitchell et al. (2022b); Huang et al. (2023d); Dai et al. (2022b) , mitigate_leaf, text width=20em ] ] [ Retrieval-Augmented Generation, color=lightgreen!100, fill=lightgreen!40, text=black [ e.g. Ram et al. (2023); Yu et al. (2023c); Gao et al. (2023a) , mitigate_leaf, text width=20em ] ] ] [ Mitigating Training-related Hallucinations, color=lightgreen!100, fill=lightgreen!60, text=black [ Mitigating Pre-training- related Hallucination, color=lightgreen!100, fill=lightgreen!40, text=black [ e.g. Li et al. (2023i); Liu et al. (2023f, a); Shi et al. (2023b) , mitigate_leaf, text width=20em ] ] [ Mitigating Misalignment Hallucination, color=lightgreen!100, fill=lightgreen!40, text=black [ e.g. Wei et al. (2023); Sharma et al. (2023); Rimsky (2023b) , mitigate_leaf, text width=20em ] ] ] [ Mitigating Inference-related Hallucinations, color=lightgreen!100, fill=lightgreen!60, text=black [ Factuality Enhanced Decoding, color=lightgreen!100, fill=lightgreen!40, text=black [ e.g. Lee et al. (2022b); Li et al. (2023f); Chuang et al. (2023) , mitigate_leaf, text width=20em ] ] [ Faithfulness Enhanced Decoding, color=lightgreen!100, fill=lightgreen!40, text=black [ e.g. Wan et al. (2023); Shi et al. (2023a); Chang et al. (2023a) , mitigate_leaf, text width=20em ] ] ] ] ] Figure 1. The main content flow and categorization of this survey. Organization of this Survey. In this survey, we present a comprehensive overview of the latest developments in LLM hallucinations, as shown in Fig 1. We commence by constructing a taxonomy of hallucinations in the realm of LLM (§ 2). Subsequently, we analyze factors contributing to LLM hallucinations in depth (§ 3), followed by a review of various strategies and benchmarks employed for the reliable detection of hallucinations in LLMs (§ 4). We then detail a spectrum of approaches designed to mitigate these hallucinations (§ 5). Concluding, we delve into the challenges faced by current RAG systems (§ 6) and delineate potential pathways for forthcoming research (§ 7). ## 2. Definitions For the sake of a comprehensive understanding of hallucinations in LLMs, we commence with a succinct introduction to LLMs (§ 2.1), delineating the scope of this survey. Subsequently, we delve into the training stages of LLMs (§ 2.2), as a thorough understanding of the training mechanisms contributes significantly to elucidating the origins of hallucinations. Lastly, we expound upon the concept of hallucinations in LLMs (§ 2.3), further categorizing it into two distinct types. ### 2.1. Large Language Models Before delving into the causes of hallucination, we first introduce the concept of LLMs. Typically, LLMs refer to a series of general-purpose models that leverage the transformer-based language model architecture and undergo extensive training on massive textual corpora with notable examples including GPT-3 (Brown et al., 2020), PaLM (Chowdhery et al., 2023), LLaMA (Touvron et al., 2023b), GPT-4 (OpenAI, 2023) and Gemini (Reid et al., 2024). By scaling the amount of data and model capacity, LLMs raise amazing emergent abilities, typically including in-context learning (ICL) (Brown et al., 2020), chain-of-thought prompting (Wei et al., 2022) and instruction following (Peng et al., 2023). ### 2.2. Training Stages of Large Language Models The attributes and behaviors of LLMs are deeply intertwined with their training processes. LLMs undergo three primary training stages: pre-training, supervised fine-tuning (SFT), and reinforcement learning from human feedback (RLHF). Analyzing these stages provides insight into hallucination origins in LLMs, as each stage equips the model with specific capabilities. #### 2.2.1. Pre-training Pre-training is widely acknowledged as a foundational stage for LLM to acquire knowledge and capabilities (Zhou et al., 2023b). During this phase, LLMs engage in autoregressive prediction of subsequent tokens within sequences. Through self-supervised training on extensive textual corpora, LLMs acquire knowledge of language syntax, world knowledge, and reasoning abilities, thereby laying a solid groundwork for further fine-tuning. Besides, recent research (Sutskever, 2023; Delétang et al., 2023) suggests that predicting subsequent words is akin to losslessly compressing significant information. The essence of LLMs lies in predicting the probability distribution for upcoming words. Accurate predictions indicate a profound grasp of knowledge, translating to a nuanced understanding of the world. #### 2.2.2. Supervised Fine-Tuning While LLMs acquire substantial knowledge and capabilities during the pre-training stage, it’s crucial to recognize that pre-training primarily optimizes for completion. Consequently, pre-trained LLMs fundamentally serve as completion machines, which can lead to a misalignment between the next-word prediction objective of LLMs and the user’s objective of obtaining desired responses. To bridge this gap, SFT (Zhang et al., 2023b) has been introduced, which involves further training LLMs using a meticulously annotated set of (instruction, response) pairs, resulting in enhanced capabilities and improved controllability of LLMs. Furthermore, recent studies (Chung et al., 2022; Iyer et al., 2022) have confirmed the effectiveness of supervised fine-tuning to achieve exceptional performance on unseen tasks, showcasing their remarkable generalization abilities. #### 2.2.3. Reinforcement Learning from Human Feedback While the SFT process successfully enables LLMs to follow user instructions, there is still room for them to better align with human preferences. Among various methods that utilize human feedback, RLHF stands out as an representative solution for aligning with human preferences through reinforcement learning (Christiano et al., 2017; Stiennon et al., 2020; Ouyang et al., 2022). Typically, RLHF employs a preference model (Bradley and Terry, 1952) trained to predict preference rankings given a prompt alongside a pair of human-labeled responses. To align with human preferences, RLHF optimizes the LLM to generate outputs that maximize the reward provided by the trained preference model, typically employing a reinforcement learning algorithm, such as Proximal Policy Optimization (PPO) (Schulman et al., 2017). Such integration of human feedback into the training loop has proven effective in enhancing the alignment of LLMs, guiding them toward producing high-quality and harmless responses. Table 1. Examples of each category of LLM hallucinations. Content marked in Red represents the hallucinatory output, while content marked in Blue indicates user instruction or provided context that contradicts the LLM hallucination. . Category Type Example Dialog Explanation Factual Contradiction Instruction: What are Thomas Edison’s main contributions to science and technology? Response: Thomas Edison developed the first practical tele- phone and invented the light bulb. The response is factually incorrect. In reality, Edison improved the light bulb, building on earlier desi- gns by others, and Alexander Graham Bell inven - ted the telephone. Factuality Hallucination Factual Fabrication Instruction: What are the major environmental impacts of the construction of the Eiffel Tower? Response: The construction of the Eiffel Tower in 1889 led to the extinction of the Parisian tiger, a species that played a crucial role in the region’s ecosystem. Additionally, it is wid- ely recognized as the event that sparked the global green architecture movement. The response erroneously states that the construction of the Eiffel Tower led to the extinction of the ‘Parisian tiger’ and sparked the global green architecture move- ment. In fact, there is no evidence of a ‘Parisian tig- er’ ever existing, making this a fabricated claim. Moreover, attributing the origins of green architec- ture to the Eiffel Tower is an exaggeration, as this movement has diverse roots and cannot be traced back to a single event. Instruction Inconsistency Instruction: Translate the English question into Spanish: ”What is the capital of France?” Response: The capital of France is Paris. The user explicitly provided an instruction to translate the English question into Spanish. Instead of perform- ing the translation as instructed, the LLM chose to answer the question. Context Inconsistency Retrieved context: The Nile is a major north-flowing river in northeastern Africa. It’s traditionally considered the longest river in the world. Originating in the Great Lakes region of central Africa, it flows through several countries before emptying into the Mediterranean Sea. Instruction: Summarize the retrieval document above. Response: The Nile originates from the mountain ranges of central Africa and flows into the Mediterranean. The summary incorrectly states that the Nile originates from the ”mountain ranges of central Africa”, while the paragraph mentions that the Nile originates in the Great Lakes region of central Africa. Faithfulness Hallucination Logical Inconsistency Instruction: Solve the equation 2x + 3 = 11 step by step. Response: Step 1: Subtract 3 from both sides to get: 2x = 8. Step 2: Divide both sides by 2 to get: x = 3. So the result of the equation is x=3. The first step correctly isolates the term with x to get 2x = 8. However, the next step inaccurately divides 8 by 2 to yield a result of x = 3, which is inconsistent with the earlier reasoning. ### 2.3. Hallucinations in Large Language Models The concept of hallucination traces its roots to the fields of pathology and psychology and is defined as the perception of an entity or event that is absent in reality (Macpherson and Platchias, 2013). Within the realm of NLP, hallucination is typically referred to as a phenomenon in which the generated content appears nonsensical or unfaithful to the provided source content (Filippova, 2020; Maynez et al., 2020). This concept bears a loose resemblance to the phenomenon of hallucination observed in human psychology. Generally, hallucinations in natural language generation tasks can be categorized into two primary types: intrinsic hallucination and extrinsic hallucination (Huang et al., 2021b; Li et al., 2022b; Ji et al., 2023a). Specifically, intrinsic hallucinations pertain to the model outputs that directly conflict with the provided source context. On the other hand, extrinsic hallucinations involve outputs that cannot be verified using the provided source context or external knowledge bases. This means the generated text is neither supported by nor directly contradicts the available information, rendering the output unverifiable and potentially misleading. However, in the era of LLMs, the versatile capabilities of these models have facilitated their widespread use across diverse fields, highlighting limitations in existing task-specific categorization paradigms. Considering that LLMs place a significant emphasis on user-centric interactions and prioritize alignment with user directives, coupled with the fact that their hallucinations predominantly surface at factual levels, we introduce a more granular taxonomy building upon the foundational work by Ji et al. (2023a). This refined taxonomy seeks to encapsulate the distinct intricacies associated with LLM hallucinations. To provide a more intuitive illustration of our definition of LLM hallucination, we present examples for each type of hallucination in Table 1, namely factuality hallucination and faithfulness hallucination. #### 2.3.1. Factuality Hallucination The emergence of LLMs marks a significant shift from traditional task-specific toolkits to AI assistants that have a heightened focus on open-domain interactions. This shift is primarily attributed to their vast parametric factual knowledge. However, existing LLMs occasionally exhibit tendencies to produce outputs that are either inconsistent with real-world facts or unverifiable (Li et al., 2024c), posing challenges to the trustworthiness of artificial intelligence. In this context, we categorize these factuality hallucinations into two primary types: Factual Contradiction refers to situations where the LLM’s output contains facts that can be grounded in real-world information, but present contradictions. This type of hallucination occurs most frequently and arises from diverse sources, encompassing the LLM’s capture, storage, and expression of factual knowledge. Depending on the error type of contradictions, it can be further divided into two subcategories: entity-error hallucination and relation-error hallucination. - Entity-error hallucination refers to the situations where the generated text of LLMs contains erroneous entities. As shown in Table 1, when asked about ”the inventor of the telephone”, the model erroneously states ”Thomas Edison”, conflicting with the real fact that it was ”Alexander Graham Bell”. - Relation-error hallucination refers to instances where the generated text of LLMs contains wrong relations between entities. As shown in Table 1, when inquired about ”the inventor of the light bulb”, the model incorrectly claims ”Thomas Edison”, despite the fact that he improved upon existing designs and did not invent it. Factual Fabrication refers to instances where the LLM’s output contains facts that are unverifiable against established real-world knowledge. This can be further divided into unverifiability hallucination and overclaim hallucination. - Unverifiability hallucination pertains to statements that are entirely non-existent or cannot be verified using available sources. As shown in Table 1, when asked about ”the major environmental impacts of the construction of the Eiffel Tower”, the model incorrectly states that ”the construction led to the extinction of the Parisian tiger”, a species that does not exist and thus, this claim cannot be substantiated by any historical or biological record. - Overclaim hallucination involves claims that lack universal validity due to subjective biases. As shown in Table 1, the model claims that ”the Eiffel Tower’s construction is widely recognized as the event that sparked the global green architecture movement.” This is an overclaim, as there is no broad consensus or substantial evidence to support the statement. #### 2.3.2. Faithfulness Hallucination LLMs are inherently trained to align with user instructions. As the use of LLMs shifts towards more user-centric applications, ensuring their consistency with user-provided instructions and contextual information becomes increasingly vital. Furthermore, LLM’s faithfulness is also reflected in the logical consistency of its generated content. From this perspective, we categorize three subtypes of faithfulness hallucinations: Instruction inconsistency refers to the LLM’s outputs that deviate from a user’s directive. While some deviations might serve safety guidelines, the inconsistencies here signify unintentional misalignment with non-malicious user instructions. As described in Table 1, the user’s actual intention is translation, However, the LLM erroneously deviated from the user’s instruction and performed a question-answering task instead. Context inconsistency points to instances where the LLM’s output is unfaithful with the user’s provided contextual information. For example, as shown in Table 1, the user mentioned the Nile’s source being in the Great Lakes region of central Africa, yet the LLM’s response contradicted the context. Logical inconsistency underscores when LLM outputs exhibit internal logical contradictions, often observed in reasoning tasks. This manifests as inconsistency both among the reasoning steps themselves and between the steps and the final answer. For example, as shown in Table 1, while the reasoning step of dividing both sides of the equation by 2 is correct, the final answer of x=4 is inconsistent with the reasoning chain, leading to an incorrect result. ## 3. Hallucination Causes LLM hallucinations have multifaceted origins, spanning the entire spectrum of LLMs’ capability acquisition process. In this section, we delve into the root causes of hallucinations in LLMs, primarily categorized into three key aspects: (1) Data (§ 3.1), (2) Training (§ 3.2), and (3) Inference (§ 3.3). ### 3.1. Hallucination from Data Data for training LLMs are comprised of two primary components: (1) pre-training data, through which LLMs acquire their general capabilities and factual knowledge (Zhou et al., 2023b), and (2) alignment data, which teach LLMs to follow user instructions and align with human preferences (Wang et al., 2023i). Although these data constantly expand the capability boundaries of LLMs, they inadvertently become the principal contributors to LLM hallucinations. This primarily manifests in three aspects: the presence of misinformation and biases in the flawed pre-training data sources (§ 3.1.1), the knowledge boundary inherently bounded by the scope of the pre-training data (§ 3.1.2), and the hallucinations induced by inferior alignment data (§ 3.1.3). #### 3.1.1. Misinformation and biases Neural networks possess an intrinsic tendency to memorize training data (Carlini et al., 2021), and this memorization tendency grows with model size (Carlini et al., 2022; Chowdhery et al., 2023). In general, the inherent memorization capability is a double-edged sword in the fight against hallucinations. On the one hand, the capacities of LLMs to memorize suggests their potential to capture profound world knowledge. On the other hand, it becomes problematic in the context of misinformation and biases present within pre-training data and may inadvertently be amplified, manifesting as imitative falsehood (Lin et al., 2022) and the reinforcement of societal biases. For a more comprehensive understanding, detailed examples are presented in Table 2. Table 2. Examples of Hallucinations from Misinformation and Biases. The table categorizes hallucinations arising from flawed data sources into imitative falsehoods, duplication bias, and social biases. Each category is accompanied by a premise outlining the data issue, user input, and the LLM’s hallucinatory output, and an explanation for the occurrence, aiding comprehension of these complex phenomena. | Type | Premise | Input | Model Ouput | Explanation | | --- | --- | --- | --- | --- | | Imitative Falsehood | Within the LLM’s training data, a frequently referenced but incorrect fact appears multiple times: Thomas Edison invented the light bulb. | Who is credited with the invention of the light bulb? | Thomas Edison is credited with the invention of the light bulb. | While Edison improved the light bulb’s design, he was not its sole inventor. The model’s answer reflects a misconception prevalent in its training data, exemplifying an imitative falsehood. | | Societal Biases | In the model’s training data, certain names may frequently be associated with specific nationalities. | Provide a summary of the following news: Dr. Kim recently published a breakthrough paper on quantum computing. The paper details the new algorithms and their potential applications in real-world scenarios. It has been hailed as revolutionary by experts. | Dr. Kim from South Korea recently published a paper on quantum computing, detailing new algorithms and their applications. | The model erroneously added ”South Korea” based on the name ”Kim” revealing its bias in linking certain names to specific nationalities, which can cause hallucinations in the summary. | Imitative Falsehood. Misinformation such as fake news and unfounded rumors has been widely spread among social media platforms and gradually serves as a significant contributor to LLM hallucinations. The increasing demand for large-scale corpora for pre-training necessitates the employment of heuristic data collection methods. While facilitating the acquisition of extensive data, challenges arise in maintaining consistent data quality, which inevitably introduces such misinformation in pre-training data (Bender et al., 2021; Weidinger et al., 2021). In this situation, LLMs will increase the likelihood of generating such false statements due to their remarkable memorization capabilities, leading to imitative falsehoods. The issue is further exacerbated by the fact that LLMs have drastically lowered the barriers to content creation, posing considerable risks to the trustworthiness of public discourse and internet ecosystems. Societal Biases. In addition to misinformation, biases are also deeply rooted in social media platforms, showing up variously, like biased hiring, prejudiced news, and hate-spewing. As a purely negative phenomenon, biases and hallucinations have obvious differences, mainly manifested as stereotypes spreading and social inequalities. However, certain biases are intrinsically tied to hallucinations, especially those related to gender (Paullada et al., 2021) and nationality (Narayanan Venkit et al., 2023; Ladhak et al., 2023). For instance, LLMs might associate the profession of nursing with females, even when gender isn’t explicitly mentioned in the user-provided context, exemplifying context inconsistency as discussed in Section (§ 2.3). Such biases can be inadvertently acquired from internet-based texts, which are rife with diverse and biased viewpoints, and subsequently be propagated into the generated content (Ladhak et al., 2023). Table 3. Example of Knowledge Boundary. Content marked in Red represents the hallucinatory output. | Type | Input | Model Ouput | Explanation | | --- | --- | --- | --- | | Long-tail Knowledge | What causes phenylketonuria? | Phenylketonuria is caused by a mutation in the hemoglobin gene, which results in an unusual interaction with hemoglobin, affecting its ability to transport oxygen in the bloodstream. | The correct cause of phenylketonuria is mutations in the PAH gene. Such specialized medical information is less frequently encountered in the training corpus, making LLMs struggle to memorize. | | Up-to-date Knowledge | Who will be the Democratic Party candidate for the 2024 US presidential election? | Michelle Obama has announced her candidacy and is leading in the polls, with significant endorsements from key Democratic figures such as Barack Obama and Bernie Sanders. She has also outlined a policy platform focusing on healthcare reform and climate change initiatives. | After President Joe Biden announced his withdrawal from the race on July 21, 2024, he endorsed Kamala Harris as the Democratic Party candidate. Such hallucinations arise because LLMs’ parametric knowledge cannot be updated in real-time, leading to potential fabrication regarding current events. | | Copyright-sensitive Knowledge | Describe the ending of the novel ”The Secrets of the Alchemist” that was published last year. | In the dramatic conclusion of ”The Secrets of the Alchemist”, the protagonist discovers the true secret behind the philosopher’s stone, leading to a peaceful resolution with all characters sharing in the wisdom. | The novel ”The Secrets of the Alchemist” is under copyright protection, and LLMs have not been trained directly on such copyrighted materials. Thus, the model’s output fabricates details about the book’s ending. | #### 3.1.2. Knowledge Boundary. While the vast pre-training corpora have empowered LLMs with extensive factual knowledge, they inherently possess knowledge boundaries. These boundaries arise primarily from two sources: (1) the inability of LLMs to memorize all factual knowledge encountered during pre-training, especially the less frequent long-tail knowledge; and (2) the intrinsic boundary of the pre-training data itself, which does not include rapidly evolving world knowledge or content restricted by copyright laws. Consequently, when LLMs encounter information that falls outside their limited knowledge boundaries, they are more susceptible to generating hallucinations. We present detailed examples for clear illustration in Table 3. Long-tail Knowledge. The distribution of knowledge within the pre-training corpora is inherently non-uniform, which results in LLMs demonstrating varying levels of proficiency across different types of knowledge. Recent studies have highlighted a strong correlation between the model’s accuracy on general domain questions and the volume of relevant documents (Kandpal et al., 2023) or entity popularity (Mallen et al., 2023) within the pre-training corpora. Furthermore, given that LLMs are predominantly trained on extensive general domain corpora (Penedo et al., 2023; Raffel et al., 2020; Gao et al., 2021a), they may exhibit deficits in domain-specific knowledge. This limitation becomes particularly evident when LLMs are confronted with tasks that require domain-specific expertise, such as medical (Li et al., 2023e; Singhal et al., 2023) and legal (Yu et al., 2022; Katz et al., 2023) questions, these models may exhibit pronounced hallucinations, often manifesting as factual fabrication. Up-to-date Knowledge. Beyond the shortfall in long-tail knowledge, another intrinsic limitation concerning the knowledge boundaries within LLMs is their constrained capacity for up-to-date knowledge. The factual knowledge embedded within LLMs exhibits clear temporal boundaries and can become outdated over time (Onoe et al., 2022; Kasai et al., 2022; Li et al., 2023g). Once these models are trained, their internal knowledge is never updated. This poses a challenge given the dynamic and ever-evolving nature of our world. When confronted with queries that transcend their temporal scope, LLMs often resort to fabricating facts or providing answers that might have been correct in the past but are now outdated. Copyright-sensitive Knowledge. Due to licensing restrictions (Reuters, 2023), existing LLMs are legally constrained to training on corpora that are publicly licensed (Gao et al., 2021a; Computer, 2023) or otherwise available for use without infringing copyright laws (Henderson et al., 2022; ArXiv, 2023). This limitation significantly impacts the breadth and diversity of knowledge that LLMs can legally acquire. A significant portion of valuable knowledge, encapsulated in copyrighted materials such as recent scientific research, proprietary data, and copyrighted literary works, remains inaccessible to LLMs. This exclusion creates a knowledge gap, leading to potential hallucinations when LLMs attempt to generate information in domains where their training data is inaccessible (Min et al., 2023a). #### 3.1.3. Inferior Alignment Data After the pre-training stage, LLMs have embedded substantial factual knowledge within their parameters, thereby establishing obvious knowledge boundaries. During the supervised fine-tuning (SFT) stage, LLMs are typically trained on instruction pairs labeled by human annotators, potentially introducing new factual knowledge that extends beyond the knowledge boundary established during pre-training. Gekhman et al. (2024) analyzed the training dynamics of incorporating new factual knowledge during the SFT process and found that LLMs struggle to acquire such new knowledge effectively. Most importantly, they discovered a correlation between the acquisition of new knowledge through SFT and increased hallucinations, suggesting that introducing new factual knowledge encourages LLMs to hallucinate. Additionally, Li et al. (2024c) conducted extensive analysis on the effect of instructions in producing hallucinations. Findings indicated that task-specific instructions which primarily focus on task format learning, tend to yield a higher proportion of hallucinatory responses. Moreover, overly complex and diverse instructions also lead to increased hallucinations. ### 3.2. Hallucination from Training As detailed in Section 2.2, the distinct stages of training impart various capabilities to LLMs, with pre-training focusing on acquiring general-purpose representations and world knowledge, and alignment enables LLMs to better align with user instructions and preferences. While these stages are critical for equipping LLMs with remarkable capabilities, shortfalls in either stage can inadvertently pave the way for hallucinations. #### 3.2.1. Hallucination from Pre-training Pre-training constitutes the foundational stage for LLMs, predominantly utilizing a transformer-based architecture following the paradigm established by GPT (Radford et al., 2018, 2019; Brown et al., 2020), and further developed by OPT (Zhang et al., 2022), Falcon (Penedo et al., 2023), and Llama-2 (Touvron et al., 2023b). This stage employs a causal language modeling objective, where models learn to predict subsequent tokens solely based on preceding ones in a unidirectional, left-to-right manner. While facilitating efficient training, it inherently limits the ability to capture intricate contextual dependencies, potentially increasing risks for the emergence of hallucination (Li et al., 2023i). Moreover, recent research has exposed that LLMs can occasionally exhibit unpredictable reasoning hallucinations spanning both long-range and short-range dependencies, which potentially arise from the limitations of soft attention (Hahn, 2020; Chiang and Cholak, 2022), where attention becomes diluted across positions as sequence length increases. Notably, the phenomenon of exposure bias (Bengio et al., 2015; Ranzato et al., 2016) has been a longstanding and serious contribution to hallucinations, resulting from the disparity between training and inference in the auto-regressive generative model. Such inconsistency can result in hallucinations (Wang and Sennrich, 2020), especially when an erroneous token generated by the model cascades errors throughout the subsequent sequence, akin to a snowball effect (Zhang et al., 2023g). #### 3.2.2. Hallucination from Supervised Fine-tuning LLMs have inherent capability boundaries established during pre-training. SFT seeks to utilize instruction data and corresponding responses to unlock these pre-acquired abilities. However, challenges arise when the demands of annotated instructions exceed the model’s pre-defined capability boundaries. In such cases, LLMs are trained to fit responses beyond their actual knowledge boundaries. As discussed in § 3.1.3, over-fitting on new factual knowledge encourages LLMs prone to fabricating content, amplifying the risk of hallucinations (Schulman, 2023; Gekhman et al., 2024). Moreover, another significant reason lies in the models’ inability to reject. Traditional SFT methods typically force models to complete each response, without allowing them to accurately express uncertainty (Yang et al., 2023a; Zhang et al., 2023a). Consequently, when faced with queries that exceed their knowledge boundaries, these models are more likely to fabricate content rather than reject it. This misalignment of knowledge boundaries, coupled with the inability to express uncertainty, are critical factors that contribute to the occurrence of hallucinations during the SFT stage. #### 3.2.3. Hallucination from RLHF Several studies (Burns et al., 2022; Azaria and Mitchell, 2023) have demonstrated that LLM’s activations encapsulate an internal belief related to the truthfulness of its generated statements. Nevertheless, misalignment can occasionally arise between these internal beliefs and the generated outputs. Even when LLMs are refined with human feedback (Ouyang et al., 2022), they can sometimes produce outputs that diverge from their internal beliefs. Such behaviors, termed as sycophancy (Cotra, 2021), underscore the model’s inclination to appease human evaluators, often at the cost of truthfulness. Recent studies indicate that models trained via RLHF exhibit pronounced behaviors of pandering to user opinions. Such sycophantic behaviors are not restricted to ambiguous questions without definitive answers (Perez et al., 2023), like political stances, but can also arise when the model chooses a clearly incorrect answer, despite being aware of its inaccuracy (Wei et al., 2023). Delving into this phenomenon, Sharma et al. (2023) suggested that the root of sycophancy may lie in the training process of RLHF models. By further exploring the role of human preferences in this behavior, the research indicates that the tendency for sycophancy is likely driven by both humans and preference models showing a bias towards sycophantic responses over truthful ones. ### 3.3. Hallucination from Inference Decoding plays an important role in manifesting the capabilities of LLMs after pretraining and alignment. However, certain shortcomings in decoding strategies can lead to LLM hallucinations. #### 3.3.1. Imperfect Decoding Strategies LLMs have demonstrated a remarkable aptitude for generating highly creative and diverse content, a proficiency that is critically dependent on the pivotal role of randomness in their decoding strategies. Stochastic sampling (Fan et al., 2018; Holtzman et al., 2020) is currently the prevailing decoding strategy employed by these LLMs. The rationale for incorporating randomness into decoding strategies stems from the realization that high likelihood sequences often result in surprisingly low-quality text, which is called likelihood trap (Stahlberg and Byrne, 2019; Holtzman et al., 2020; Meister et al., 2020; Zhang et al., 2021). The diversity introduced by the randomness in decoding strategies comes at a cost, as it is positively correlated with an increased risk of hallucinations (Dziri et al., 2021a; Chuang et al., 2023). An elevation in the sampling temperature results in a more uniform token probability distribution, increasing the likelihood of sampling tokens with lower frequencies from the tail of the distribution. Consequently, this heightened tendency to sample infrequently occurring tokens exacerbates the risk of hallucinations (Aksitov et al., 2023). #### 3.3.2. Over-confidence Prior studies in conditional text generation (Miao et al., 2021; Chen et al., 2022a) have highlighted the issue of over-confidence which stems from an excessive focus on the partially generated content, often prioritizing fluency at the expense of faithfully adhering to the source context. While LLMs, primarily adopting the causal language model architecture, have gained widespread usage, the over-confidence phenomenon continues to persist. During the generation process, the prediction of the next word is conditioned on both the language model context and the partially generated text. However, as demonstrated in prior studies (Voita et al., 2019; Beltagy et al., 2020; Liu et al., 2023f), language models often exhibit a localized focus within their attention mechanisms, giving priority to nearby words and resulting in a notable deficit in context attention (Shi et al., 2023a). Furthermore, this concern is further amplified in LLMs that exhibit a proclivity for generating lengthy and comprehensive responses. In such cases, there is even a heightened susceptibility to the risk of instruction forgetting (Chen et al., 2023a; Liu et al., 2023i). This insufficient attention can directly contribute to faithfulness hallucinations, wherein the model outputs content that deviates from the original context. #### 3.3.3. Softmax Bottleneck The majority of language models utilize a softmax layer that operates on the final layer’s representation within the language model, in conjunction with a word embedding, to compute the ultimate probability associated with word prediction. Nevertheless, the efficacy of Softmax-based language models is impeded by a recognized limitation known as the Softmax bottleneck (Yang et al., 2018a), wherein the employment of softmax in tandem with distributed word embeddings constrains the expressivity of the output probability distributions given the context which prevents LMs from outputting the desired distribution. Additionally, Chang and McCallum (2022) discovered that when the desired distribution within the output word embedding space exhibits multiple modes, language models face challenges in accurately prioritizing words from all the modes as the top next words, which also introduces the risk of hallucination. #### 3.3.4. Reasoning Failure Beyond the challenges with long-tail knowledge, effective utilization of knowledge is inextricably linked with reasoning capabilities. For instance, in multi-hop question-answering scenarios, even if the LLM possesses the necessary knowledge, it may struggle to produce accurate results if multiple associations exist between questions, due to its limitations in reasoning (Zheng et al., 2023). Furthermore, Berglund et al. (2023) unveiled a specific reasoning failure in LLMs termed the Reversal Curse. Specifically, while the model can correctly answer when the question is formulated as ”A is B”, it exhibits a failed logical deduction when asked the converse ”B is A”. This discrepancy in reasoning extends beyond simple deductions. ## 4. Hallucination Detection and Benchmarks The issue of hallucinations within LLMs has garnered considerable attention, raising concerns about the reliability of LLMs and their deployment in practical applications. As LLMs become increasingly adept at generating human-like text, accurately distinguishing between hallucinated versus factual content becomes increasingly vital. Moreover, effectively measuring the level of hallucination in LLM is crucial for improving their reliability. Thus, in this section, we delve into hallucination detection approaches (§ 4.1) and benchmarks for assessing LLM hallucinations (§ 4.2). ### 4.1. Hallucination Detection Existing strategies for detecting hallucinations in LLMs can be categorized based on the type of hallucination: (1) factuality hallucination detection, which aims to identify factual inaccuracies in the model’s outputs, and (2) faithfulness hallucination detection, which focuses on evaluating the faithfulness of model’s outputs to the contextual information provided. #### 4.1.1. Factuality Hallucination Detection Factuality hallucination detection involves assessing whether the output of LLMs aligns with real-world facts. Typical methods generally fall into two categories: fact-checking, which involves verifying the factuality of the generated response against trusted knowledge sources, and uncertainty estimation, which focuses on detecting factual inconsistency via internal uncertainty signals. Fact-checking. Given that the output of LLMs is typically comprehensive and consists of multiple factual statements, the fact-checking approach is generally divided into two primary steps: (1) fact extraction, which involves extracting independent factual statements within the model’s outputs (2) fact verification, which aims at verifying the correctness of these factual statements against trusted knowledge sources. Depending on the type of knowledge sources employed for verification, fact-checking methodologies can be broadly categorized into two distinct parts: external retrieval and internal checking. - External retrieval: The most intuitive strategy for fact verification is external retrieval. Min et al. (2023b) developed FACTSCORE, a fine-grained factual metric tailored for evaluating long-form text generation. It first decomposes the generation content into atomic facts and subsequently computes the percentage supported by reliable knowledge sources. Expanding on this concept, Chern et al. (2023) proposed a unified framework that equips LLMs with the capability to identify factual inaccuracies by utilizing a collection of external tools dedicated to evidence gathering. In addition to retrieving supporting evidence solely based on decomposited claims, Huo et al. (2023) improved the retrieval process through query expansion. By combining the original question with the LLM-generated answer, they effectively addressed the issue of topic drift, ensuring that the retrieved evidence aligns with both the question and the LLM’s response. - Internal checking: Given the extensive factual knowledge encoded in their parameters, LLMs have been explored as factual knowledge sources for fact-checking. Dhuliawala et al. (2023) introduced the Chain-of-Verification (CoVe), where an LLM first generates verification questions for a draft response and subsequently leverages its parametric knowledge to assess the consistency of the answer against the original response, thereby detecting potential inconsistencies. Kadavath et al. (2022) and Zhang et al. (2024d) calculates the probability $p(True)$ to assess the factuality of the response to a boolean question, relying exclusively on the model’sinternal knowledge. Additionally, Li et al. (2024c) observed that most atomic statements are interrelated, some may serve as contextual backgrounds for others, which potentially leads to incorrect judgments. Thus, they instruct the LLM to directly predict hallucination judgments considering all factual statements. However, as LLMs are not inherently reliable factual databases (Zheng et al., 2024), solely relying on LLMs’ parametric knowledge for fact-checking may result in inaccurate assessments. Uncertainty Estimation. While many approaches to hallucination detection rely on external knowledge sources for fact-checking, several methods have been devised to address this issue in zero-resource settings, thus eliminating the need for retrieval. The foundational premise behind these strategies is that the origin of LLM hallucinations is inherently tied to the model’s uncertainty. Therefore, by estimating the uncertainty of the factual content generated by the model, it becomes feasible to detect hallucinations. The methodologies in uncertainty estimation can broadly be categorized into two approaches: based on LLM internal states and LLM behavior, as shown in Fig. 2. <details> <summary>x1.png Details</summary>  ### Visual Description ## Diagram: LLM Uncertainty and Correction Mechanisms ### Overview This image is a conceptual diagram illustrating two key concepts related to the behavior and internal states of Large Language Models (LLMs). It uses a specific example question—"What is the highest peak in the world?"—to demonstrate how an LLM might generate incorrect information (hallucination) and how mechanisms like self-consistency and multi-debate can lead to correction. The diagram is divided into two primary sections: (a) LLM Internal States and (b) LLM Behavior. ### Components/Axes The diagram is structured into three main vertical panels, separated by dashed lines. **Top Element:** * A yellow text box at the top center contains the prompt: `Question: What is the highest peak in the world?` **Left Panel: (a) LLM Internal States** * **Central Icon:** A blue robot icon labeled `Large Language Model`. * **Output Bubble:** A speech bubble from the robot contains the text: `The highest peak in the world is Mount Fuji.` The words `highest peak`, `world`, and `Mount Fuji` are highlighted in red. * **Uncertainty Bar:** Below the robot is a horizontal gradient bar. The left end is labeled `low uncertainty` (light pink/white), and the right end is labeled `high uncertainty` (solid red). An arrow points from the robot's output bubble down to the `high uncertainty` end of the bar. * **Caption:** The panel is labeled `(a) LLM Internal States` at the bottom. **Center & Right Panels: (b) LLM Behavior** This section is subdivided into two processes. **(1) Self-Consistency (Center Panel):** * **Source:** The same blue robot icon from the left panel is shown. * **Outputs:** Three speech bubbles branch out from the robot: 1. `Mount Everest stands as the tallest peak in the world.` (Text `Mount Everest` is in green). 2. `As far as I know, the highest peak in the world is Mount Fuji in Japan.` (Text `Mount Fuji` is in red). 3. `The highest peak is Mount Everest located in the Himalayas.` (Text `Mount Everest` is in green). * **Flow:** Arrows from all three bubbles point to a central pink box labeled `Consistency`. * **Caption:** This subsection is labeled `(1) Self-Consistency` at the bottom. **(2) Multi-Debate (Right Panel):** * **Initial Exchange:** * A blue robot icon states: `The highest peak in the world is Mount Fuji.` (`Mount Fuji` in red). * A red robot icon (different design) responds: `I must correct you. Mount Fuji is the highest peak in Japan. The highest peak in the world is Mount Everest in the Himalayas range.` (`Mount Everest` in green). * **Resolution:** A blue robot icon (same as the first) replies: `I stand corrected, you are right.` * **Caption:** This subsection is labeled `(2) Multi-Debate` at the bottom. ### Detailed Analysis The diagram presents a narrative flow from error to correction. 1. **Initial Error Generation (Left Panel):** The LLM is depicted as confidently generating a factually incorrect statement ("Mount Fuji is the highest peak"). This output is explicitly linked to a state of `high uncertainty` within the model's internal state, suggesting a disconnect between the model's confidence and its accuracy. 2. **Path to Correction via Self-Consistency (Center Panel):** The same LLM, when prompted multiple times (implied by the three outputs), generates a mix of correct and incorrect answers. Two outputs correctly identify `Mount Everest`, while one repeats the `Mount Fuji` error. The `Consistency` box implies that by comparing these multiple outputs, a majority vote or consistency check could identify the correct answer (`Mount Everest`) as the most consistent response. 3. **Path to Correction via Multi-Debate (Right Panel):** This shows a social correction mechanism. One LLM instance (blue) makes the initial error. A second, distinct LLM instance (red) acts as a critic, providing a detailed correction with geographical context. The first LLM then accepts the correction. This models a debate or peer-review process between AI agents. ### Key Observations * **Color Coding:** The diagram uses a consistent color scheme to denote correctness: **red** for incorrect information (`Mount Fuji`) and **green** for correct information (`Mount Everest`). * **Iconography:** Different robot icons are used to distinguish between the primary LLM (blue, friendly face) and the correcting agent in the debate (red, different design). * **Spatial Flow:** The overall flow is from left to right, moving from the internal state of error generation, through one correction mechanism (self-consistency), to another (multi-debate). * **Textual Emphasis:** Key factual entities (`Mount Fuji`, `Mount Everest`) and evaluative terms (`highest peak`, `Consistency`) are highlighted within the text bubbles for emphasis. ### Interpretation This diagram serves as a pedagogical tool to explain two strategies for mitigating factual errors (hallucinations) in Large Language Models. * **The Core Problem:** It visually establishes that an LLM can produce confident but wrong answers (`high uncertainty` state linked to a wrong output), highlighting the challenge of reliability. * **Self-Consistency as a Solution:** It proposes that generating multiple responses to the same query and selecting the most frequent or "consistent" answer can filter out one-off errors. The diagram suggests the model's internal uncertainty might manifest as variability in its outputs. * **Multi-Debate as a Solution:** It illustrates a more interactive approach where multiple AI agents, potentially with different training or prompting, can critique and correct each other. This mimics human collaborative reasoning and fact-checking. * **Underlying Message:** The diagram argues that while individual LLM outputs may be unreliable, structured processes—either internal (sampling and consistency checking) or external (debate between agents)—can harness the model's capabilities to arrive at more accurate and trustworthy conclusions. It shifts the focus from the infallibility of a single output to the robustness of a *process* involving the LLM. </details> Figure 2. Taxonomy of Uncertainty Estimation Methods in Factual Hallucination Detection, featuring a) LLM Internal States and b) LLM Behavior, with LLM Behavior encompassing two main categories: Self-Consistency and Multi-Debate. - LLM internal states: The internal states of LLMs can serve as informative indicators of their uncertainty, often manifested through metrics like token probability or entropy. Varshney et al. (2023) determined the model’s uncertainty towards key concepts quantified by considering the minimal token probability within those concepts. The underlying rationale is that a low probability serves as a strong indicator of the model’s uncertainty, with less influence from higher probability tokens present in the concept. Similarly, Luo et al. (2023a) employed a self-evaluation-based approach for uncertainty estimation by grounding in the rationale that a language model’s ability to adeptly reconstruct an original concept from its generated explanation is indicative of its proficiency with that concept. By initially prompting the model to generate an explanation for a given concept and then employing constrained decoding to have the model recreate the original concept based on its generated explanation, the probability score from the response sequence can serve as a familiarity score for the concept. Furthermore, Yao et al. (2023a) interpreted hallucination through the lens of adversarial attacks. Utilizing gradient-based token replacement, they devised prompts to induce hallucinations. Notably, they observed that the first token generated from a raw prompt typically exhibits low entropy, compared to those from adversarial attacks. Based on this observation, they proposed setting an entropy threshold to define such hallucination attacks. - LLM behavior: However, when systems are only accessible via API calls (OpenAI, 2022; Google, 2023; Microsoft, 2023), access to the output’s token-level probability distribution might be unavailable. Given this constraint, several studies have shifted their focus to probing a model’s uncertainty, either through natural language prompts (Xiong et al., 2023; Kadavath et al., 2022) or by examining its behavioral manifestations. For instance, by sampling multiple responses from an LLM for the same prompt, Manakul et al. (2023) detected hallucinations via evaluating the consistency among the factual statements. However, these methods predominantly rely on direct queries that explicitly solicit information or verification from the model. Agrawal et al. (2023), inspired by investigative interviews, advocated for the use of indirect queries. Unlike direct ones, these indirect counterparts often pose open-ended questions to elicit specific information. By employing these indirect queries, consistency across multiple model generations can be better evaluated. Beyond assessing uncertainty from the self-consistency of a single LLM’s multiple generations, one can embrace a multi-agent perspective by incorporating additional LLMs. Drawing inspiration from legal cross-examination practices, Cohen et al. (2023) introduced the LMvLM approach. This strategy leverages an examiner LM to question an examinee LM, aiming to unveil inconsistencies of claims during multi-turn interaction. #### 4.1.2. Faithfulness Hallucination Detection Ensuring the faithfulness of LLMs to provide context or user instructions is pivotal for their practical utility in IR applications, from conversational search to interactive dialogue systems. We categorize existing hallucination detection metrics tailored to faithfulness into the following groups, with an overview shown in Fig. 3: (1) Fact-based (2) Classifier-based (3) QA-based (4) Uncertainty-based (5) LLM-based. <details> <summary>x2.png Details</summary>  ### Visual Description ## Diagram Set: LLM Evaluation Metrics and Uncertainty Estimation ### Overview The image is a composite of five distinct diagrams, labeled (a) through (e), illustrating different methodological frameworks for evaluating the outputs of Large Language Models (LLMs) and for estimating their uncertainty. The diagrams use a consistent visual language with icons representing a human user (with a laptop) and an AI model (robot), along with color-coded boxes and arrows to denote data flow and processing steps. ### Components/Axes The image is divided into five panels: * **Top Left (a):** "Fact-based Metric" * **Middle Left (b):** "Classifier-based Metric" * **Bottom Left (c):** "QA-based Metric" * **Top Right (d):** "Uncertainty Estimation" * **Bottom Right (e):** "Prompting-based Metric" **Common Visual Elements:** * **Icons:** A human user icon (person with laptop) and an AI model icon (robot head). * **Color Coding:** * Yellow boxes: "User Query" * Light Blue boxes: "LLM Generation" * Pink boxes: Processing/Model steps (e.g., "Information Extraction", "NLI Model", "Question Answering"). * Gray boxes: Final outputs or comparison metrics (e.g., "Fact Overlap", "Entailment", "Answer Overlap"). * In diagram (d): Red boxes = "high uncertainty", Green boxes = "low uncertainty". * **Arrows:** Indicate the direction of data flow between components. ### Detailed Analysis #### **(a) Fact-based Metric** * **Flow:** Two parallel pipelines. 1. **Top Pipeline:** `User Query` -> `Information Extraction` -> `Facts` (represented as a stack of documents). 2. **Bottom Pipeline:** `LLM Generation` -> `Information Extraction` -> `Facts` (another stack). * **Comparison:** The two sets of extracted `Facts` are fed into a `Fact Overlap` module, which presumably computes a similarity or accuracy score based on shared factual elements. #### **(b) Classifier-based Metric** * **Flow:** A single, converging pipeline. * **Process:** Both the `User Query` and the `LLM Generation` are input simultaneously into an `NLI Model` (Natural Language Inference Model). * **Output:** The NLI Model produces an `Entailment` judgment, classifying the relationship between the query and the generation (e.g., entailment, contradiction, neutral). #### **(c) QA-based Metric** * **Flow:** A complex, multi-stage pipeline with a feedback loop. * **Process:** 1. The `User Query` is sent to a `Question Answering` module to produce a set of `Answers`. 2. The `LLM Generation` is sent to an `Answer Selection` module. 3. A `Question Generation` module creates `Questions` from the LLM's generation. These questions are fed back into the `Question Answering` module. 4. The `Answers` generated from the original user query and the answers selected/derived from the LLM generation are compared via an `Answer Overlap` metric. #### **(d) Uncertainty Estimation** * **Structure:** This diagram is distinct, showing the internal state of an LLM. * **Components:** * A large central block labeled `Large Language Model`. * **Input:** A `User Query` (yellow box) is fed into the model from the bottom. * **Output Sequence:** The model generates a sequence of tokens. Each token is associated with an uncertainty level, visualized as a colored box above it. * **Legend (Top Left):** Red box = `high uncertainty`, Green box = `low uncertainty`. * **Sequence Pattern:** The generated sequence shows an alternating pattern of high (red) and low (green) uncertainty tokens, culminating in an `EOS` (End of Sequence) token, which is marked with low uncertainty (green). * **Visual Metaphor:** Dotted, curved arrows above the uncertainty boxes suggest the model's internal sampling or probability distribution over the vocabulary at each step. #### **(e) Prompting-based Metric** * **Flow:** A direct evaluation pipeline. * **Process:** Both the `User Query` and the `LLM Generation` are fed into a central evaluation module, represented by a robot icon (suggesting another LLM or a specialized judge model). * **Outputs:** This evaluator produces one of two types of output: 1. `binary judgement` (e.g., good/bad, correct/incorrect). 2. `k-point Likert scale` (e.g., a rating from 1 to 5). ### Key Observations 1. **Methodological Spectrum:** The diagrams present a spectrum of evaluation complexity, from direct comparison of extracted facts (a) and classifier-based judgment (b), to more elaborate QA-based retrieval (c) and prompting-based LLM-as-judge approaches (e). 2. **Uncertainty Visualization:** Diagram (d) uniquely visualizes the token-by-token uncertainty of an LLM's generation process, highlighting that confidence can vary significantly within a single response. 3. **Common Inputs:** All evaluation metrics (a, b, c, e) fundamentally rely on comparing the `LLM Generation` against the original `User Query` or information derived from it. 4. **Abstraction Level:** The diagrams are high-level flowcharts. They abstract away specific model architectures, datasets, or scoring algorithms, focusing instead on the conceptual pipeline of each evaluation strategy. ### Interpretation This set of diagrams serves as a taxonomy of approaches for assessing LLM performance and reliability. It moves beyond simple accuracy metrics to illustrate more nuanced evaluation paradigms. * **Fact-based (a) and QA-based (c) Metrics** emphasize **grounding and retrieval**. They treat evaluation as a problem of verifying whether the LLM's output contains or aligns with specific, extractable pieces of information from a source (the query or an implicit knowledge base). The QA-based method is particularly sophisticated, using question generation to probe the LLM's output for answerable content. * **Classifier-based Metric (b)** frames evaluation as a **natural language inference task**, leveraging pre-trained models to judge logical relationships (entailment) between query and response. * **Prompting-based Metric (e)** represents the emerging paradigm of using **LLMs themselves as evaluators**, leveraging their language understanding to provide nuanced, scalable judgments that mimic human rating scales. * **Uncertainty Estimation (d)** addresses a critical, often hidden, dimension of LLM behavior: **model confidence**. The alternating high/low uncertainty pattern suggests that even within a coherent response, the model may be highly confident about some tokens (common words, factual anchors) while being much less sure about others (specific details, novel phrasing). This has profound implications for trustworthiness; a response might be fluent overall but contain uncertain, potentially hallucinated, elements. **Collectively, these diagrams argue that robust LLM evaluation is multi-faceted.** It requires checking factual consistency, logical soundness, answerability, and the model's own confidence. The choice of metric depends on the application's needs: a medical QA system might prioritize fact-based and uncertainty-aware metrics, while a creative writing assistant might be better served by a prompting-based Likert scale evaluation. The image provides a conceptual map for navigating these choices. </details> Figure 3. The illustration of detection methods for faithfulness hallucinations: a) Fact-based Metrics, which assesses faithfulness by measuring the overlap of facts between the generated content and the source content; b) Classifier-based Metrics, utilizing trained classifiers to distinguish the level of entailment between the generated content and the source content; c) QA-based Metrics, employing question-answering systems to validate the consistency of information between the source content and the generated content; d) Uncertainty Estimation, which assesses faithfulness by measuring the model’s confidence in its generated outputs; e) Prompting-based Metrics, wherein LLMs are induced to serve as evaluators, assessing the faithfulness of generated content through specific prompting strategies. Fact-based Metrics. In the realm of assessing faithfulness, one of the most intuitive methods involves measuring the overlap of pivotal facts between the generated content and the source content. Given the diverse manifestations of facts, faithfulness can be measured based on n-gram, entities, and relation triples. Traditional n-gram-based metrics, such as BLEU (Papineni et al., 2002), ROUGE (Lin, 2004) and PARENT-T (Wang et al., 2020b), typically fall short in differentiating the nuanced discrepancies between the generated content and the source content (Maynez et al., 2020). Entity-based metrics (Nan et al., 2021) make a step further by calculating the overlap of entities, as any omission or inaccurate generation of these key entities could lead to an unfaithful response. Notably, even if entities match, the relations between them might be erroneous. Thus, relation-based metrics (Goodrich et al., 2019) focus on the overlap of relation tuples and introduce a metric that computes the overlap of relation tuples extracted using trained end-to-end fact extraction models. Classifier-based Metrics. Beyond computing fact overlap, another straightforward approach to assessing the faithfulness of the model generation involves utilizing classifiers trained on data from related tasks such as natural language inference (NLI) and fact-checking, or data comprised of synthetically task-specific hallucinated and faithful content. A foundational principle for assessing the faithfulness of generated text is anchored on the idea that genuinely faithful content should inherently be entailed by its source content. In line with this, numerous studies (Falke et al., 2019; Maynez et al., 2020) have trained classifiers on NLI datasets to identify factual inaccuracies, especially in the context of abstract summarization. However, Mishra et al. (2021) highlighted that the mismatch in input granularity between conventional NLI datasets and inconsistency detection datasets limits their applicability for effectively detecting inconsistencies. Building on this, more advanced studies have proposed methods such as fine-tuning on adversarial datasets (Barrantes et al., 2020), decomposing the entailment decisions at the dependency arc level (Goyal and Durrett, 2020), and segmenting documents into sentence units then aggregating scores between sentence pairs (Laban et al., 2022). While using data from related tasks to fine-tune the classifier has shown promise in evaluating faithfulness, it’s essential to recognize the inherent gap between related tasks and the downstream task. The scarcity of annotated data further constrains their applicability. In response to this challenge, a surge of research explores leveraging data-augmentation methods to construct synthetical data for fine-tuning the classifier, either by rule-based perturbation (Dziri et al., 2021b; Kryscinski et al., 2020; Santhanam et al., 2021) or generation (Zhou et al., 2021). QA-based Metrics. In contrast to classifier-based metrics, QA-based metrics (Durmus et al., 2020; Wang et al., 2020a; Scialom et al., 2021; Honovich et al., 2021) have recently garnered attention for their enhanced ability to capture information overlap between the model’s generation and its source. These metrics operate by initially selecting target answers from the information units within the LLM’s output, and then questions are generated by the question-generation module. The questions are subsequently used to generate source answers based on the user context. Finally, the faithfulness of the LLM’s responses is calculated by comparing the matching scores between the source and target answers. Although these methodologies share a common thematic approach, they exhibit variability in aspects like answer selection, question generation, and answer overlap, leading to diverse performance outcomes. Building on this foundational work, Fabbri et al. (2022) conducted an in-depth evaluation of the components within QA-based metrics, yielding further enhancements in faithfulness evaluation. Uncertainty-based Metrics. Drawing parallels with the uncertainty-based approaches employed for detecting factuality hallucinations (§ 4.1.1), the application of uncertainty estimation in assessing faithfulness has been widely explored, typically characterized by entropy and log-probability. For entropy-based uncertainty, Xiao and Wang (2021) has revealed a positive correlation between hallucination likelihood in data-to-text generation and predictive uncertainty, which is estimated by deep ensembles (Lakshminarayanan et al., 2017). In a related vein, Guerreiro et al. (2023b) leveraged the variance in hypotheses yielded by Monte Carlo Dropout (Gal and Ghahramani, 2016) as an uncertainty measure within neural machine translation. More recently, van der Poel et al. (2022) employed conditional entropy (Xu et al., 2020) to assess model uncertainty in abstractive summarization. Regarding log-probability, it can be applied at different levels of granularity, such as word or sentence level. Notably, several studies (Guerreiro et al., 2023b; Yuan et al., 2021; Fu et al., 2023) have adopted length-normalized sequence log-probability to measure model confidence. Furthermore, considering the hallucinated token can be assigned high probability when the preceding context contains the same hallucinated information, Zhang et al. (2023h) focused on the most informative and important keywords and introduced a penalty mechanism to counteract the propagation of hallucinated content. LLM-based Judgement. Recently, the remarkable instruction-following ability of LLMs has underscored their potential for automatic evaluation (Chiang and Lee, 2023; Liu et al., 2023c; Wang et al., 2023d). Exploiting this capability, researchers have ventured into novel paradigms for assessing the faithfulness of model-generated content (Luo et al., 2023b; Laban et al., 2023; Adlakha et al., 2023; Gao et al., 2023b; Jain et al., 2023). By providing LLMs with concrete evaluation guidelines and feeding them both the model-generated and source content, they can effectively assess faithfulness. The final evaluation output can either be a binary judgment on faithfulness (Luo et al., 2023b) or a k-point Likert scale indicating the degree of faithfulness (Gao et al., 2023b). For prompt selection, evaluation prompt can either be direct prompting, chain-of-thought prompting (Adlakha et al., 2023), using in-context-learning (Jain et al., 2023) or allowing the model to generate evaluation results accompanying with explanations (Laban et al., 2023). ### 4.2. Hallucination Benchmarks In this section, we present a comprehensive overview of existing hallucination benchmarks, which can be categorized into two primary domains: Hallucination Evaluation Benchmarks (§ 4.2.1), which assess the extent of hallucinations generated by existing cutting-edge LLMs, and Hallucination Detection Benchmarks (§ 4.2.2), designed specifically to evaluate the performance of existing hallucination detection methods. Collectively, these benchmarks establish a unified framework, enabling a nuanced and thorough exploration of hallucinatory patterns in LLMs. Table 4. An overview of existing hallucination benchmarks. For Attribute, Factuality and Faithfulness represent whether the benchmark is used to evaluate LLM’s factuality or to detect faithfulness hallucination, and Manual represents whether the inputs in the data are handwritten. | | | | | Attribute | Task | | | | | | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | Benchmark | Datasets | Data Size | Language | Factuality | Faithfulness | Manual | Task Type | Input | Label | Metric | | Generative QA | | | | | | | | | | | | TruthfulQA (Lin et al., 2022) | - | 817 | English | ✔ | ✗ | ✔ | Multi-Choice QA | Question | Answer | LLM-Judge & Human | | Multi-Choice QA | | | Acc | | | | | | | | | REALTIMEQA (Kasai et al., 2022) | - | Dynamic | English | ✔ | ✗ | ✔ | Generative QA | Question | Answer | EM & F1 | | SelfCheckGPT-Wikibio (Miao et al., 2023) | - | 1,908 | English | ✗ | ✔ | ✗ | Detection | Paragraph & Concept | Passage | AUROC | | Task-specific | 30,000 | English | ✗ | ✔ | ✗ | Detection | Query | Response | Acc | | | HaluEval (Li et al., 2023b) | General | 5,000 | English | ✗ | ✔ | ✗ | Detection | Task Input | Response | Acc | | Med-HALT (Umapathi et al., 2023) | - | 4,916 | Multilingual | ✔ | ✗ | ✗ | Multi-Choice QA | Question | Choice | Pointwise Score & Acc | | Wiki-FACTOR | 2,994 | English | ✔ | ✗ | ✗ | Multi-Choice QA | Question | Answer | likelihood | | | FACTOR (Muhlgay et al., 2023) | News-FACTOR | 1,036 | English | ✔ | ✗ | ✗ | Multi-Choice QA | Question | Answer | likelihood | | SenHallu | 200 | English | ✗ | ✔ | ✗ | Detection | Paper | Summary | P & R & F1 | | | BAMBOO (Dong et al., 2023) | AbsHallu | 200 | English | ✗ | ✔ | ✗ | Detection | Paper | Summary | P & R & F1 | | ChineseFactEval (Wang et al., 2023b) | - | 125 | Chinese | ✔ | ✗ | ✔ | Generative QA | Question | - | Score | | Misleading | 175 | Chinese | ✔ | ✗ | ✔ | Generative QA | Question | Answer | LLM-Judge | | | Misleading-hard | 69 | Chinese | ✔ | ✗ | ✔ | Generative QA | Question | Answer | LLM-Judge | | | HaluQA (Cheng et al., 2023) | Knowledge | 206 | Chinese | ✔ | ✗ | ✔ | Generative QA | Question | Answer | LLM-Judge | | Never-changing | 150 | English | ✔ | ✗ | ✔ | Generative QA | Question | Answer | Human | | | Slow-changing | 150 | English | ✔ | ✗ | ✔ | Generative QA | Question | Answer | Human | | | Fast-changing | 150 | English | ✔ | ✗ | ✔ | Generative QA | Question | Answer | Human | | | FreshQA (Vu et al., 2023) | False-premise | 150 | English | ✔ | ✗ | ✔ | Generative QA | Question | Answer | Human | | FELM (Chen et al., 2023e) | - | 3,948 | English | ✔ | ✔ | ✗ | Detection | Question | Response | Balanced Acc & F1 | | PHD-LOW | 100 | English | ✗ | ✔ | ✗ | Detection | Entity | Response | P & R & F1 | | | PHD-Meidum | 100 | English | ✗ | ✔ | ✗ | Detection | Entity | Response | P & R & F1 | | | PHD (Yang et al., 2023b) | PHD-High | 100 | English | ✗ | ✔ | ✗ | Detection | Entity | Response | P & R & F1 | | ScreenEval (Lattimer et al., 2023) | - | 52 | English | ✗ | ✔ | ✗ | Detection | Document | Summary | AUROC | | COVID-QA | N/A | English | ✗ | ✔ | ✗ | Detection | Question | Answer | AUROC | | | DROP | N/A | English | ✗ | ✔ | ✗ | Detection | Question | Answer | AUROC | | | Open Assistant | N/A | English | ✗ | ✔ | ✗ | Detection | Question | Answer | AUROC | | | RealHall (Friel and Sanyal, 2023) | TriviaQA | N/A | English | ✗ | ✔ | ✗ | Detection | Question | Answer | AUROC | | LSum (Feng et al., 2023a) | - | 6,166 | English | ✗ | ✔ | ✗ | Detection | Document | Summary | Balanced Acc | | HotpotQA | 250 | English | ✗ | ✔ | ✗ | Detection | Question | Answer | AUROC | | | SAC 3 (Zhang et al., 2023e) | NQ-Open | 250 | English | ✗ | ✔ | ✗ | Detection | Question | Answer | AUROC | | Biomedicine | 1,535 | English | ✔ | ✗ | ✗ | Generative QA | Question | Answer | MiHR & MaHR | | | Finance | 1,125 | English | ✔ | ✗ | ✗ | Generative QA | Question | Answer | MiHR & MaHR | | | Science | 1,409 | English | ✔ | ✗ | ✗ | Generative QA | Question | Answer | MiHR & MaHR | | | Education | 1,701 | English | ✔ | ✗ | ✗ | Generative QA | Question | Answer | MiHR & MaHR | | | HaluEval 2.0 (Li et al., 2024c) | Open domain | 3,000 | English | ✔ | ✗ | ✗ | Generative QA | Question | Answer | MiHR & MaHR | #### 4.2.1. Hallucination Evaluation Benchmarks Hallucination evaluation benchmarks are devised to quantify the tendency of LLMs to generate hallucinations, particularly emphasizing factual inaccuracies and inconsistency from the given contexts. Given the adeptness of LLMs at memorizing high-frequency count knowledge, the primary focus of current hallucination evaluation benchmarks targets long-tailed knowledge and challenging questions that can easily elicit imitative falsehood. As for evaluating, these benchmarks typically utilize multiple choice QA, where performance is measured through accuracy metrics, or generative QA, evaluated either through human judgment or scores given by proxy models. Long-tail Factual Knowledge. The selection criteria for gathering long-tail factual question-answering samples typically include the frequency of appearance, recency, and specific domains. Regarding the frequency of appearance, benchmarks such as PopQA (Mallen et al., 2023) and Head-to-Tail (Sun et al., 2023) are constructed based on entity popularity derived directly from Wikipedia. Considering that world knowledge is constantly evolving, it becomes crucial to validate the LLM’s factuality concerning the current world. Among benchmarks characterized by ever-changing, REALTIMEQA (Kasai et al., 2022) and FreshQA (Vu et al., 2023) stands out. REALTIMEQA offers real-time, open-domain multiple-choice questions that are regularly updated to reflect the latest developments. These questions are derived from newly published news articles, encompassing a broad spectrum of topics, including politics, business, sports, and entertainment. Similarly, FreshQA challenges LLMs with questions designed to represent varying degrees of temporal change—categorized into never-changing, slow-changing, and fast-changing world knowledge. This benchmark is further enriched by including questions based on false premises, requiring debunking, thus comprising a total of 600 meticulously hand-crafted questions. Moreover, long-tail knowledge often pertains to specific domains. For instance, Med-HALT (Umapathi et al., 2023) is distinguished by its focus on the medical domain, challenging LLMs with multiple-choice questions derived from a variety of countries. Additionally, Malaviya et al. (2023) collected expert-curated questions across 32 fields of study, resulting in a high-quality long-form QA dataset with 2,177 questions. Imitative Falsehood Knowledge. Imitative falsehood knowledge is specifically designed to challenge LLMs through adversarial prompting. This approach crafts questions in such a way that they are prone to misleading LLMs due to false beliefs or misconceptions. The two most representative benchmarks are TruthfulQA (Lin et al., 2022) and HalluQA (Cheng et al., 2023). TruthfulQA comprises 817 questions that span 38 diverse categories, such as health, law, finance, and politics. Crafted using an adversarial methodology, it aims to elicit ”imitative falsehoods”—misleading responses that models might generate due to their frequent presence in training data. The benchmark is divided into two parts, one of which contains manually curated questions that were further refined by filtering out those correctly answered by GPT-3, resulting in 437 filtered questions. The other part includes 380 unfiltered non-adversarial questions. Drawing from the construction approach of TruthfulQA, HalluQA is crafted to specifically assess hallucinations in Chinese LLMs, focusing on imitative falsehoods and factual errors. The benchmark comprises 450 handcrafted adversarial questions across 30 domains and is categorized into two parts. The misleading section captures questions that successfully deceive GLM-130B, while the knowledge section retains questions that both ChatGPT and Puyu consistently answer incorrectly. To comprehensively evaluate LLM hallucinations across various domains, Li et al. (2024c) constructed an upgraded hallucination evaluation benchmark, HaluEval 2.0, based on (Li et al., 2023b). This benchmark includes 8,770 questions that LLMs are prone to hallucination, across five domains: biomedicine, finance, science, education, and open domain. #### 4.2.2. Hallucination Detection Benchmarks For hallucination detection benchmarks, most prior studies have primarily concentrated on task-specific hallucinations, such as abstractive summarization (Kryscinski et al., 2020; Wang et al., 2020a; Maynez et al., 2020; Fabbri et al., 2021; Goyal and Durrett, 2021; Pagnoni et al., 2021), data-to-text (Tian et al., 2019; Parikh et al., 2020), and machine translation (Zhou et al., 2021). However, the content generated in these studies often originates from models with lesser capabilities, such as BART (Lewis et al., 2020a) and PEGASUS (Zhang et al., 2020). As a result, they may not accurately reflect the effectiveness of hallucination detection strategies, underlining the necessity for a significant shift toward developing benchmarks that encapsulate more complex scenarios reflective of the era of LLMs. For example, SelfCheckGPT-Wikibio (Miao et al., 2023) offers a sentence-level dataset created by generating synthetic Wikipedia articles with GPT-3, manually annotated for factuality, highlighting the challenge of detecting hallucinations in the biography domain. Complementing this, HaluEval (Li et al., 2023b) combines automated generation with human annotation to evaluate LLMs’ ability to recognize hallucinations across 5,000 general user queries and 30,000 task-specific samples, leveraging a ”sampling-then-filtering” approach. Building upon existing research predominantly focused on short documents, BAMBOO (Dong et al., 2023) and ScreenEval (Lattimer et al., 2023) extend the scope in long-form hallucination detection. Further, FELM (Chen et al., 2023e), distinguishes itself by assessing factuality across diverse domains including world knowledge, science, and mathematics, producing 817 samples annotated for various facets of factual accuracy, thereby addressing the need for cross-domain evaluation of factuality in LLM-generated content. On a different note, PHD (Yang et al., 2023b), shifts the focus towards passage-level detection of non-factual content by analyzing entities from Wikipedia, thus offering a nuanced view on the knowledge depth of LLMs. RealHall (Friel and Sanyal, 2023) and SAC 3 (Zhang et al., 2023e) align closely with real-world applications focusing on open-domain question-answering, whereas LSum (Feng et al., 2023a) concentrating on summarization tasks. ## 5. Hallucination Mitigation In this section, we present a comprehensive review of contemporary methods aimed at mitigating hallucinations in LLMs. Drawing from insights discussed in Hallucination Causes (§ 3), we systematically categorize these methods based on the underlying causes of hallucinations. Specifically, we focus on approaches addressing Data-related Hallucinations (§ 5.1), Training-related Hallucinations (§ 5.2) and Inference-related Hallucinations (§ 5.3), each offering tailored solutions to tackle specific challenges inherent to their respective cause. ### 5.1. Mitigating Data-related Hallucinations As analyzed in § 3.1, data-related hallucinations generally emerge as a byproduct of misinformation, biases, and knowledge gaps, which are fundamentally rooted in the pre-training data. Several methods are proposed to mitigate such hallucinations, primarily categorized into three distinct parts: (1) data filtering aiming at selecting high-quality data to avoid introducing misinformation and biases, (2) model editing focusing on injecting up-to-date knowledge by editing model’s parameters, and (3) retrieval-augmented generation leveraging external non-parametric database for knowledge supplying. #### 5.1.1. Data Filtering To reduce the presence of misinformation and biases, an intuitive approach involves the careful selection of high-quality pre-training data from reliable sources. In this way, we can ensure the factual correctness of data while also minimizing the introduction of social biases. As early as the advent of GPT-2, Radford et al. (2019) underscored the significance of exclusively scraping web pages that had undergone rigorous curation and filtration by human experts. However, as pre-training datasets continue to scale, manual curation becomes a challenge. Given that academic or specialized domain data is typically factually accurate, gathering high-quality data emerges as a primary strategy. Notable examples include the Pile (Gao et al., 2021a) and “textbook-like” data sources (Gunasekar et al., 2023; Li et al., 2023a). Additionally, up-sampling factual data during the pre-training phase has been proven effective in enhancing the factual correctness of LLMs (Touvron et al., 2023b), thus alleviating hallucination. In addition to strictly controlling the source of data, deduplication serves as a crucial procedure. Existing practices typically fall into two categories: exact duplicates and near-duplicates. For exact duplicates, the most straightforward method involves exact substring matching to identify identical strings. However, given the vastness of pre-training data, this process can be computationally intensive, a more efficient method utilizes the construction of a suffix array (Manber and Myers, 1993), enabling effective computation of numerous substring queries in linear time. Regarding near-duplicates, the identification often involves approximate full-text matching, typically utilizing hash-based techniques to identify document pairs with significant n-gram overlap. Furthermore, MinHash (Broder, 1997) stands out as a prevalent algorithm for large-scale deduplication tasks (Gyawali et al., 2020). Additionally, SemDeDup (Abbas et al., 2023) makes use of embeddings from pre-trained models to identify semantic duplicates, which refers to data pairs with semantic similarities but not identical. Discussion. Since data filtering works directly at the source of hallucinations, it effectively mitigates hallucinations by ensuring the use of high-quality, factually accurate sources. Despite its effectiveness, the efficiency and scalability of current data filtering methods pose significant challenges as data volumes expand. Additionally, these methods often overlook the influence of LLM-generated content, which can introduce new risks and inaccuracies. To advance, future research must focus on developing more efficient, automated data filtering algorithms that can keep pace with the rapid expansion of datasets and the complexities of LLM-generated content. #### 5.1.2. Model Editing Model editing (Sinitsin et al., 2020; Wang et al., 2023j; Zhang et al., 2024e) has garnered rising attention from researchers, which aims to rectify model behavior by incorporating additional knowledge. Current model editing techniques can be categorized into two classes: locate-then-edit and meta-learning. Locate-then-edit. Locate-then-edit methods (Dai et al., 2022a; Meng et al., 2022) consist of two stages, which first locate the “buggy” part of the model parameters and then apply an update to them to alter the model’s behavior. For example, ROME (Meng et al., 2022) located the edits-related layer by destroying and subsequently restoring the activations and then updates the parameters of FFN in a direct manner to edit knowledge. MEMIT (Meng et al., 2023) employed the same knowledge locating methods as ROME, enabling the concurrent updating of multiple layers to facilitate the simultaneous integration of thousands of editing knowledge. However, Yao et al. (2023b) found that these methods lack non-trivial generalization capabilities and varying performance and applicability to different model architectures. The best-performing methods ROME and MEMIT empirically only work well on decoder-only LLMs. Meta-learning. Meta-learning methods (De Cao et al., 2021; Mitchell et al., 2022a) train an external hyper-network to predict the weight update of the original model. Nevertheless, meta-learning methods often require additional training and memory cost, where MEND (Mitchell et al., 2022a) utilized a low-rank decomposition with a specialized design to reduce the size of hyper-networks. Notably, MEND would exhibit a cancellation effect, where parameter shifts corresponding to different keys significantly counteract each other. MALMEN (Tan et al., 2024b) further addressed this issue by framing the parameter shift aggregation as a least squares problem rather than a simple summation, thereby greatly enhancing its capacity for extensive editing. While these methods can fine-grainedly adjust the behavior of the model, modifications to the parameters could have a potentially harmful impact on the inherent knowledge of the model. Discussion. Model editing provides a precise way to mitigate hallucinations induced by specific misinformation without extensive retraining. However, these methods struggle with large-scale updates and can adversely affect the model’s overall performance, particularly when continuous edits are applied. Consequently, future research should focus on improving model editing to handle large-scale knowledge updates more efficiently and address hallucinations caused by social biases. #### 5.1.3. Retrieval-Augmented Generation Typically, retrieval-augmented generation (RAG) (Lewis et al., 2020b; Guu et al., 2020; Shuster et al., 2021) follows a retrieve-then-read pipeline, where relevant knowledge is firstly retrieved by a retriever (Karpukhin et al., 2020a) from external sources, and then the final response is generated by a generator conditioning on both user query and retrieved documents. By decoupling external knowledge from LLM, RAG can effectively alleviate the hallucination caused by the knowledge gap without affecting the performance of LLM. Common practices can be divided into three parts, as shown in Fig 4: one-time retrieval, iterative retrieval, and post-hoc retrieval, depending on the timing of retrieval. <details> <summary>x3.png Details</summary>  ### Visual Description ## Diagram: Retrieval-Augmented Generation (RAG) Architectures ### Overview The image is a technical diagram illustrating three distinct architectural patterns for Retrieval-Augmented Generation (RAG) systems. It is divided into three vertical panels, separated by dashed lines, each depicting a different workflow labeled (a), (b), and (c). The diagram uses a consistent color scheme and iconography to represent components like queries, databases, language models, and outputs. ### Components/Axes The diagram is not a chart with axes but a flow diagram. The components are represented by colored boxes and icons, connected by directional arrows indicating data flow. **Common Components & Labels:** * **Query:** Pink rectangular box. Present at the start of all three workflows. * **Database/Retrieval Icon:** A gray cylinder with a magnifying glass symbol. Represents the knowledge source or retrieval mechanism. * **Large Language Model (LLM):** Light blue rounded rectangle. The core processing unit. * **Output:** Light green rectangular box. Represents intermediate results. * **Answer:** Light yellow rectangular box. The final generated response. * **Revisor:** Light orange rectangular box. A component for refining answers. * **Action Labels:** Text along arrows describes the process step (e.g., "Retrieve", "Generate", "Revise"). * **Process Labels:** Dashed boxes group iterative or revision loops, labeled "Iteration" or "Revision". **Spatial Layout:** * **Panel (a) - Left:** Titled "(a) One-time Retrieval". A linear, top-to-bottom flow. * **Panel (b) - Center:** Titled "(b) Iterative Retrieval". Features a repeating loop within a dashed "Iteration" box. * **Panel (c) - Right:** Titled "(c) Post-hoc Retrieval". Features a linear generation step followed by a revision loop within a dashed "Revision" box. ### Detailed Analysis **Panel (a): One-time Retrieval** 1. **Flow:** `Query` -> (Retrieve) -> `Database` -> `Large Language Model` -> (Generate) -> `Answer`. 2. **Description:** This is a straightforward, single-pass architecture. The user query triggers a single retrieval operation from the database. The retrieved information and the original query are then fed into the LLM, which generates the final answer in one step. **Panel (b): Iterative Retrieval** 1. **Flow:** `Query` -> Enters an `Iteration` loop -> `Database` -> `LLM` -> `Output` -> loops back to `Database` for the next iteration. After one or more iterations, the process exits the loop to produce the final `Answer`. 2. **Description:** This architecture employs a multi-step retrieval process. The system can iteratively refine its understanding by making multiple calls to the database. Each iteration uses the context from the previous `Output` to inform the next retrieval query, potentially gathering more relevant or nuanced information before final answer generation. **Panel (c): Post-hoc Retrieval** 1. **Flow:** `Query` -> `Large Language Model` -> (Generate) -> `Answer`. This initial `Answer` then enters a `Revision` loop -> (Retrieve) -> `Database` -> `Revisor` -> (Revise) -> back to `Answer`. 2. **Description:** This pattern separates initial generation from verification and refinement. The LLM first generates an answer based on its parametric knowledge. This answer is then passed to a "Revisor" module, which can retrieve supporting or contradicting evidence from the database to fact-check and revise the answer, creating a self-correcting loop. ### Key Observations 1. **Progression of Complexity:** The architectures progress from a simple linear pipeline (a) to systems with internal feedback loops (b and c). 2. **Timing of Retrieval:** The core differentiator is *when* retrieval occurs relative to LLM processing: * (a) **Before** LLM processing (single time). * (b) **During** LLM processing (multiple times, interleaved). * (c) **After** initial LLM processing (for verification). 3. **Component Roles:** The "LLM" in (a) and (b) is the primary answer generator. In (c), the "LLM" is the initial generator, while the "Revisor" is a specialized component for refinement. 4. **Loop Structures:** Panel (b) shows a *generation loop* (improving the answer draft). Panel (c) shows a *revision loop* (correcting a completed draft). ### Interpretation This diagram effectively contrasts fundamental strategies for integrating external knowledge with Large Language Models. It moves beyond the basic "retrieve-then-generate" model to illustrate more sophisticated, agentic patterns. * **One-time Retrieval (a)** is the simplest and most common RAG setup, suitable for straightforward QA where a single, focused retrieval is sufficient. Its weakness is a lack of opportunity to correct initial retrieval mistakes. * **Iterative Retrieval (b)** suggests a more exploratory or complex reasoning process. It mimics how a human might research a topic—performing an initial search, reading the results, and then conducting follow-up searches based on new understanding. This is powerful for multi-faceted questions but increases latency and cost. * **Post-hoc Retrieval (c)** introduces a critical self-check mechanism. It acknowledges that LLMs can hallucinate or make errors even with retrieved context. By generating an answer first and then challenging it with retrieved evidence, this architecture aims for higher factual reliability and is akin to a "draft and fact-check" workflow. The "Revisor" could be a separate, smaller model or a specialized prompting strategy. The diagram visually argues that advanced RAG systems are not static pipelines but dynamic processes involving loops of retrieval, generation, and evaluation. The choice between these architectures involves a trade-off between simplicity, depth of reasoning, and computational cost. </details> Figure 4. The illustration of three distinct approaches for Retrieval-Augmented Generation: a) One-time Retrieval, where relevant information is retrieved once before text generation; b) Iterative Retrieval, involving multiple retrieval iterations during text generation for dynamic information integration; and c) Post-hoc Retrieval, where the retrieval process happens after an answer is generated, aiming to refine and fact-check the generated content. One-time Retrieval. One-time retrieval aims to directly prepend the external knowledge obtained from a single retrieval to the LLMs’ prompt. Ram et al. (2023) introduced In-context RALM, which entails a straightforward yet effective strategy of prepending chosen documents to the input text of LLMs. Beyond conventional knowledge repositories such as Wikipedia, ongoing research endeavors have explored alternative avenues, specifically the utilization of knowledge graphs (KGs). These KGs serve as a pivotal tool for prompting LLMs, facilitating their interaction with the most recent knowledge, and eliciting robust reasoning pathways (Wen et al., 2023; Qi et al., 2023; Baek et al., 2023). Varshney et al. (2023) introduce the Parametric Knowledge Guiding (PKG) framework, enhancing LLMs with domain-specific knowledge. PKG employs a trainable background knowledge module, aligning it with task knowledge and generating relevant contextual information. Iterative Retrieval. When confronted with intricate challenges like multi-step reasoning (Yang et al., 2018b) and long-form question answering (Fan et al., 2019; Stelmakh et al., 2022), traditional one-time retrieval may fall short. Addressing these demanding information needs, recent studies have proposed iterative retrieval, which allows for continuously gathering knowledge throughout the generation process. Recognizing the substantial advancements chain-of-thought prompting (Wei et al., 2022) has brought to LLMs in multi-step reasoning, numerous studies (Yao et al., 2022; Trivedi et al., 2023; He et al., 2023b) try to incorporate external knowledge at each reasoning step and further guide retrieval process based on ongoing reasoning, reducing factual errors in reasoning chains. Building upon chain-of-thought prompting, Press et al. (2022) introduced self-ask. Diverging from the conventional continuous, undelineated chain-of-thought prompting, self-ask delineates the question it intends to address at each step, subsequently incorporating a search action based on the follow-up question. Instead of solely depending on chain-of-thought prompting for retrieval guidance, both Feng et al. (2023b) and Shao et al. (2023) employed an iterative retrieval-generation collaborative framework, where a model’s response serves as an insightful context to procure more relevant knowledge, subsequently refining the response in the succeeding iteration. Beyond multi-step reasoning tasks, Jiang et al. (2023b) shifted their emphasis to long-form generation. They proposed an active retrieval augmented generation framework, which iteratively treats the upcoming prediction as a query to retrieve relevant documents. If the prediction contains tokens of low confidence, the sentence undergoes regeneration. In addition to using iterative retrieval to improve intermediate generations, Zhang et al. (2023f) presented MixAlign, which iteratively refines user questions using model-based guidance and seeking clarifications from users, ultimately enhancing the alignment between questions and knowledge. Post-hoc Retrieval. Beyond the traditional retrieve-then-read paradigm, a line of work has delved into post-hoc retrieval, refining LLM outputs through subsequent retrieval-based revisions. To enhance the trustworthiness and attribution of LLMs, Gao et al. (2023a) adopted the research-then-revise workflow, which initially research relevant evidence and subsequently revise the initial generation based on detected discrepancies with the evidence. Similarly, Zhao et al. (2023b) introduced the verify-and-edit framework to enhance the factual accuracy of reasoning chains by incorporating external knowledge. For reasoning chains that show lower-than-average consistency, the framework generates verifying questions and then refines the rationales based on retrieved knowledge, ensuring a more factual response. Yu et al. (2023c) enhanced the post-hoc retrieval method through diverse answer generation. Instead of generating just a single answer, they sample various potential answers, allowing for a more comprehensive retrieval feedback. Additionally, by employing an ensembling technique that considers the likelihood of the answer before and after retrieval, they further mitigate the risk of misleading retrieval feedback. Discussion. One crucial advantage of retrieval-augmented generation methodology is its effectiveness in mitigating hallucinations caused by knowledge gaps, and their generality, which allows for application across any domain. This flexibility is further enhanced by the modularity of the approach, treating external knowledge bases like plug-ins that can be swapped or modified as needed. In terms of the drawbacks, it can be easily impacted by irrelevant retrievals, which may decrease the overall performance by introducing noise or incorrect information into the response generation process. Furthermore, the current paradigm exhibits shallow interactions between the retriever and generator components, leading to suboptimal knowledge utilization. Hence, future research should focus on developing a robust RAG system that minimizes the impact of irrelevant retrieval, as well as integrating adaptive learning components that can dynamically adjust retrieval strategies based on the context of the query and the performance of previous interactions. ### 5.2. Mitigating Training-related Hallucination Training-related hallucinations typically arise from the intrinsic limitations of the architecture and training strategies adopted by LLMs. In this context, we discuss various optimization methods ranging from training stages (§ 5.2.1) and alignment stages (SFT & RLHF) (§ 5.2.2), aiming to mitigate hallucinations within the training process. #### 5.2.1. Mitigating Pretraining-related Hallucination One significant avenue of research in mitigating pretraining-related hallucination centers on the limitations inherent in model architectures, especially unidirectional representation and attention glitches. In light of this, numerous studies have delved into designing novel model architectures specifically tailored to address these flaws. To address the limitations inherent in unidirectional representation, Li et al. (2023i) introduced BATGPT which employs a bidirectional autoregressive approach. This design allows the model to predict the next token based on all previously seen tokens, considering both past and future contexts, thus capturing dependencies in both directions. Building on this idea, Liu et al. (2023f) highlighted the potential of encoder-decoder models to make better use of their context windows, suggesting a promising direction for future LLMs architecture design. Besides, recognizing the limitations of soft attention within self-attention-based architecture, Liu et al. (2023a) proposed attention-sharpening regularizers. This plug-and-play approach specifies self-attention architectures using differentiable loss terms (Zhang et al., 2018) to promote sparsity, leading to a significant reduction in reasoning hallucinations. In the pre-training phase of LLMs, the choice of objective plays a pivotal role in determining the model’s performance. However, conventional objectives can lead to fragmented representations and inconsistencies in model outputs. Recent advancements have sought to address these challenges by refining pre-training strategies, ensuring richer context comprehension, and circumventing biases. Addressing the inherent limitations in training LLMs, where unstructured factual knowledge at a document level often gets chunked due to GPU memory constraints and computational efficiency, leading to fragmented information and incorrect entity associations, Lee et al. (2022b) introduced a factuality-enhanced training method. By appending a TOPICPREFIX to each sentence in factual documents, the approach transforms them into standalone facts, significantly reducing factual errors and enhancing the model’s comprehension of factual associations. Similarly, considering that randomly concatenating shorter documents during pre-training might introduce inconsistencies in model outputs, Shi et al. (2023b) proposed In-Context Pretraining, an innovative approach in which LLMs are trained on sequences of related documents. By altering the document order, this method aims to maximize similarity within the context windows. It explicitly encourages LLMs to reason across document boundaries, potentially bolstering the logical consistency between generations. Discussion. Strategies designed to mitigate pretraining-related hallucinations typically are fundamental, potentially yielding significant improvements. However, they typically involve modifications to pre-training architectures and objectives, which are computationally intensive. Moreover, these integrations may lack broad applicability. Moving forward, the focus should be on developing adaptable and efficient strategies that can be universally applied without extensive system overhaul. #### 5.2.2. Mitigating Misalignment Hallucination Hallucinations induced during alignment often stem from capability misalignment and belief misalignment. However, defining the knowledge boundary of LLMs proves challenging, making it difficult to bridge the gap between LLMs’ inherent capabilities and the knowledge presented in human-annotated data. While limited research addresses capability misalignment, the focus mainly shifts toward belief misalignment. Hallucinations stemming from belief misalignment often manifest as sycophancy, a tendency of LLMs to seek human approval in undesirable ways. This sycophantic behavior can be attributed to the fact that human preference judgments often favor sycophantic responses over more truthful ones (Sharma et al., 2023), paving the way for reward hacking (Saunders et al., 2022). To address this, a straightforward strategy is to improve human preference judgments and, by extension, the preference model. Recent research (Bowman et al., 2022; Saunders et al., 2022) has investigated the use of LLMs to assist human labelers in identifying overlooked flaws. Additionally, Sharma et al. (2023) discovered that aggregating multiple human preferences enhances feedback quality, thereby reducing sycophancy. Besides, modifications to LLMs’ internal activations have also shown the potential to alter model behavior. This can be achieved through methods like fine-tuning (Wei et al., 2023) or activation steering during inference (Dathathri et al., 2020; Subramani et al., 2022; Hernandez et al., 2023). Specifically, Wei et al. (2023) proposed a synthetic-data intervention, fine-tuning language models using synthetic data where the claim’s ground truth is independent of a user’s opinion, aiming to reduce sycophantic tendencies. Another avenue of research (Rimsky, 2023b, a) has been to mitigate sycophancy through activation steering. This approach involves using pairs of sycophantic/non-sycophantic prompts to generate the sycophancy steering vector, derived from averaging the differences in intermediate activations. During inference, subtracting this vector can produce less sycophantic LLM outputs. Discussion. Mitigating hallucinations through post-training methods represents a direct and effective approach, bypassing the complexities associated with data sourcing and pre-training. However, a notable gap in current research is the limited attention given to capability misalignment within LLMs. Future research should prioritize understanding the knowledge boundaries in capability alignment to address hallucinations effectively. ### 5.3. Mitigating Inference-related Hallucination Decoding strategies in LLMs play a pivotal role in determining the factuality and faithfulness of the generated content. However, as analyzed in Section § 3.3, imperfect decoding often results in outputs that might lack factuality or stray from the original context. In this subsection, we explore two advanced strategies aimed at refining the decoding strategy to enhance both the factuality and faithfulness of the LLMs’ outputs. #### 5.3.1. Factuality Enhanced Decoding Factuality Enhanced Decoding aims to improve the reliability of outputs from LLMs by prioritizing the factuality of the information they generate. This line of methods focuses on aligning model outputs closely with established real-world facts, thereby minimizing the risk of disseminating false or misleading information. Factuality Decoding. Considering the randomness in the sampling process can introduce non-factual content into open-ended text generation, Lee et al. (2022b) introduced the factual-nucleus sampling algorithm that dynamically adjusts the nucleus probability $p$ throughout sentence generation. By dynamically adjusting the nucleus probability based on decay factors and lower boundaries and resetting the nucleus probability at the beginning of every new sentence, the decoding strategy strikes a balance between generating factual content and preserving output diversity. Moreover, some studies (Burns et al., 2022; Moschella et al., 2022) posit that the activation space of LLMs contains interpretable structures related to factuality. Building on this idea, Li et al. (2023f) introduced Inference-Time Intervention (ITI). This method first identifies a direction in the activation space associated with factually correct statements and then adjusts activations along the truth-correlated direction during inference. By repeatedly applying such intervention, LLMs can be steered towards producing more factual responses. Similarly, Chuang et al. (2023) delved into enhancing the factuality of LLM’s decoding process from a perspective of factual knowledge storage. They exploit the hierarchical encoding of factual knowledge within transformer LLMs, noting that lower-level information is captured in earlier layers and semantic information in the later ones. Drawing inspiration from (Li et al., 2022a), they introduce DoLa, a strategy that dynamically selects and contrasts logits from different layers to refine decoding factuality. By placing emphasis on knowledge from higher layers and downplaying that from the lower layers, DoLa showcases its potential to make LLMs more factual, thus reducing hallucinations. Post-editing Decoding. Unlike methods that directly modify the probability distribution to prevent hallucinations during the initial decoding, post-editing decoding seeks to harness the self-correction capabilities of LLMs (Pan et al., 2023) to refine the originally generated content without relying on an external knowledge base. Dhuliawala et al. (2023) introduced the Chain-of-Verification (COVE), which operates under the assumption that, when appropriately prompted, LLMs can self-correct their mistakes and provide more accurate facts. Starting with an initial draft, it first formulates verification questions and then systematically answers those questions in order to finally produce an improved revised response. Similarly, Ji et al. (2023b) focused on the medical domain and introduced an iterative self-reflection process. This process leverages the inherent ability of LLMs to first generate factual knowledge and then refine the response until it aligns consistently with the provided background knowledge. Discussion. Factuality decoding methods, which typically assess the factuality at each decoding step, can offer substantial improvements. Furthermore, due to their plug-and-play nature, they allow for application without the need for computation-intensive training. Nevertheless, one of the primary limitations of these methods lies in balancing factual accuracy with maintaining the diversity and informativeness of the generated content, which can sometimes lead to compromises in either aspect. On the other hand, post-editing decoding strategies, despite their effectiveness, heavily rely on the self-correction capabilities of LLMs, which may be unreliable. Furthermore, applying self-reflection can be time-consuming, limiting their practicality for real-time applications. Hence, it is crucial to achieve an optimal balance between factuality and computational efficiency. #### 5.3.2. Faithfulness Enhanced Decoding On the other hand, Faithfulness Enhanced Decoding prioritizes alignment with the provided context and also emphasizes enhancing the consistency within the generated content. Thus, in this section, we summarize existing work into two categories, including Context Consistency and Logical Consistency. Context Consistency. In the era of LLMs, the issue of faithfulness hallucination typically lies in insufficient attention to the given context, which inspired numerous research to design inference-time strategies to enhance context consistency. Shi et al. (2023a) proposed context-aware decoding (CAD), which modifies the model’s original output distribution in a contrastive formulation (Li et al., 2022a). By amplifying the difference between output probabilities with and without context, CAD encourages the LLM to focus more on contextual information rather than over-rely on prior knowledge. However, due to the inherent trade-off between diversity and context attribution (Zhang et al., 2021; Gu et al., 2022), overemphasizing contextual information can reduce diversity. To address this, Chang et al. (2023a) introduced a dynamic decoding algorithm to bolster faithfulness while preserving diversity. Specifically, the algorithm involves two parallel decoding steps, one with the context and one without. During the decoding, the KL divergence between two token distributions serves as a guiding signal, indicating the relevance of the source context. This signal is utilized to dynamically adjust the sampling temperature to improve source attribution when the source is relevant. In a parallel line of work, Choi et al. (2023) introduced knowledge-constrained decoding (KCD), which employed a token-level hallucination detection discriminator to identify contextual hallucinations and then guides the faithful generation process by reweighing the token distribution. In addition to modifying output distribution in place to enhance contextual attention, another line of work has explored a generic post-edit approach to enhance faithfulness. Gao et al. (2023a) adopted a research-and-revise workflow, where the research stage raises questions about various aspects of the model’s initial response and gathers evidence for each query, while the revision stage detects and revises any disagreements between the model’s response and the evidence. Similarly, Lei et al. (2023) first detected contextual hallucinations at both the sentence and entity levels and then incorporated the judgments to refine the generated response. Moreover, several studies have explored methods to overcome the softmax bottleneck, which constrains the expression of diversity and faithful representations. These approaches include employing a mixture of Softmax, which uses multiple hidden states to compute softmax multiple times and merge the resulting distributions (Yang et al., 2019) and incorporating pointer networks, which enables LLMs to copy the context words (Chang et al., 2023b), thereby reducing context hallucinations. Logical Consistency. Inspired by the human thinking process, chain-of-thought (Wei et al., 2022) has been introduced to encourage LLMs to decompose complex problems into explicitly intermediate steps, thereby enhancing the reliability of the reasoning process (Chu et al., 2023b). Despite effective, recent research (Lanham et al., 2023; Turpin et al., 2023) demonstrated that the intermediate rationales generated by LLMs do not faithfully capture their underlying behavior. A branch of research has been inspired to improve the consistency of intermediate rationales generated by LLMs, particularly in multi-step reasoning (Cobbe et al., 2021) and logical reasoning (Basso, 1993). To enhance the self-consistency in chain-of-thought, Wang et al. (2023f) employed a knowledge distillation framework. They first generate a consistent rationale using contrastive decoding (Li et al., 2022a) and then fine-tune the student model with a counterfactual reasoning objective, which effectively eliminates reasoning shortcuts (Branco et al., 2021) that derive answers without considering the rationale. Furthermore, by employing contrastive decoding directly, LLMs can reduce surface-level copying and prevent missed reasoning steps (O’Brien and Lewis, 2023). In addition, Li et al. (2024a) conducted a deep analysis of the causal relevance among the context, CoT, and answer during unfaithful reasoning. Analysis revealed that the unfaithfulness issue lies in the inconsistencies in the context information obtained by the CoT and the answer. To address this, they proposed inferential bridging, which takes the attribution method to recall contextual information as hints to enhance CoT reasoning and filter out noisy CoTs that have low semantic consistency and attribution scores to the context. Paul et al. (2024) decomposed the reasoning process into two modules: an inference module, which employs Direct Preference Optimization (Rafailov et al., 2023) to align the LLM towards preferring correct reasoning chains over counterfactual chains, and a reasoning module, which encourages the LLM to reason faithfully over the reasoning steps using a counterfactual and causal preference objective. Compared to natural language reasoning, logical reasoning demands rigorous logical calculation, whereas plain text often lacks precise logical structure, leading to unfaithful reasoning. To address this, Xu et al. (2024) introduced Symbolic CoT (SymbCoT), which incorporates symbolic expressions within CoT to describe intermediate reasoning steps. Specifically, SymbCoT translates the natural language context into a symbolic representation and then formulates a step-by-step plan to address the logical reasoning problem, followed by a verifier to check the translation and reasoning chain, thereby ensuring faithful logical reasoning. Discussion. Faithfulness Enhanced Decoding significantly advances the alignment of LLM outputs with provided contexts and enhances the internal consistency of the generated content. However, strategies such as context-aware decoding often lack adaptive mechanisms, limiting their effectiveness in scenarios that demand dynamic attention to context. Furthermore, many decoding strategies require the integration of additional models that do not focus on context, introducing significant computational overhead and reducing efficiency. ## 6. Hallucinations in Retrieval Augmented Generation Retrieval Augmented Generation (RAG) has emerged as a promising strategy to mitigate hallucinations and improve the factuality of LLM outputs (Lewis et al., 2020b; Izacard et al., 2023; Ram et al., 2023; Shi et al., 2023c). By incorporating large-scale external knowledge bases during inference, RAG equips LLMs with up-to-date knowledge, thus reducing the potential risk of hallucination due to the inherent knowledge boundaries of LLMs (Ren et al., 2023a). Despite being designed to mitigate LLM hallucinations, retrieval-augmented LLMs can still produce hallucinations (Barnett et al., 2024). Hallucinations in RAG present considerable complexities, manifesting as outputs that are either factually inaccurate or misleading. These hallucinations occur when the content generated by the LLM does not align with real-world facts, fails to accurately reflect the user’s query, or is not supported by the retrieved information. Such hallucinations can stem from two primary factors: retrieval failure (§ 6.1) and generation bottleneck (§ 6.2). Through a comprehensive analysis of the limitations present in current RAG systems, we aim to shed light on potential improvements for retrieval-augmented LLMs, paving the way for more reliable information retrieval systems. ### 6.1. Retrieval Failure The retrieval process is a crucial initial step in the RAG framework, tasked with retrieving the most relevant information for information-seeking queries. Consequently, failures in the retrieval stage can have serious downstream effects on the RAG pipeline, leading to hallucinations. These failures typically stem from three primary parts: the formulation of user queries, the reliability and scope of retrieval sources, and the effectiveness of the retriever. #### 6.1.1. User Queries User queries play a fundamental role in guiding the retrieval process with RAG systems. The specificity and clarity of these queries critically influence the effectiveness of retrieval outcomes. In this section, we discuss factors that may contribute to hallucinations from three perspectives: blind retrieval, misinterpretation of ambiguous queries, and the challenges in accurate retrieval of complex queries. Some examples are presented in Table 5 for a better understanding. Table 5. Examples of each category of User Queries that may lead to LLM hallucinations. Content marked in Red represents the hallucinatory output. Type Example Dialog Explanation Retrieval Intent Retrieved context: At high altitudes, water boils at lower temperatures than at sea level, such as 95°C at elevations above 2,000 meters. This altitude-induced reduction can affect cooking times and methods. Query: What is the boiling point of water at sea level? Response: The boiling point of water at sea level is 95°C. This query involves a basic scientific fact that should be available within the model’s parametric knowledge. However, the system inappropriately retrieves and uses altitude-specific information about boiling temperatures, leading to an incorrect response for sea-level conditions. Ambiguous Queries Retrieved context: The Hennessey Venom GT famously set a speed record by completing a mile on a racetrack in 32 seconds, highlighting its performance in motorsport events. Query: What’s the record for the fastest mile run on track? Response: The fastest mile on a track was completed in 32 seconds by the Hennessey Venom GT. The query is ambiguous because it does not specify whe- ther it refers to a human runner or a vehicle. The retrieval system erroneously pulls information related to car racing records instead of athletics. Complex Queries Retrieved context: Alexander Fleming, known for his ground- breaking work in discovering penicillin, significantly influenced modern antibiotics. Fleming’s discovery saved countless lives. Query: Which university did the scientist who discovered penicillin graduate from? Response: Alexander Fleming graduated from Oxford University. The retrieval system only manages to fetch information about Fleming’s professional achievements in the discovery of penicillin. However, the document does not provide informa- tion about his educational background, thus the model generates a hallucinatory answer. Retrieval Intent Decisions. Not all queries necessitate retrieval. Blind retrieval for queries that do not require external knowledge can counterproductively lead to misleading responses. As shown in Table 5, the query about ”the boiling point of water at sea level” pertains to a basic scientific fact that the model could address without external retrieval. However, the retrieval system was inappropriately activated, blindly retrieving inaccurate information and consequently leading to an undesirable response. Consequently, several studies (Mallen et al., 2023; Ding et al., 2024; Ni et al., 2024; Zhang et al., 2024b) have proposed to make a shift from passive retrieval to adaptive retrieval. In general, these strategies can be divided into two categories: heuristic-based and self-aware judgment. Heuristic-based methods employ heuristic rules to determine the necessity of retrieval. For instance, Mallen et al. (2023) observed a positive correlation between LLMs’ memorization capabilities and entity popularity and suggested triggering retrieval only when the entity popularity in the user query falls below a certain threshold. Similarly, Jeong et al. (2024) determined the timing of retrieval based on the query complexity, whereas Asai et al. (2023) considered whether the query is factual relevant. Self-aware judgment leverages the models’ intrinsic judgment to decide the necessity for information retrieval. Feng et al. (2023c), Ren et al. (2023b) and Wang et al. (2023c) directly prompted LLMs for retrieval decisions, recognizing that LLMs possess a certain level of awareness regarding their knowledge boundaries (Kadavath et al., 2022; Yin et al., 2023b). Moreover, Jiang et al. (2023b) introduced an active retrieval strategy that triggers retrieval only when the LLM generates low-probability tokens. Similarly, Su et al. (2024a) not only considered the uncertainty of each token but also its semantic contribution and impact on the subsequent context. More recently, Cheng et al. (2024) proposed four orthogonal criteria for determining the retrieval timing, which include intent-aware, knowledge-aware, time-sensitive-aware, and self-aware. Ambiguous Queries. Ambiguous user queries, containing omission, coreference, and ambiguity, significantly complicate the retrieval system’s ability to fetch precisely relevant information, thereby increasing the likelihood of generating undesirable responses. As shown in Table 5, due to the ambiguity of the query about ”the record for the fastest mile run on track”, the retrieval system erroneously retrieved information from automobile racing events, which led the model to generate a response suited for vehicles instead of athletes. A prevalent mitigation strategy is query rewriting, where queries are refined and decontextualized to better match relevant documents. Wang et al. (2023h) and Jagerman et al. (2023) have explored prompting approaches where the LLM is prompted to generate a pseudo-document or rationale based on the original query, which is then used for further retrieval. Additionally, Ma et al. (2023) introduced a trainable rewriter which is trained using the feedback from the LLM via reinforcement learning. Mao et al. (2024) employed the feedback signals from the reranker to train the rewrite model, thus eliminating the reliance on annotated data. However, the challenges deepen in conversational search, which encounters a more complex issue of context-dependent query understanding with the lengthy conversational history. Addressing this, Yoon et al. (2024) proposed a similar framework for optimizing the LLM to generate retriever-preferred query rewrites. This operated by generating a variety of queries and then using the preference of the rank of retrieved passage to optimize the query rewriting model. Complex Queries. Complex user queries, characterized by requiring intensive reasoning (Su et al., 2024b) or encompassing multiple aspects (Shao et al., 2024; Wang et al., 2024b), pose significant challenges to the retrieval system. Such queries require advanced understanding and decomposition capabilities, which may exceed the current capabilities of the current retrieval methods based on keyword or semantic matching, often leading to partial or incorrect retrievals. For example, as shown in Table 5, due to the multi-step nature of the query about ”Which university did the scientist who discovered penicillin graduate from?”, direct retrieval often leads to incomplete results, thereby resulting in hallucinatory responses. A common approach involves query decomposition, where the complex query is decomposed into sub-queries to facilitate more accurate information retrieval. For instance, Wang et al. (2024b) implemented a sub-aspect explorer that utilizes the extensive world knowledge embedded LLMs to identify potential sub-aspects of user queries, thereby providing explicit insights into the user’s underlying intents. Similarly, Shao et al. (2024) concentrated on the demanding task of expository writing, aiming at retrieving comprehensive information to compose Wikipedia-like articles from scratch on a specific topic. This approach involves decomposing the topic into various perspectives and simulating multi-turn conversations with LLMs, each personified with different perspectives for question asking. Additionally, Cao et al. (2023b) and Chu et al. (2024) explored knowledge-intensive complex reasoning and employed a divide-and-conquer strategy. This strategy begins with decomposing complex questions into question trees, where at each node, the LLM retrieves and aggregates answers from diverse knowledge sources. #### 6.1.2. Retrieval Sources The reliability and scope of retrieval sources are crucial determinants of the efficacy of RAG systems. Effective retrieval depends not only on the clarity of the user queries but also on the quality and comprehensiveness of the sources from which information is retrieved. When these sources contain factually incorrect or outdated information, the risk of retrieval failures increases significantly, potentially leading to the generation of incorrect or misleading information. As the landscape of content creation evolves with the rapid advancement of Artificial Intelligence Generated Content (AIGC) (Cao et al., 2023a), an increasing volume of LLM-generated content is permeating the internet, subsequently becoming integrated into retrieval sources (Chen and Shu, 2023). This integration is reshaping the dynamics of information retrieval, as evidenced by recent empirical studies (Dai et al., 2023; Xu et al., 2023a) suggesting that modern retrieval models tend to favor LLM-generated content over human-authored content. Recent research (Chen et al., 2024) has explored the implications of progressively integrating LLM-generated content into RAG systems. The findings indicate that, without appropriate intervention, human-generated content may progressively lose its influence within RAG systems. Additionally, Tan et al. (2024a) investigated the performance of RAG systems when incorporating LLM-generated into retrieved contexts, revealing a significant bias favoring generated contexts. This bias stems from the high similarity between generated context and questions, as well as the semantic incompleteness of retrieved contexts. More seriously, the propensity of LLMs to produce factually inaccurate hallucinations exacerbates the reliability issues of retrieval sources. As LLM-generated content often contains factual errors, its integration into retrieval sources can mislead retrieval systems, further diminishing the accuracy and reliability of the information retrieved. To combat these biases, several approaches have been explored. Inspired by common practice in pre-training data processing (Black et al., 2022), Asai et al. (2024) proposed a scenario that incorporates a quality filter designed to ensure the high quality of the retrieval datastore. Additionally, Pan et al. (2024) proposed Credibility-aware Generation (CAG), which equips LLMs with the ability to discern and handle information based on its credibility. This approach assigns different credibility levels to information, considering its relevance, temporal context, and the trustworthiness of its source, thus effectively reducing the impact of flawed information in RAG systems. #### 6.1.3. Retriever When the user query is explicit and the retrieval source is reliable, the effectiveness of the retrieval process depends crucially on the performance of the retriever. In such scenarios, the retriever’s effectiveness is significantly compromised by improper chunking and embedding practices. Chunking. Given the extensive nature of retrieval sources, which often encompass lengthy documents like web pages, it poses significant challenges for LLMs with limited context length. Thus, chunking emerges as an indispensable step in RAG, which involves segmenting these voluminous documents into smaller, more manageable chunks to provide precise and relevant evidence for LLMs. According to actual needs, the chunking granularity ranges from documents to paragraphs, even sentences. However, inappropriate retrieval granularity can compromise the semantic integrity and affect the relevance of retrieved information (Nair et al., 2023), thereby affecting the performance of LLMs. Fixed-size chunking, which typically breaks down the documents into chunks of a specified length such as 100-word paragraphs, serves as the most crude and prevalent strategy of chunking, which is widely used in RAG systems (Guu et al., 2020; Lewis et al., 2020b; Borgeaud et al., 2022). Considering fixed-size chunking falls short in capture structure and dependency of lengthy documents, Sarthi et al. (2024) proposed RAPTOR, an indexing and retrieval system. By recursively embedding, clustering, and summarizing chunks of text, RAPTOR constructs a tree to capture both high-level and low-level details. When retrieval, RAPTOR enables LLMs to integrate information from different levels of abstraction, providing a more comprehensive context for user queries. Instead of chunking text with a fixed chunk size, semantic chunking adaptively identifies breakpoints between sentences through embedding similarity, thereby preserving semantic continuity (Kamradt, 2024). Furthermore, Chen et al. (2023c) pointed out the limitations of the existing retrieval granularity. On the one hand, while a coarser retrieval with a longer context can theoretically provide a more comprehensive context, it often includes extraneous details that could potentially distract LLMs. On the other hand, a fine-grain level can provide more precise and relevant information, it has limitations such as not being self-contained and lacking necessary contextual information. To address these shortcomings, Chen et al. (2023c) introduced a novel retrieval granularity, proposition, which is defined as atomic expressions within the text, each encapsulating a distinct factoid and presented in a concise self-contained natural language format. Embedding. Once the retrieval text is chunked, text chunks are subsequently transformed into vector representation via an embedding model. Such a representation scheme is supported by the well-known data structure of vector database (Jing et al., 2024), which systematically organizes data as key-value pairs for efficient text retrieval. In this manner, the relevance score can be computed according to the similarity function between the text representation and query representation. However, a sub-optimal embedding model may compromise performance, which affects the similarity and matching of chunks to user queries, potentially misleading LLMs. Typically, a standard embedding model (Izacard et al., 2022; Gao et al., 2021b; Karpukhin et al., 2020b; Zhao et al., 2024b) learns the query and text representations with encoder-based architecture (e.g. BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019)) via contrastive learning (van den Oord et al., 2018), where the loss is constructed by contrasting a positive pair of query-document against a set of random negative pairs. However, these embeddings showcase their limitations when applied to new domains, such as medical and financial applications (Thakur et al., 2021; Muennighoff et al., 2023). In these cases, recent studies (Shi et al., 2023c; Xiao et al., 2023; Ovadia et al., 2023; de Luis Balaguer et al., 2024) propose to fine-tune the embedding models on domain-specific data to enhance retrieval relevance. For example, REPLUG (Shi et al., 2023c) utilizes language modeling scores of the answers as a proxy signal to train the dense retriever. More recently, Muennighoff et al. (2024) have introduced generative representational instruction tuning where a single LLM is trained to handle both generative and embedding tasks, which largely reduces inference latency in RAG by caching representations. Despite these advancements, the field faces challenges, particularly with the fine-tuning of high-performing yet inaccessible embedding models, such as OpenAI’s text-embedding-ada-002. Addressing this gap, Zhang et al. (2024c) introduced a novel approach for fine-tuning a black-box embedding model by augmenting it with a trainable embedding model which significantly enhances the performance of the black-box embeddings. ### 6.2. Generation Bottleneck After the retrieval process, the generation stage emerges as a pivotal point, responsible for generating content that faithfully reflects the retrieved information. However, this stage can encounter significant bottlenecks that may lead to hallucinations. We summarize two key capabilities of LLMs that are closely related to these bottlenecks: contextual awareness and contextual alignment. Each plays an important role in ensuring the reliability and credibility of the RAG system. #### 6.2.1. Contextual Awareness Contextual awareness involves understanding and effectively utilizing contextual information retrieved. This section discusses the key factors that impact the LLM’s ability to maintain contextual awareness, which can be categorized into three main parts: (1) the presence of noisy retrieval in context, (2) context conflicts, and (3) insufficient utilization of context information. Noisy Context. As emphasized in § 6.1, the failure in the retrieval process may inevitably introduce irrelevant information, which will propagate into the generation stage. When the generator is not robust enough to these irrelevant retrievals, it will mislead the generator and even introduce hallucinations (Cuconasu et al., 2024). Yoran et al. (2023) conducted a comprehensive analysis on the robustness of current retrieval-augmented LLMs, revealing a significant decrease in performance with random retrieval. While using an NLI model to filter out irrelevant passages is effective, this method comes with the trade-off of inadvertently discarding some relevant passages. A more effective solution is to train LLMs to ignore irrelevant contexts by incorporating irrelevant contexts in training data. Similarly, Yu et al. (2023d) introduced Chain-of-Note, which enables LLMs to first generate reading notes for retrieved contexts and subsequently formulate the final answer. In this way, LLMs can not only filter irrelevant retrieval to improve noise robustness but also respond with unknown when retrieval is insufficient to answer user queries. In addition to improving LLM robustness by learning to ignore irrelevant content in the context, several studies (Li, 2023; Jiang et al., 2023a; Xu et al., 2023b; Wang et al., 2023a) propose to compress the context to filter out irrelevant information. Specifically, Li (2023) and Jiang et al. (2023a) made use of small language models to compute self-information and perplexity for prompt compression, finding the most informative content. Similarly, Wang et al. (2023a) proposed to filter out irrelevant content and leave precisely supporting content based on lexical and information-theoretic approaches. Besides, efforts have been also made to employ summarization models as compressors. Xu et al. (2023b) presented both extractive and abstractive compressors, which are trained to improve LLMs’ performance while keeping the prompt concise. Liu et al. (2023e) involved summarization compression and semantic compression, where the former achieves compression by summarizing while the latter removes tokens with a lower impact on the semantic. Context Conflict. Retrieval-augmented LLMs generate answers through the combined effect of parametric knowledge and contextual knowledge. As discussed in § 3.3.2, LLMs may sometimes exhibit over-confidence, which can bring new challenges to the faithfulness of RAG systems when facing knowledge conflicts. Knowledge conflicts in RAG are situations where contextual knowledge contradicts LLMs’ parametric knowledge. Longpre et al. (2021) first investigated knowledge conflicts in open-domain question answering, where conflicts are automatically created by replacing all spans of the gold answer in the retrieval context with a substituted entity. Findings demonstrate that generative QA reader models (e.g. T5) tend to trust parametric memory over contextual information. By further training the retriever to learn to trust the contextual evidence with augmented training examples by entity substitution, the issue of over-reliance on parametric knowledge is mitigated. Similar findings are also reported by Li et al. (2023g) who demonstrated that fine-tuning LLMs on counterfactual contexts can effectively improve the controllability of LLMs when dealing with contradicts contexts. Also building upon counterfactual data augmentation, Neeman et al. (2023) trained models to predict two disentangled answers, one based on contextual knowledge and the other leveraging parametric knowledge to address knowledge conflicts. Besides, Zhou et al. (2023c) introduced two effective prompting-based strategies, namely opinion-based prompts and counterfactual demonstrations. Opinion-based prompts transform the context to narrators’ statements, soliciting the narrators’ opinions, whereas counterfactual demonstrations employ counterfactual instances to improve faithfulness in situations of knowledge conflict. While Longpre et al. (2021) and Li et al. (2023g) concentrated their research on the context of a limited single evidence setting, Chen et al. (2022b) further expanded this study to consider a more realistic scenario in which models consider multiple evidence passages and find models rely almost exclusively on contextual evidence. Considering previous studies (Longpre et al., 2021; Li et al., 2023g) mostly focused on smaller models, Xie et al. (2023) raised doubts about the applicability of their conclusions in the era of LLMs. Such heuristic entity-level substitution may lead to incoherent counter-memory, thereby making it trivial for LLMs to overlook the construct knowledge conflicts. By directly eliciting LLMs to generate a coherent counter-memory that factually conflicts with the parametric memory, LLMs exhibit their high receptivity to external evidence. Context Utilization. Despite successfully retrieving evidence relevant to factoid queries, LLMs can encounter a significant performance degradation due to insufficient utilization of the context, especially for information located in the middle of the long context window, a notable issue known as the lost-in-the-middle phenomenon (Liu et al., 2023f). Beyond factoid QA, recent studies have further demonstrated such a middle-curse also holds in abstractive summarization (Ravaut et al., 2024), long-form QA (Chen et al., 2023d) and passage ranking (Tang et al., 2023). One potential explanation lies in the use of rotary positional embedding (RoPE) (Su et al., 2021), which is widely used in open-source LLMs, due to its excellent performance in length extrapolation (Zhao et al., 2023a). As a representative relative position embedding, RoPE features a long-term decay property, which inherently biases the LLM to give precedence to current or proximate tokens, thereby diminishing its attention on those that are more distant. Another contributing factor is that the most salient information often resides at the beginning or the end of pre-training data, a characteristic commonly observed in news reports (Ravaut et al., 2024). Such an issue brings forth challenges in retrieval-augmented LLMs, as retrieval-augmented LLMs are typically designed with extensive lengths to accommodate more retrieval documents. To mitigate this crucial issue, He et al. (2023a) introduced several tasks specially designed for information seeking to enhance the capability of information utilization by explicitly repeating the question and extracting the index of supporting documents before generating answers. Furthermore, Zhang et al. (2024a) introduced Multi-scale Positional Encoding (Ms-PoE), which mitigates the long-term decay effect characteristic of RoPE by rescaling position indices. Ms-PoE provides a plug-and-play solution to enhance the ability of LLMs to effectively capture information in the middle of the context without the need for additional fine-tuning. Besides, Ravaut et al. (2024) proposed hierarchical and incremental summarization, which effectively preserves the salient information and compresses the length of context to avoid the middle-curse. #### 6.2.2. Contextual Alignment Contextual alignment ensures that LLM outputs faithfully align with relevant context. This section outlines the primary components of contextual alignment, which include: (1) source attribution and (2) faithful decoding. Source Attribution. Source attribution (Huang et al., 2024b) in retrieval-augmented LLMs refers to the process by which the model identifies and utilizes the origins of information within its generation process. This component is crucial for ensuring that the outputs of RAG systems are not only relevant but also verifiable and grounded in credible sources. To achieve source attribution in RAG systems, recent studies have been explored, which can be categorized into three lines based on the type of attribution. (1) Plan-then-Generate: Fierro et al. (2024) introduced the blueprint model for attribution, which conceptualizes text plans as a series of questions that serve as blueprints for generation process, dictating both the content and the sequence of the output. Compared with abstractive questions, Huang et al. (2024a) enabled the model to first ground to extractive evidence spans, which guides the subsequent generation process. Leveraging either abstract questions or extractive spans as planning facilitates a built-in attribution mechanism, as they provide a natural link between retrieved information and the subsequent generation. Similarly, Slobodkin et al. (2024) broke down the conventional end-to-end generation process into three intuitive stages: content selection, sentence planning, and sentence fusion. By initially identifying relevant source segments and subsequently conditioning the generation process on them, the selected segments naturally serve as attributions. (2) Generate-then-Reflect: Asai et al. (2023) proposed training the LLM to generate text with reflection tokens. These reflection tokens empower the LLM to decide whether to retrieve, assess the relevance of the retrieved document, and critique its own generation to ensure attributability. By critiquing its generation. Furthermore, Ye et al. (2023) introduced AGREE, designed to facilitate self-grounding in LLMs. AGREE trains LLMs to generate well-grounded claims with citations and identify claims that lack verification. An iterative retrieval process is then employed to actively seek additional information for these unsupported statements. (3) Self-Attribution: In addition to leveraging external supervised signals for attribution, Qi et al. (2024) proposed a self-attribution mechanism that utilizes model-internal signals. It operates by first identifying context-sensitive answer tokens, which are then paired with retrieved documents that contributed to the model generation via saliency methods. Faithful Decoding. Despite significant optimizations in the RAG pipeline that facilitate the incorporation of highly relevant content into the model’s context, current LLMs still cannot guarantee faithful generation. The unfaithful utilization of relevant context by LLMs undermines the reliability of their outputs, even when the sources of information are verifiably accurate. Wu et al. (2024b) analyzed the model’s knowledge preference when internal knowledge conflicts with contextual information and observed the tug-of-war between the LLM’s internal prior and external evidence. To tackle this issue, recent research (Shi et al., 2023a; Wu et al., 2024a) has focused on faithful decoding within RAG systems, aiming to improve the models’ ability to generate content that faithfully aligns with contextual information. Shi et al. (2023a) presented context-aware decoding, which modifies the model’s original output probability distribution into the pointwise mutual information (PMI) formulation. The strategy operates by amplifying the difference between the output probabilities when a model is used with and without context, thereby enhancing the faithfulness of LLMs to the provided context. Li et al. (2024b) adopted a semi-parametric language modeling approach (Khandelwal et al., 2020) which facilitates the integration of contextual spans of arbitrary length into LM generations. The generation is then verified via speculative decoding, further ensuring model faithfulness. More recently, Wu et al. (2024a) proposed faithfulness-oriented decoding, which leverages a lightweight faithfulness detector to monitor the beam-search process. The detector leverages fine-grained decoding dynamics including sequence likelihood, uncertainty quantification, context influence, and semantic alignment to synchronously detect unfaithful sentences. When an unfaithful generation is detected, it triggers the backtrack operation and selects the beam with the more faithful score, thus ensuring greater faithfulness to the retrieval sources. ## 7. Future discussion As the field of research on hallucinations in LLMs continues to evolve, our focus shifts towards the next horizon of inquiry. We explore prospective areas of study, notably the phenomenon of hallucinations in vision-language models (§ 7.1) and the challenge of delineating and understanding knowledge boundaries within LLMs (§ 7.2). ### 7.1. Hallucination in Large Vision-Language Models Enabling the visual perception ability, along with exceptional language understanding and generation capabilities, Large Vision-Language Models (LVLMs) have exhibited remarkable vision-language capabilities (Zhu et al., 2023a; Liu et al., 2023d; Yu et al., 2021; Huang et al., 2023b; Maaz et al., 2023; Chen et al., 2023b; Yu et al., 2023a; Zellers et al., 2019). Unlike previous pre-trained multi-modal models that gain limited vision-language abilities from large-scale visual-language pre-training datasets (Wang et al., 2022; Li et al., 2023d; Luo et al., 2020; Zhong et al., 2023), LVLMs exploit advanced LLMs to unleash the power of interacting with humans and the environment. The consequent diverse applications of LVLMs also bring new challenges to maintaining the reliability of such systems. Recent studies have revealed that current LVLMs are suffering from multi-modal hallucinations, where models provide responses misaligned with the corresponding visual information (Liu et al., 2024; Tong et al., 2024; Guan et al., 2023). Such multi-modal hallucinations could cause unexpected behaviors when applying LVLMs to real-world scenarios, which therefore had to be further investigated and mitigated. Li et al. (2023c) and Lovenia et al. (2023) took the first step towards evaluating the object hallucinations in the LVLMs. Evaluations and experiments reveal that current LVLMs are prone to generate inconsistent responses with respect to the associated image, including non-existent objects, wrong object types, and attributes, incorrect semantic relationships, etc. (Wang et al., 2023g; Zhai et al., 2023). Furthermore, Liu et al. (2023g), Zong et al. (2023) and Liu et al. (2023b) show that LVLMs can be easily fooled and experience a severe performance drop due to their over-reliance on the strong language prior, as well as its inferior ability to defend against inappropriate user inputs (Jeong, 2023; Han et al., 2024). Jiang et al. (2024), Wang et al. (2023g) and Jing et al. (2023) took a step forward to holistically evaluate multi-modal hallucination. What’s more, when presented with multiple images, LVLMs sometimes mix or miss parts of the visual context, as well as fail to understand temporal or logical connections between them, which might hinder their usage in many scenarios, yet properly identifying the reason for such disorders and tackling them still requires continued efforts. Despite the witnessed perception errors, LVLMs can generate flawed logical reasoning results even when correctly recognizing all visual elements, which remains further investigation. Efforts have been made towards building a more robust large vision-language model. Gunjal et al. (2023), Lu et al. (2023), Wang et al. (2024a), and Liu et al. (2023g) proposed to further finetune the model for producing more truthful and helpful responses. Another line of work chooses to post-hoc rectify the generated inconsistent content, such as (Zhou et al., 2023a), and (Yin et al., 2023a), which introduced expert models. To free from the external tools, Leng et al. (2023), Huang et al. (2023c), and Zhao et al. (2024a) tried to fully utilize the LVLM itself to alleviate hallucinations. Though proved to be effective, those methods usually require additional data annotations, visual experts, training phases, and computational costs, which prevent LVLMs from effectively scaling and generalizing to various fields. Thus, more universal approaches are expected to build a more reliable system, such as faithful and large-scale visual-text pre-training and alignment methods. ### 7.2. Understanding Knowledge Boundary in LLMs Despite the impressive capacity to capture factual knowledge from extensive data, LLMs still face challenges in recognizing their own knowledge boundaries. This shortfall leads to the occurrence of hallucinations, where LLMs confidently produce falsehoods without an awareness of their own knowledge limits (Pacchiardi et al., 2023; Ren et al., 2023b; Zhao et al., 2023c). Numerous studies delve into probing knowledge boundaries of LLMs, utilizing strategies such as evaluating the probability of a correct response in a multiple-choice setting (Kadavath et al., 2022), or quantifying the model’s output uncertainty by evaluating the similarity among sets of sentences with uncertain meanings. Furthermore, a line of work (Moschella et al., 2022; Burns et al., 2022; Li et al., 2023f; Azaria and Mitchell, 2023) has revealed that LLMs contain latent structures within their activation space that relate to beliefs about truthfulness. Recent research (Slobodkin et al., 2023) also found substantial evidence for LLMs’ ability to encode the unanswerability of questions, despite the fact that these models exhibit overconfidence and produce hallucinations when presented with unanswerable questions. Nonetheless, Levinstein and Herrmann (2023) have employed empirical and conceptual tools to probe whether or not LLMs have beliefs. Their empirical results suggest that current lie-detector methods for LLMs are not yet fully reliable, and the probing methods proposed by Burns et al. (2022) and Azaria and Mitchell (2023) do not adequately generalize. Consequently, whether we can effectively probe LLMs’ internal beliefs is ongoing, requiring further research. ## 8. Conclusion In this comprehensive survey, we have undertaken an in-depth examination of hallucinations within large language models, delving into the intricacies of their underlying causes, pioneering detection methodologies as well as related benchmarks, and effective mitigation strategies. Although significant strides have been taken, the conundrum of hallucination in LLMs remains a compelling and ongoing concern that demands continuous investigation. Moreover, we envision this survey as a guiding beacon for researchers dedicated to advancing robust information retrieval systems and trustworthy artificial intelligence. By navigating the complex landscape of hallucinations, we hope to empower these dedicated individuals with invaluable insights that drive the evolution of AI technologies toward greater reliability and safety. ## References - (1) - Abbas et al. (2023) Amro Abbas, Kushal Tirumala, Dániel Simig, Surya Ganguli, and Ari S Morcos. 2023. SemDeDup: Data-efficient learning at web-scale through semantic deduplication. ArXiv preprint abs/2303.09540 (2023). https://arxiv.org/abs/2303.09540 - Adlakha et al. (2023) Vaibhav Adlakha, Parishad BehnamGhader, Xing Han Lu, Nicholas Meade, and Siva Reddy. 2023. Evaluating correctness and faithfulness of instruction-following models for question answering. ArXiv preprint abs/2307.16877 (2023). https://arxiv.org/abs/2307.16877 - Agrawal et al. (2023) Ayush Agrawal, Lester Mackey, and Adam Tauman Kalai. 2023. Do Language Models Know When They’re Hallucinating References? ArXiv preprint abs/2305.18248 (2023). https://arxiv.org/abs/2305.18248 - AI (2023) Perplexity AI. 2023. Perplexity AI. https://www.perplexity.ai/ - Aksitov et al. (2023) Renat Aksitov, Chung-Ching Chang, David Reitter, Siamak Shakeri, and Yun-Hsuan Sung. 2023. Characterizing Attribution and Fluency Tradeoffs for Retrieval-Augmented Large Language Models. ArXiv preprint abs/2302.05578 (2023). https://arxiv.org/abs/2302.05578 - AlKhamissi et al. (2022) Badr AlKhamissi, Millicent Li, Asli Celikyilmaz, Mona T. Diab, and Marjan Ghazvininejad. 2022. A Review on Language Models as Knowledge Bases. CoRR abs/2204.06031 (2022). https://doi.org/10.48550/ARXIV.2204.06031 arXiv:2204.06031 - Anil et al. (2023) Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Slav Petrov, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy P. Lillicrap, Angeliki Lazaridou, Orhan Firat, James Molloy, Michael Isard, Paul Ronald Barham, Tom Hennigan, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, Ryan Doherty, Eli Collins, Clemens Meyer, Eliza Rutherford, Erica Moreira, Kareem Ayoub, Megha Goel, George Tucker, Enrique Piqueras, Maxim Krikun, Iain Barr, Nikolay Savinov, Ivo Danihelka, Becca Roelofs, Anaïs White, Anders Andreassen, Tamara von Glehn, Lakshman Yagati, Mehran Kazemi, Lucas Gonzalez, Misha Khalman, Jakub Sygnowski, and et al. 2023. Gemini: A Family of Highly Capable Multimodal Models. CoRR abs/2312.11805 (2023). https://doi.org/10.48550/ARXIV.2312.11805 arXiv:2312.11805 - Anthropic (2023) Anthropic. 2023. Claude. https://claude.ai/ - Antropic (2024) Antropic. 2024. Claude 3 haiku: our fastest model yet. 2024. https://www.anthropic.com/news/claude-3-haiku - ArXiv (2023) ArXiv. 2023. arxiv dataset. https://www.kaggle.com/datasets/Cornell-University/arxiv/versions/134 - Asai et al. (2023) Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2023. Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection. CoRR abs/2310.11511 (2023). https://doi.org/10.48550/ARXIV.2310.11511 arXiv:2310.11511 - Asai et al. (2024) Akari Asai, Zexuan Zhong, Danqi Chen, Pang Wei Koh, Luke Zettlemoyer, Hannaneh Hajishirzi, and Wen-tau Yih. 2024. Reliable, Adaptable, and Attributable Language Models with Retrieval. CoRR abs/2403.03187 (2024). https://doi.org/10.48550/ARXIV.2403.03187 arXiv:2403.03187 - Azaria and Mitchell (2023) Amos Azaria and Tom M. Mitchell. 2023. The Internal State of an LLM Knows When its Lying. ArXiv preprint abs/2304.13734 (2023). https://arxiv.org/abs/2304.13734 - Baek et al. (2023) Jinheon Baek, Alham Fikri Aji, and Amir Saffari. 2023. Knowledge-Augmented Language Model Prompting for Zero-Shot Knowledge Graph Question Answering. ArXiv preprint abs/2306.04136 (2023). https://arxiv.org/abs/2306.04136 - Bang et al. (2023) Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. 2023. A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity. ArXiv preprint abs/2302.04023 (2023). https://arxiv.org/abs/2302.04023 - Barnett et al. (2024) Scott Barnett, Stefanus Kurniawan, Srikanth Thudumu, Zach Brannelly, and Mohamed Abdelrazek. 2024. Seven Failure Points When Engineering a Retrieval Augmented Generation System. CoRR abs/2401.05856 (2024). https://doi.org/10.48550/ARXIV.2401.05856 arXiv:2401.05856 - Barrantes et al. (2020) Mario Barrantes, Benedikt Herudek, and Richard Wang. 2020. Adversarial nli for factual correctness in text summarisation models. ArXiv preprint abs/2005.11739 (2020). https://arxiv.org/abs/2005.11739 - Basso (1993) Pierre Basso. 1993. Conditional Causal Logic: A Formal Theory of the Meaning Generating Processes in a Cognitive System. In Proceedings of the 13th International Joint Conference on Artificial Intelligence. Chambéry, France, August 28 - September 3, 1993, Ruzena Bajcsy (Ed.). Morgan Kaufmann, 845–851. http://ijcai.org/Proceedings/93-2/Papers/002.pdf - Beltagy et al. (2020) Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The Long-Document Transformer. ArXiv preprint abs/2004.05150 (2020). https://arxiv.org/abs/2004.05150 - Bender et al. (2021) Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?. In FAccT ’21: 2021 ACM Conference on Fairness, Accountability, and Transparency, Virtual Event / Toronto, Canada, March 3-10, 2021, Madeleine Clare Elish, William Isaac, and Richard S. Zemel (Eds.). ACM, 610–623. https://doi.org/10.1145/3442188.3445922 - Bengio et al. (2015) Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. 2015. Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett (Eds.). 1171–1179. https://proceedings.neurips.cc/paper/2015/hash/e995f98d56967d946471af29d7bf99f1-Abstract.html - Berglund et al. (2023) Lukas Berglund, Meg Tong, Max Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, and Owain Evans. 2023. The Reversal Curse: LLMs trained on” A is B” fail to learn” B is A”. ArXiv preprint abs/2309.12288 (2023). https://arxiv.org/abs/2309.12288 - Black et al. (2022) Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. 2022. GPT-NeoX-20B: An Open-Source Autoregressive Language Model. CoRR abs/2204.06745 (2022). https://doi.org/10.48550/ARXIV.2204.06745 arXiv:2204.06745 - Borgeaud et al. (2022) Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack W. Rae, Erich Elsen, and Laurent Sifre. 2022. Improving Language Models by Retrieving from Trillions of Tokens. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA (Proceedings of Machine Learning Research, Vol. 162), Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (Eds.). PMLR, 2206–2240. https://proceedings.mlr.press/v162/borgeaud22a.html - Bowman et al. (2022) Samuel R Bowman, Jeeyoon Hyun, Ethan Perez, Edwin Chen, Craig Pettit, Scott Heiner, Kamile Lukosuite, Amanda Askell, Andy Jones, Anna Chen, et al. 2022. Measuring progress on scalable oversight for large language models. ArXiv preprint abs/2211.03540 (2022). https://arxiv.org/abs/2211.03540 - Bradley and Terry (1952) Ralph Allan Bradley and Milton E Terry. 1952. Rank analysis of incomplete block designs: I. The method of paired comparisons. Biometrika 39, 3/4 (1952), 324–345. https://www.jstor.org/stable/2334029 - Branco et al. (2021) Ruben Branco, António Branco, João António Rodrigues, and João Ricardo Silva. 2021. Shortcutted Commonsense: Data Spuriousness in Deep Learning of Commonsense Reasoning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 1504–1521. https://doi.org/10.18653/v1/2021.emnlp-main.113 - Broder (1997) Andrei Z Broder. 1997. On the resemblance and containment of documents. In Proceedings. Compression and Complexity of SEQUENCES 1997 (Cat. No. 97TB100171). IEEE, 21–29. - Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (Eds.). https://proceedings.neurips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html - Bubeck et al. (2023) Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott M. Lundberg, Harsha Nori, Hamid Palangi, Marco Túlio Ribeiro, and Yi Zhang. 2023. Sparks of Artificial General Intelligence: Early experiments with GPT-4. ArXiv preprint abs/2303.12712 (2023). https://arxiv.org/abs/2303.12712 - Burns et al. (2022) Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. 2022. Discovering latent knowledge in language models without supervision. ArXiv preprint abs/2212.03827 (2022). https://arxiv.org/abs/2212.03827 - Cao et al. (2023b) Shulin Cao, Jiajie Zhang, Jiaxin Shi, Xin Lv, Zijun Yao, Qi Tian, Lei Hou, and Juanzi Li. 2023b. Probabilistic Tree-of-thought Reasoning for Answering Knowledge-intensive Complex Questions. In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, 12541–12560. https://doi.org/10.18653/V1/2023.FINDINGS-EMNLP.835 - Cao et al. (2023a) Yihan Cao, Siyu Li, Yixin Liu, Zhiling Yan, Yutong Dai, Philip S. Yu, and Lichao Sun. 2023a. A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPT. CoRR abs/2303.04226 (2023). https://doi.org/10.48550/ARXIV.2303.04226 arXiv:2303.04226 - Carlini et al. (2022) Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2022. Quantifying memorization across neural language models. ArXiv preprint abs/2202.07646 (2022). https://arxiv.org/abs/2202.07646 - Carlini et al. (2021) Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. 2021. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21). 2633–2650. - Chang et al. (2023a) Chung-Ching Chang, David Reitter, Renat Aksitov, and Yun-Hsuan Sung. 2023a. KL-Divergence Guided Temperature Sampling. ArXiv preprint abs/2306.01286 (2023). https://arxiv.org/abs/2306.01286 - Chang et al. (2023b) Haw-Shiuan Chang, Zonghai Yao, Alolika Gon, Hong Yu, and Andrew McCallum. 2023b. Revisiting the Architectures like Pointer Networks to Efficiently Improve the Next Word Distribution, Summarization Factuality, and Beyond. In Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023, Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (Eds.). Association for Computational Linguistics, 12707–12730. https://doi.org/10.18653/V1/2023.FINDINGS-ACL.805 - Chang and McCallum (2022) Haw-Shiuan Chang and Andrew McCallum. 2022. Softmax Bottleneck Makes Language Models Unable to Represent Multi-mode Word Distributions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Dublin, Ireland, 8048–8073. https://doi.org/10.18653/v1/2022.acl-long.554 - Chen and Shu (2023) Canyu Chen and Kai Shu. 2023. Combating Misinformation in the Age of LLMs: Opportunities and Challenges. CoRR abs/2311.05656 (2023). https://doi.org/10.48550/ARXIV.2311.05656 arXiv:2311.05656 - Chen et al. (2022b) Hung-Ting Chen, Michael J. Q. Zhang, and Eunsol Choi. 2022b. Rich Knowledge Sources Bring Complex Knowledge Conflicts: Recalibrating Models to Reflect Conflicting Evidence. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (Eds.). Association for Computational Linguistics, 2292–2307. https://doi.org/10.18653/V1/2022.EMNLP-MAIN.146 - Chen et al. (2023d) Hung-Ting Chen, Fangyuan Xu, Shane A Arora, and Eunsol Choi. 2023d. Understanding Retrieval Augmentation for Long-Form Question Answering. ArXiv preprint abs/2310.12150 (2023). https://arxiv.org/abs/2310.12150 - Chen et al. (2023e) Shiqi Chen, Yiran Zhao, Jinghan Zhang, I-Chun Chern, Siyang Gao, Pengfei Liu, and Junxian He. 2023e. FELM: Benchmarking Factuality Evaluation of Large Language Models. ArXiv preprint abs/2310.00741. https://arxiv.org/abs/2310.00741 - Chen et al. (2023c) Tong Chen, Hongwei Wang, Sihao Chen, Wenhao Yu, Kaixin Ma, Xinran Zhao, Hongming Zhang, and Dong Yu. 2023c. Dense X Retrieval: What Retrieval Granularity Should We Use? CoRR abs/2312.06648 (2023). https://doi.org/10.48550/ARXIV.2312.06648 arXiv:2312.06648 - Chen et al. (2024) Xiaoyang Chen, Ben He, Hongyu Lin, Xianpei Han, Tianshu Wang, Boxi Cao, Le Sun, and Yingfei Sun. 2024. Spiral of Silence: How is Large Language Model Killing Information Retrieval? – A Case Study on Open Domain Question Answering. arXiv:2404.10496 [cs.IR] https://arxiv.org/abs/2404.10496 - Chen et al. (2022a) Xiuying Chen, Mingzhe Li, Xin Gao, and Xiangliang Zhang. 2022a. Towards Improving Faithfulness in Abstractive Summarization. In NeurIPS. http://papers.nips.cc/paper_files/paper/2022/hash/9b6d7202750e8e32cd5270eb7fc131f7-Abstract-Conference.html - Chen et al. (2023a) Yijie Chen, Yijin Liu, Fandong Meng, Yufeng Chen, Jinan Xu, and Jie Zhou. 2023a. Improving Translation Faithfulness of Large Language Models via Augmenting Instructions. ArXiv preprint abs/2308.12674 (2023). https://arxiv.org/abs/2308.12674 - Chen et al. (2023b) Yangyi Chen, Karan Sikka, Michael Cogswell, Heng Ji, and Ajay Divakaran. 2023b. Measuring and Improving Chain-of-Thought Reasoning in Vision-Language Models. ArXiv preprint abs/2309.04461 (2023). https://arxiv.org/abs/2309.04461 - Cheng et al. (2024) Qinyuan Cheng, Xiaonan Li, Shimin Li, Qin Zhu, Zhangyue Yin, Yunfan Shao, Linyang Li, Tianxiang Sun, Hang Yan, and Xipeng Qiu. 2024. Unified Active Retrieval for Retrieval Augmented Generation. CoRR abs/2406.12534 (2024). https://doi.org/10.48550/ARXIV.2406.12534 arXiv:2406.12534 - Cheng et al. (2023) Qinyuan Cheng, Tianxiang Sun, Wenwei Zhang, Siyin Wang, Xiangyang Liu, Mozhi Zhang, Junliang He, Mianqiu Huang, Zhangyue Yin, Kai Chen, and Xipeng Qiu. 2023. Evaluating Hallucinations in Chinese Large Language Models. arXiv:2310.03368 [cs.CL] - Chern et al. (2023) I Chern, Steffi Chern, Shiqi Chen, Weizhe Yuan, Kehua Feng, Chunting Zhou, Junxian He, Graham Neubig, Pengfei Liu, et al. 2023. FacTool: Factuality Detection in Generative AI–A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarios. ArXiv preprint abs/2307.13528 (2023). https://arxiv.org/abs/2307.13528 - Chiang and Lee (2023) Cheng-Han Chiang and Hung-yi Lee. 2023. Can Large Language Models Be an Alternative to Human Evaluations? ArXiv preprint abs/2305.01937 (2023). https://arxiv.org/abs/2305.01937 - Chiang and Cholak (2022) David Chiang and Peter Cholak. 2022. Overcoming a Theoretical Limitation of Self-Attention. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Dublin, Ireland, 7654–7664. https://doi.org/10.18653/v1/2022.acl-long.527 - Choi et al. (2023) Sehyun Choi, Tianqing Fang, Zhaowei Wang, and Yangqiu Song. 2023. KCTS: Knowledge-Constrained Tree Search Decoding with Token-Level Hallucination Detection. ArXiv preprint abs/2310.09044 (2023). https://arxiv.org/abs/2310.09044 - Chowdhery et al. (2023) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2023. PaLM: Scaling Language Modeling with Pathways. J. Mach. Learn. Res. 24 (2023), 240:1–240:113. http://jmlr.org/papers/v24/22-1144.html - Christiano et al. (2017) Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep Reinforcement Learning from Human Preferences. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (Eds.). 4299–4307. https://proceedings.neurips.cc/paper/2017/hash/d5e2c0adad503c91f91df240d0cd4e49-Abstract.html - Chu et al. (2024) Zheng Chu, Jingchang Chen, Qianglong Chen, Haotian Wang, Kun Zhu, Xiyuan Du, Weijiang Yu, Ming Liu, and Bing Qin. 2024. BeamAggR: Beam Aggregation Reasoning over Multi-source Knowledge for Multi-hop Question Answering. arXiv:2406.19820 [cs.CL] https://arxiv.org/abs/2406.19820 - Chu et al. (2023a) Zheng Chu, Jingchang Chen, Qianglong Chen, Weijiang Yu, Tao He, Haotian Wang, Weihua Peng, Ming Liu, Bing Qin, and Ting Liu. 2023a. A Survey of Chain of Thought Reasoning: Advances, Frontiers and Future. ArXiv preprint abs/2309.15402 (2023). https://arxiv.org/abs/2309.15402 - Chu et al. (2023b) Zheng Chu, Jingchang Chen, Qianglong Chen, Weijiang Yu, Tao He, Haotian Wang, Weihua Peng, Ming Liu, Bing Qin, and Ting Liu. 2023b. A Survey of Chain of Thought Reasoning: Advances, Frontiers and Future. CoRR abs/2309.15402 (2023). https://doi.org/10.48550/ARXIV.2309.15402 arXiv:2309.15402 - Chuang et al. (2023) Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James Glass, and Pengcheng He. 2023. Dola: Decoding by contrasting layers improves factuality in large language models. ArXiv preprint abs/2309.03883 (2023). https://arxiv.org/abs/2309.03883 - Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling instruction-finetuned language models. ArXiv preprint abs/2210.11416 (2022). https://arxiv.org/abs/2210.11416 - Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training Verifiers to Solve Math Word Problems. CoRR abs/2110.14168 (2021). arXiv:2110.14168 https://arxiv.org/abs/2110.14168 - Cohen et al. (2023) Roi Cohen, May Hamri, Mor Geva, and Amir Globerson. 2023. LM vs LM: Detecting Factual Errors via Cross Examination. ArXiv preprint abs/2305.13281 (2023). https://arxiv.org/abs/2305.13281 - Computer (2023) Together Computer. 2023. RedPajama: an Open Dataset for Training Large Language Models. https://github.com/togethercomputer/RedPajama-Data - Cotra (2021) Ajeya Cotra. 2021. Why AI alignment could be hard with modern deep learning. https://www.cold-takes.com/why-ai-alignment-could-be-hard-with-modern-deep-learning/ Cold Takes. - Cuconasu et al. (2024) Florin Cuconasu, Giovanni Trappolini, Federico Siciliano, Simone Filice, Cesare Campagnano, Yoelle Maarek, Nicola Tonellotto, and Fabrizio Silvestri. 2024. The Power of Noise: Redefining Retrieval for RAG Systems. CoRR abs/2401.14887 (2024). https://doi.org/10.48550/ARXIV.2401.14887 arXiv:2401.14887 - Dai et al. (2022a) Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. 2022a. Knowledge Neurons in Pretrained Transformers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Dublin, Ireland, 8493–8502. https://doi.org/10.18653/v1/2022.acl-long.581 - Dai et al. (2022b) Damai Dai, Wenbin Jiang, Qingxiu Dong, Yajuan Lyu, Qiaoqiao She, and Zhifang Sui. 2022b. Neural knowledge bank for pretrained transformers. ArXiv preprint abs/2208.00399 (2022). https://arxiv.org/abs/2208.00399 - Dai et al. (2023) Sunhao Dai, Yuqi Zhou, Liang Pang, Weihao Liu, Xiaolin Hu, Yong Liu, Xiao Zhang, and Jun Xu. 2023. LLMs may Dominate Information Access: Neural Retrievers are Biased Towards LLM-Generated Texts. CoRR abs/2310.20501 (2023). https://doi.org/10.48550/ARXIV.2310.20501 arXiv:2310.20501 - Dathathri et al. (2020) Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. 2020. Plug and Play Language Models: A Simple Approach to Controlled Text Generation. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net. https://openreview.net/forum?id=H1edEyBKDS - De Cao et al. (2021) Nicola De Cao, Wilker Aziz, and Ivan Titov. 2021. Editing Factual Knowledge in Language Models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 6491–6506. https://doi.org/10.18653/v1/2021.emnlp-main.522 - de Luis Balaguer et al. (2024) Maria Angels de Luis Balaguer, Vinamra Benara, Renato Luiz de Freitas Cunha, Roberto de M. Estevão Filho, Todd Hendry, Daniel Holstein, Jennifer Marsman, Nick Mecklenburg, Sara Malvar, Leonardo O. Nunes, Rafael Padilha, Morris Sharp, Bruno Silva, Swati Sharma, Vijay Aski, and Ranveer Chandra. 2024. RAG vs Fine-tuning: Pipelines, Tradeoffs, and a Case Study on Agriculture. CoRR abs/2401.08406 (2024). https://doi.org/10.48550/ARXIV.2401.08406 arXiv:2401.08406 - Delétang et al. (2023) Grégoire Delétang, Anian Ruoss, Paul-Ambroise Duquenne, Elliot Catt, Tim Genewein, Christopher Mattern, Jordi Grau-Moya, Li Kevin Wenliang, Matthew Aitchison, Laurent Orseau, et al. 2023. Language Modeling Is Compression. ArXiv preprint abs/2309.10668 (2023). https://arxiv.org/abs/2309.10668 - Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), Jill Burstein, Christy Doran, and Thamar Solorio (Eds.). Association for Computational Linguistics, 4171–4186. https://doi.org/10.18653/V1/N19-1423 - Dhuliawala et al. (2023) Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, and Jason Weston. 2023. Chain-of-Verification Reduces Hallucination in Large Language Models. ArXiv preprint abs/2309.11495 (2023). https://arxiv.org/abs/2309.11495 - Ding et al. (2024) Hanxing Ding, Liang Pang, Zihao Wei, Huawei Shen, and Xueqi Cheng. 2024. Retrieve Only When It Needs: Adaptive Retrieval Augmentation for Hallucination Mitigation in Large Language Models. CoRR abs/2402.10612 (2024). https://doi.org/10.48550/ARXIV.2402.10612 arXiv:2402.10612 - Dong et al. (2023) Zican Dong, Tianyi Tang, Junyi Li, Wayne Xin Zhao, and Ji-Rong Wen. 2023. BAMBOO: A Comprehensive Benchmark for Evaluating Long Text Modeling Capacities of Large Language Models. ArXiv preprint abs/2309.13345 (2023). https://arxiv.org/abs/2309.13345 - Durmus et al. (2020) Esin Durmus, He He, and Mona Diab. 2020. FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Online, 5055–5070. https://doi.org/10.18653/v1/2020.acl-main.454 - Dziri et al. (2021a) Nouha Dziri, Andrea Madotto, Osmar Zaïane, and Avishek Joey Bose. 2021a. Neural Path Hunter: Reducing Hallucination in Dialogue Systems via Path Grounding. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 2197–2214. https://doi.org/10.18653/v1/2021.emnlp-main.168 - Dziri et al. (2021b) Nouha Dziri, Hannah Rashkin, Tal Linzen, and David Reitter. 2021b. Evaluating groundedness in dialogue systems: The begin benchmark. ArXiv preprint abs/2105.00071 (2021). https://arxiv.org/abs/2105.00071 - Fabbri et al. (2022) Alexander Fabbri, Chien-Sheng Wu, Wenhao Liu, and Caiming Xiong. 2022. QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, Seattle, United States, 2587–2601. https://doi.org/10.18653/v1/2022.naacl-main.187 - Fabbri et al. (2021) Alexander R. Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. SummEval: Re-evaluating Summarization Evaluation. Transactions of the Association for Computational Linguistics 9 (2021), 391–409. https://doi.org/10.1162/tacl_a_00373 - Falke et al. (2019) Tobias Falke, Leonardo F. R. Ribeiro, Prasetya Ajie Utama, Ido Dagan, and Iryna Gurevych. 2019. Ranking Generated Summaries by Correctness: An Interesting but Challenging Application for Natural Language Inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Florence, Italy, 2214–2220. https://doi.org/10.18653/v1/P19-1213 - Fan et al. (2019) Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. 2019. ELI5: Long Form Question Answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Florence, Italy, 3558–3567. https://doi.org/10.18653/v1/P19-1346 - Fan et al. (2018) Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical Neural Story Generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Melbourne, Australia, 889–898. https://doi.org/10.18653/v1/P18-1082 - Feng et al. (2023a) Huawen Feng, Yan Fan, Xiong Liu, Ting-En Lin, Zekun Yao, Yuchuan Wu, Fei Huang, Yongbin Li, and Qianli Ma. 2023a. Improving Factual Consistency of Text Summarization by Adversarially Decoupling Comprehension and Embellishment Abilities of LLMs. CoRR abs/2310.19347 (2023). https://doi.org/10.48550/ARXIV.2310.19347 arXiv:2310.19347 - Feng et al. (2023c) Shangbin Feng, Weijia Shi, Yuyang Bai, Vidhisha Balachandran, Tianxing He, and Yulia Tsvetkov. 2023c. Cook: Empowering general-purpose language models with modular and collaborative knowledge. arXiv preprint arXiv:2305.09955 (2023). - Feng et al. (2023b) Zhangyin Feng, Xiaocheng Feng, Dezhi Zhao, Maojin Yang, and Bing Qin. 2023b. Retrieval-Generation Synergy Augmented Large Language Models. ArXiv preprint abs/2310.05149 (2023). https://arxiv.org/abs/2310.05149 - Fierro et al. (2024) Constanza Fierro, Reinald Kim Amplayo, Fantine Huot, Nicola De Cao, Joshua Maynez, Shashi Narayan, and Mirella Lapata. 2024. Learning to Plan and Generate Text with Citations. CoRR abs/2404.03381 (2024). https://doi.org/10.48550/ARXIV.2404.03381 arXiv:2404.03381 - Filippova (2020) Katja Filippova. 2020. Controlled Hallucinations: Learning to Generate Faithfully from Noisy Data. In Findings of the Association for Computational Linguistics: EMNLP 2020. Association for Computational Linguistics, Online, 864–870. https://doi.org/10.18653/v1/2020.findings-emnlp.76 - Friel and Sanyal (2023) Robert Friel and Atindriyo Sanyal. 2023. Chainpoll: A high efficacy method for LLM hallucination detection. arXiv:2310.18344 [cs.CL] - Fu et al. (2023) Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2023. GPTScore: Evaluate as You Desire. CoRR abs/2302.04166 (2023). https://doi.org/10.48550/ARXIV.2302.04166 arXiv:2302.04166 - Gal and Ghahramani (2016) Yarin Gal and Zoubin Ghahramani. 2016. Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning. In Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016 (JMLR Workshop and Conference Proceedings, Vol. 48), Maria-Florina Balcan and Kilian Q. Weinberger (Eds.). JMLR.org, 1050–1059. http://proceedings.mlr.press/v48/gal16.html - Gao et al. (2021a) Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. 2021a. The pile: An 800gb dataset of diverse text for language modeling. ArXiv preprint abs/2101.00027 (2021). https://arxiv.org/abs/2101.00027 - Gao et al. (2023a) Luyu Gao, Zhuyun Dai, Panupong Pasupat, Anthony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vincent Y. Zhao, Ni Lao, Hongrae Lee, Da-Cheng Juan, and Kelvin Guu. 2023a. RARR: Researching and Revising What Language Models Say, Using Language Models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (Eds.). Association for Computational Linguistics, 16477–16508. https://aclanthology.org/2023.acl-long.910 - Gao et al. (2023b) Mingqi Gao, Jie Ruan, Renliang Sun, Xunjian Yin, Shiping Yang, and Xiaojun Wan. 2023b. Human-like summarization evaluation with chatgpt. ArXiv preprint abs/2304.02554 (2023). https://arxiv.org/abs/2304.02554 - Gao et al. (2021b) Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021b. SimCSE: Simple Contrastive Learning of Sentence Embeddings. CoRR abs/2104.08821 (2021). arXiv:2104.08821 https://arxiv.org/abs/2104.08821 - Gao et al. (2023c) Yunfan Gao, Tao Sheng, Youlin Xiang, Yun Xiong, Haofen Wang, and Jiawei Zhang. 2023c. Chat-REC: Towards Interactive and Explainable LLMs-Augmented Recommender System. CoRR abs/2303.14524 (2023). https://doi.org/10.48550/ARXIV.2303.14524 arXiv:2303.14524 - Gekhman et al. (2024) Zorik Gekhman, Gal Yona, Roee Aharoni, Matan Eyal, Amir Feder, Roi Reichart, and Jonathan Herzig. 2024. Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations? CoRR abs/2405.05904 (2024). https://doi.org/10.48550/ARXIV.2405.05904 arXiv:2405.05904 - Goodrich et al. (2019) Ben Goodrich, Vinay Rao, Peter J. Liu, and Mohammad Saleh. 2019. Assessing The Factual Accuracy of Generated Text. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2019, Anchorage, AK, USA, August 4-8, 2019, Ankur Teredesai, Vipin Kumar, Ying Li, Rómer Rosales, Evimaria Terzi, and George Karypis (Eds.). ACM, 166–175. https://doi.org/10.1145/3292500.3330955 - Google (2023) Google. 2023. Bard. https://bard.google.com/ - Goyal and Durrett (2020) Tanya Goyal and Greg Durrett. 2020. Evaluating Factuality in Generation with Dependency-level Entailment. In Findings of the Association for Computational Linguistics: EMNLP 2020. Association for Computational Linguistics, Online, 3592–3603. https://doi.org/10.18653/v1/2020.findings-emnlp.322 - Goyal and Durrett (2021) Tanya Goyal and Greg Durrett. 2021. Annotating and Modeling Fine-grained Factuality in Summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, Online, 1449–1462. https://doi.org/10.18653/v1/2021.naacl-main.114 - Gu et al. (2022) Yuxuan Gu, Xiaocheng Feng, Sicheng Ma, Jiaming Wu, Heng Gong, and Bing Qin. 2022. Improving Controllable Text Generation with Position-Aware Weighted Decoding. In Findings of the Association for Computational Linguistics: ACL 2022. Association for Computational Linguistics, Dublin, Ireland, 3449–3467. https://doi.org/10.18653/v1/2022.findings-acl.272 - Guan et al. (2023) Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, et al. 2023. Hallusionbench: An advanced diagnostic suite for entangled language hallucination & visual illusion in large vision-language models. arXiv preprint arXiv:2310.14566 (2023). - Guerreiro et al. (2023a) Nuno Miguel Guerreiro, Duarte M. Alves, Jonas Waldendorf, Barry Haddow, Alexandra Birch, Pierre Colombo, and André F. T. Martins. 2023a. Hallucinations in Large Multilingual Translation Models. ArXiv preprint abs/2303.16104 (2023). https://arxiv.org/abs/2303.16104 - Guerreiro et al. (2023b) Nuno M. Guerreiro, Elena Voita, and André Martins. 2023b. Looking for a Needle in a Haystack: A Comprehensive Study of Hallucinations in Neural Machine Translation. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics. Association for Computational Linguistics, Dubrovnik, Croatia, 1059–1075. https://aclanthology.org/2023.eacl-main.75 - Gunasekar et al. (2023) Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. 2023. Textbooks Are All You Need. ArXiv preprint abs/2306.11644 (2023). https://arxiv.org/abs/2306.11644 - Gunjal et al. (2023) Anisha Gunjal, Jihan Yin, and Erhan Bas. 2023. Detecting and preventing hallucinations in large vision language models. ArXiv preprint abs/2308.06394 (2023). https://arxiv.org/abs/2308.06394 - Guu et al. (2020) Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020. Retrieval Augmented Language Model Pre-Training. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event (Proceedings of Machine Learning Research, Vol. 119). PMLR, 3929–3938. http://proceedings.mlr.press/v119/guu20a.html - Gyawali et al. (2020) Bikash Gyawali, Lucas Anastasiou, and Petr Knoth. 2020. Deduplication of Scholarly Documents using Locality Sensitive Hashing and Word Embeddings. In Proceedings of the Twelfth Language Resources and Evaluation Conference. European Language Resources Association, Marseille, France, 901–910. https://aclanthology.org/2020.lrec-1.113 - Hahn (2020) Michael Hahn. 2020. Theoretical Limitations of Self-Attention in Neural Sequence Models. Transactions of the Association for Computational Linguistics 8 (2020), 156–171. https://doi.org/10.1162/tacl_a_00306 - Han et al. (2024) Tianyang Han, Qing Lian, Rui Pan, Renjie Pi, Jipeng Zhang, Shizhe Diao, Yong Lin, and Tong Zhang. 2024. The Instinctive Bias: Spurious Images lead to Hallucination in MLLMs. arXiv preprint arXiv:2402.03757 (2024). - He et al. (2023b) Hangfeng He, Hongming Zhang, and Dan Roth. 2023b. Rethinking with retrieval: Faithful large language model inference. ArXiv preprint abs/2301.00303 (2023). https://arxiv.org/abs/2301.00303 - He et al. (2023a) Junqing He, Kunhao Pan, Xiaoqun Dong, Zhuoyang Song, Yibo Liu, Yuxin Liang, Hao Wang, Qianguo Sun, Songxin Zhang, Zejian Xie, and Jiaxing Zhang. 2023a. Never Lost in the Middle: Improving Large Language Models via Attention Strengthening Question Answering. CoRR abs/2311.09198 (2023). https://doi.org/10.48550/ARXIV.2311.09198 arXiv:2311.09198 - Henderson et al. (2022) Peter Henderson, Mark S. Krass, Lucia Zheng, Neel Guha, Christopher D. Manning, Dan Jurafsky, and Daniel E. Ho. 2022. Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Dataset. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022, Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (Eds.). http://papers.nips.cc/paper_files/paper/2022/hash/bc218a0c656e49d4b086975a9c785f47-Abstract-Datasets_and_Benchmarks.html - Hendrycks et al. (2021) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring Massive Multitask Language Understanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net. https://openreview.net/forum?id=d7KBjmI3GmQ - Hernandez et al. (2023) Evan Hernandez, Belinda Z Li, and Jacob Andreas. 2023. Inspecting and editing knowledge representations in language models. ArXiv preprint abs/2304.00740 (2023). https://arxiv.org/abs/2304.00740 - Holtzman et al. (2020) Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The Curious Case of Neural Text Degeneration. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net. https://openreview.net/forum?id=rygGQyrFvH - Honovich et al. (2021) Or Honovich, Leshem Choshen, Roee Aharoni, Ella Neeman, Idan Szpektor, and Omri Abend. 2021. $Q^2$ : Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answering. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 7856–7870. https://doi.org/10.18653/v1/2021.emnlp-main.619 - Huang et al. (2024a) Lei Huang, Xiaocheng Feng, Weitao Ma, Yuxuan Gu, Weihong Zhong, Xiachong Feng, Weijiang Yu, Weihua Peng, Duyu Tang, Dandan Tu, and Bing Qin. 2024a. Learning Fine-Grained Grounded Citations for Attributed Large Language Models. In Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association for Computational Linguistics, 14095–14113. https://doi.org/10.18653/V1/2024.FINDINGS-ACL.838 - Huang et al. (2024b) Lei Huang, Xiaocheng Feng, Weitao Ma, Liang Zhao, Yuchun Fan, Weihong Zhong, Dongliang Xu, Qing Yang, Hongtao Liu, and Bing Qin. 2024b. Advancing Large Language Model Attribution through Self-Improving. arXiv:2410.13298 [cs.CL] https://arxiv.org/abs/2410.13298 - Huang et al. (2023c) Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. 2023c. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation. arXiv preprint arXiv:2311.17911 (2023). - Huang et al. (2023b) Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, et al. 2023b. Language is not all you need: Aligning perception with language models. ArXiv preprint abs/2302.14045 (2023). https://arxiv.org/abs/2302.14045 - Huang et al. (2023a) Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, et al. 2023a. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. ArXiv preprint abs/2305.08322 (2023). https://arxiv.org/abs/2305.08322 - Huang et al. (2021a) Yi-Chong Huang, Xia-Chong Feng, Xiao-Cheng Feng, and Bing Qin. 2021a. The Factual Inconsistency Problem in Abstractive Text Summarization: A Survey. CoRR abs/2104.14839 (2021). arXiv:2104.14839 https://arxiv.org/abs/2104.14839 - Huang et al. (2021b) Yi-Chong Huang, Xia-Chong Feng, Xiao-Cheng Feng, and Bing Qin. 2021b. The Factual Inconsistency Problem in Abstractive Text Summarization: A Survey. ArXiv preprint abs/2104.14839 (2021). https://arxiv.org/abs/2104.14839 - Huang et al. (2023d) Zeyu Huang, Yikang Shen, Xiaofeng Zhang, Jie Zhou, Wenge Rong, and Zhang Xiong. 2023d. Transformer-Patcher: One Mistake Worth One Neuron. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net. https://openreview.net/pdf?id=4oYUGeGBPm - Huo et al. (2023) Siqing Huo, Negar Arabzadeh, and Charles L. A. Clarke. 2023. Retrieving Supporting Evidence for LLMs Generated Answers. ArXiv preprint abs/2306.13781 (2023). https://arxiv.org/abs/2306.13781 - Iyer et al. (2022) Srinivasan Iyer, Xi Victoria Lin, Ramakanth Pasunuru, Todor Mihaylov, Daniel Simig, Ping Yu, Kurt Shuster, Tianlu Wang, Qing Liu, Punit Singh Koura, et al. 2022. Opt-iml: Scaling language model instruction meta learning through the lens of generalization. ArXiv preprint abs/2212.12017 (2022). https://arxiv.org/abs/2212.12017 - Izacard et al. (2022) Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2022. Unsupervised Dense Information Retrieval with Contrastive Learning. Trans. Mach. Learn. Res. 2022 (2022). https://openreview.net/forum?id=jKN1pXi7b0 - Izacard et al. (2023) Gautier Izacard, Patrick S. H. Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2023. Atlas: Few-shot Learning with Retrieval Augmented Language Models. J. Mach. Learn. Res. 24 (2023), 251:1–251:43. http://jmlr.org/papers/v24/23-0037.html - Jagerman et al. (2023) Rolf Jagerman, Honglei Zhuang, Zhen Qin, Xuanhui Wang, and Michael Bendersky. 2023. Query Expansion by Prompting Large Language Models. CoRR abs/2305.03653 (2023). https://doi.org/10.48550/ARXIV.2305.03653 arXiv:2305.03653 - Jain et al. (2023) Sameer Jain, Vaishakh Keshava, Swarnashree Mysore Sathyendra, Patrick Fernandes, Pengfei Liu, Graham Neubig, and Chunting Zhou. 2023. Multi-Dimensional Evaluation of Text Summarization with In-Context Learning. ArXiv preprint abs/2306.01200 (2023). https://arxiv.org/abs/2306.01200 - Jeong (2023) Joonhyun Jeong. 2023. Hijacking Context in Large Multi-modal Models. arXiv preprint arXiv:2312.07553 (2023). - Jeong et al. (2024) Soyeong Jeong, Jinheon Baek, Sukmin Cho, Sung Ju Hwang, and Jong C. Park. 2024. Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity. CoRR abs/2403.14403 (2024). https://doi.org/10.48550/ARXIV.2403.14403 arXiv:2403.14403 - Ji et al. (2023a) Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Andrea Madotto, and Pascale Fung. 2023a. Survey of Hallucination in Natural Language Generation. ACM Comput. Surv. 55, 12 (2023), 248:1–248:38. https://doi.org/10.1145/3571730 - Ji et al. (2023b) Ziwei Ji, Tiezheng Yu, Yan Xu, Nayeon Lee, Etsuko Ishii, and Pascale Fung. 2023b. Towards Mitigating Hallucination in Large Language Models via Self-Reflection. ArXiv preprint abs/2310.06271 (2023). https://arxiv.org/abs/2310.06271 - Jiang et al. (2024) Chaoya Jiang, Wei Ye, Mengfan Dong, Hongrui Jia, Haiyang Xu, Ming Yan, Ji Zhang, and Shikun Zhang. 2024. Hal-Eval: A Universal and Fine-grained Hallucination Evaluation Framework for Large Vision Language Models. arXiv preprint arXiv:2402.15721 (2024). - Jiang et al. (2023a) Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. 2023a. LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models. arXiv:2310.05736 [cs.CL] - Jiang et al. (2023b) Zhengbao Jiang, Frank F. Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023b. Active Retrieval Augmented Generation. ArXiv preprint abs/2305.06983 (2023). https://arxiv.org/abs/2305.06983 - Jing et al. (2023) Liqiang Jing, Ruosen Li, Yunmo Chen, Mengzhao Jia, and Xinya Du. 2023. Faithscore: Evaluating hallucinations in large vision-language models. arXiv preprint arXiv:2311.01477 (2023). - Jing et al. (2024) Zhi Jing, Yongye Su, Yikun Han, Bo Yuan, Haiyun Xu, Chunjiang Liu, Kehai Chen, and Min Zhang. 2024. When Large Language Models Meet Vector Databases: A Survey. CoRR abs/2402.01763 (2024). https://doi.org/10.48550/ARXIV.2402.01763 arXiv:2402.01763 - Kadavath et al. (2022) Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. 2022. Language models (mostly) know what they know. ArXiv preprint abs/2207.05221 (2022). https://arxiv.org/abs/2207.05221 - Kamradt (2024) Greg Kamradt. 2024. The 5 Levels Of Text Splitting For Retrieval. youtube. https://www.youtube.com/watch?v=8OJC21T2SL4 - Kandpal et al. (2023) Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. 2023. Large Language Models Struggle to Learn Long-Tail Knowledge. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA (Proceedings of Machine Learning Research, Vol. 202), Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (Eds.). PMLR, 15696–15707. https://proceedings.mlr.press/v202/kandpal23a.html - Karpukhin et al. (2020a) Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020a. Dense Passage Retrieval for Open-Domain Question Answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Online, 6769–6781. https://doi.org/10.18653/v1/2020.emnlp-main.550 - Karpukhin et al. (2020b) Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick S. H. Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020b. Dense Passage Retrieval for Open-Domain Question Answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (Eds.). Association for Computational Linguistics, 6769–6781. https://doi.org/10.18653/V1/2020.EMNLP-MAIN.550 - Kasai et al. (2022) Jungo Kasai, Keisuke Sakaguchi, Yoichi Takahashi, Ronan Le Bras, Akari Asai, Xinyan Yu, Dragomir Radev, Noah A Smith, Yejin Choi, and Kentaro Inui. 2022. RealTime QA: What’s the Answer Right Now? ArXiv preprint abs/2207.13332 (2022). https://arxiv.org/abs/2207.13332 - Katz et al. (2023) Daniel Martin Katz, Michael James Bommarito, Shang Gao, and Pablo Arredondo. 2023. Gpt-4 passes the bar exam. Available at SSRN 4389233 (2023). https://www.datascienceassn.org/sites/default/files/GPT-4%20Passes%20the%20Bar%20Exam.pdf - Khandelwal et al. (2020) Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2020. Generalization through Memorization: Nearest Neighbor Language Models. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net. https://openreview.net/forum?id=HklBjCEKvH - Kojima et al. (2022) Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems 35 (2022), 22199–22213. - Kryscinski et al. (2020) Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2020. Evaluating the Factual Consistency of Abstractive Text Summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Online, 9332–9346. https://doi.org/10.18653/v1/2020.emnlp-main.750 - Laban et al. (2023) Philippe Laban, Wojciech Kryściński, Divyansh Agarwal, Alexander R Fabbri, Caiming Xiong, Shafiq Joty, and Chien-Sheng Wu. 2023. LLMs as Factual Reasoners: Insights from Existing Benchmarks and Beyond. ArXiv preprint abs/2305.14540 (2023). https://arxiv.org/abs/2305.14540 - Laban et al. (2022) Philippe Laban, Tobias Schnabel, Paul N. Bennett, and Marti A. Hearst. 2022. SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization. Transactions of the Association for Computational Linguistics 10 (2022), 163–177. https://doi.org/10.1162/tacl_a_00453 - Ladhak et al. (2023) Faisal Ladhak, Esin Durmus, Mirac Suzgun, Tianyi Zhang, Dan Jurafsky, Kathleen McKeown, and Tatsunori Hashimoto. 2023. When Do Pre-Training Biases Propagate to Downstream Tasks? A Case Study in Text Summarization. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics. Association for Computational Linguistics, Dubrovnik, Croatia, 3206–3219. https://aclanthology.org/2023.eacl-main.234 - Lakshminarayanan et al. (2017) Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. 2017. Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (Eds.). 6402–6413. https://proceedings.neurips.cc/paper/2017/hash/9ef2ed4b7fd2c810847ffa5fa85bce38-Abstract.html - Lanham et al. (2023) Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, Danny Hernandez, Dustin Li, Esin Durmus, Evan Hubinger, Jackson Kernion, Kamile Lukosiute, Karina Nguyen, Newton Cheng, Nicholas Joseph, Nicholas Schiefer, Oliver Rausch, Robin Larson, Sam McCandlish, Sandipan Kundu, Saurav Kadavath, Shannon Yang, Thomas Henighan, Timothy Maxwell, Timothy Telleen-Lawton, Tristan Hume, Zac Hatfield-Dodds, Jared Kaplan, Jan Brauner, Samuel R. Bowman, and Ethan Perez. 2023. Measuring Faithfulness in Chain-of-Thought Reasoning. CoRR abs/2307.13702 (2023). https://doi.org/10.48550/ARXIV.2307.13702 arXiv:2307.13702 - Lattimer et al. (2023) Barrett Martin Lattimer, Patrick Chen, Xinyuan Zhang, and Yi Yang. 2023. Fast and Accurate Factual Inconsistency Detection Over Long Documents. arXiv:2310.13189 [cs.CL] - Lee et al. (2022a) Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. 2022a. Deduplicating Training Data Makes Language Models Better. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Dublin, Ireland, 8424–8445. https://doi.org/10.18653/v1/2022.acl-long.577 - Lee et al. (2022b) Nayeon Lee, Wei Ping, Peng Xu, Mostofa Patwary, Pascale N Fung, Mohammad Shoeybi, and Bryan Catanzaro. 2022b. Factuality enhanced language models for open-ended text generation. Advances in Neural Information Processing Systems 35 (2022), 34586–34599. - Lei et al. (2023) Deren Lei, Yaxi Li, Mingyu Wang, Vincent Yun, Emily Ching, Eslam Kamal, et al. 2023. Chain of Natural Language Inference for Reducing Large Language Model Ungrounded Hallucinations. ArXiv preprint abs/2310.03951 (2023). https://arxiv.org/abs/2310.03951 - Leng et al. (2023) Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. 2023. Mitigating object hallucinations in large vision-language models through visual contrastive decoding. arXiv preprint arXiv:2311.16922 (2023). - Levinstein and Herrmann (2023) BA Levinstein and Daniel A Herrmann. 2023. Still No Lie Detector for Language Models: Probing Empirical and Conceptual Roadblocks. ArXiv preprint abs/2307.00175 (2023). https://arxiv.org/abs/2307.00175 - Lewis et al. (2020a) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020a. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Online, 7871–7880. https://doi.org/10.18653/v1/2020.acl-main.703 - Lewis et al. (2020b) Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020b. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (Eds.). https://proceedings.neurips.cc/paper/2020/hash/6b493230205f780e1bc26945df7481e5-Abstract.html - Li et al. (2023g) Daliang Li, Ankit Singh Rawat, Manzil Zaheer, Xin Wang, Michal Lukasik, Andreas Veit, Felix X. Yu, and Sanjiv Kumar. 2023g. Large Language Models with Controllable Working Memory. In Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023, Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (Eds.). Association for Computational Linguistics, 1774–1793. https://doi.org/10.18653/v1/2023.findings-acl.112 - Li et al. (2024a) Jiachun Li, Pengfei Cao, Yubo Chen, Kang Liu, and Jun Zhao. 2024a. Towards Faithful Chain-of-Thought: Large Language Models are Bridging Reasoners. CoRR abs/2405.18915 (2024). https://doi.org/10.48550/ARXIV.2405.18915 arXiv:2405.18915 - Li et al. (2024c) Junyi Li, Jie Chen, Ruiyang Ren, Xiaoxue Cheng, Wayne Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. 2024c. The Dawn After the Dark: An Empirical Study on Factuality Hallucination in Large Language Models. CoRR abs/2401.03205 (2024). https://doi.org/10.48550/ARXIV.2401.03205 arXiv:2401.03205 - Li et al. (2023b) Junyi Li, Xiaoxue Cheng, Wayne Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. 2023b. HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models. CoRR abs/2305.11747 (2023). https://doi.org/10.48550/ARXIV.2305.11747 arXiv:2305.11747 - Li et al. (2023d) Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023d. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. ArXiv preprint abs/2301.12597 (2023). https://arxiv.org/abs/2301.12597 - Li et al. (2023h) Jinming Li, Wentao Zhang, Tian Wang, Guanglei Xiong, Alan Lu, and Gerard Medioni. 2023h. GPT4Rec: A Generative Framework for Personalized Recommendation and User Interests Interpretation. In Proceedings of the 2023 SIGIR Workshop on eCommerce co-located with the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2023), Taipei, Taiwan, July 27, 2023 (CEUR Workshop Proceedings, Vol. 3589), Surya Kallumadi, Yubin Kim, Tracy Holloway King, Shervin Malmasi, Maarten de Rijke, and Jacopo Tagliabue (Eds.). CEUR-WS.org. https://ceur-ws.org/Vol-3589/paper_2.pdf - Li et al. (2023f) Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. 2023f. Inference-Time Intervention: Eliciting Truthful Answers from a Language Model. ArXiv preprint abs/2306.03341 (2023). https://arxiv.org/abs/2306.03341 - Li et al. (2024b) Minghan Li, Xilun Chen, Ari Holtzman, Beidi Chen, Jimmy Lin, Wen-tau Yih, and Xi Victoria Lin. 2024b. Nearest Neighbor Speculative Decoding for LLM Generation and Attribution. CoRR abs/2405.19325 (2024). https://doi.org/10.48550/ARXIV.2405.19325 arXiv:2405.19325 - Li et al. (2022b) Wei Li, Wenhao Wu, Moye Chen, Jiachen Liu, Xinyan Xiao, and Hua Wu. 2022b. Faithfulness in Natural Language Generation: A Systematic Survey of Analysis, Evaluation and Optimization Methods. ArXiv preprint abs/2203.05227 (2022). https://arxiv.org/abs/2203.05227 - Li et al. (2022a) Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. 2022a. Contrastive decoding: Open-ended text generation as optimization. ArXiv preprint abs/2210.15097 (2022). https://arxiv.org/abs/2210.15097 - Li (2023) Yucheng Li. 2023. Unlocking Context Constraints of LLMs: Enhancing Context Efficiency of LLMs with Self-Information-Based Content Filtering. arXiv:2304.12102 [cs.CL] - Li et al. (2023a) Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. 2023a. Textbooks Are All You Need II: phi-1.5 technical report. ArXiv preprint abs/2309.05463 (2023). https://arxiv.org/abs/2309.05463 - Li et al. (2023c) Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. 2023c. Evaluating Object Hallucination in Large Vision-Language Models. https://arxiv.org/abs/2305.10355 - Li et al. (2023e) Yunxiang Li, Zihan Li, Kai Zhang, Ruilong Dan, and You Zhang. 2023e. ChatDoctor: A Medical Chat Model Fine-tuned on LLaMA Model using Medical Domain Knowledge. ArXiv preprint abs/2303.14070 (2023). https://arxiv.org/abs/2303.14070 - Li et al. (2023i) Zuchao Li, Shitou Zhang, Hai Zhao, Yifei Yang, and Dongjie Yang. 2023i. BatGPT: A Bidirectional Autoregessive Talker from Generative Pre-trained Transformer. ArXiv preprint abs/2307.00360 (2023). https://arxiv.org/abs/2307.00360 - Lin (2004) Chin-Yew Lin. 2004. ROUGE: A Package for Automatic Evaluation of Summaries. In Text Summarization Branches Out. Association for Computational Linguistics, Barcelona, Spain, 74–81. https://aclanthology.org/W04-1013 - Lin et al. (2022) Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. TruthfulQA: Measuring How Models Mimic Human Falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Dublin, Ireland, 3214–3252. https://doi.org/10.18653/v1/2022.acl-long.229 - Liu et al. (2023a) Bingbin Liu, Jordan T. Ash, Surbhi Goel, Akshay Krishnamurthy, and Cyril Zhang. 2023a. Exposing Attention Glitches with Flip-Flop Language Modeling. ArXiv preprint abs/2306.00946 (2023). https://arxiv.org/abs/2306.00946 - Liu et al. (2023b) Fuxiao Liu, Tianrui Guan, Zongxia Li, Lichang Chen, Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou. 2023b. HallusionBench: You See What You Think? Or You Think What You See? An Image-Context Reasoning Benchmark Challenging for GPT-4V(Ision), LLaVA-1.5, and Other Multi-Modality Models. https://arxiv.org/abs/2310.14566 - Liu et al. (2023g) Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. 2023g. Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning. arXiv:2306.14565 [cs.CV] - Liu et al. (2023d) Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023d. Visual instruction tuning. ArXiv preprint abs/2304.08485 (2023). https://arxiv.org/abs/2304.08485 - Liu et al. (2024) Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiutian Zhao, Ke Wang, Liping Hou, Rongjun Li, and Wei Peng. 2024. A survey on hallucination in large vision-language models. arXiv preprint arXiv:2402.00253 (2024). - Liu et al. (2023e) Junyi Liu, Liangzhi Li, Tong Xiang, Bowen Wang, and Yiming Qian. 2023e. TCRA-LLM: Token Compression Retrieval Augmented Large Language Model for Inference Cost Reduction. In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, 9796–9810. https://aclanthology.org/2023.findings-emnlp.655 - Liu et al. (2023f) Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2023f. Lost in the Middle: How Language Models Use Long Contexts. ArXiv preprint abs/2307.03172 (2023). https://arxiv.org/abs/2307.03172 - Liu et al. (2023c) Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023c. Gpteval: Nlg evaluation using gpt-4 with better human alignment. ArXiv preprint abs/2303.16634 (2023). https://arxiv.org/abs/2303.16634 - Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. CoRR abs/1907.11692 (2019). arXiv:1907.11692 http://arxiv.org/abs/1907.11692 - Liu et al. (2023h) Yang Liu, Yuanshun Yao, Jean-Francois Ton, Xiaoying Zhang, Ruocheng Guo, Hao Cheng, Yegor Klochkov, Muhammad Faaiz Taufiq, and Hang Li. 2023h. Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models’ Alignment. ArXiv preprint abs/2308.05374 (2023). https://arxiv.org/abs/2308.05374 - Liu et al. (2023i) Yijin Liu, Xianfeng Zeng, Fandong Meng, and Jie Zhou. 2023i. Instruction Position Matters in Sequence Generation with Large Language Models. ArXiv preprint abs/2308.12097 (2023). https://arxiv.org/abs/2308.12097 - Longpre et al. (2021) Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. 2021. Entity-Based Knowledge Conflicts in Question Answering. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (Eds.). Association for Computational Linguistics, 7052–7063. https://doi.org/10.18653/V1/2021.EMNLP-MAIN.565 - Lovenia et al. (2023) Holy Lovenia, Wenliang Dai, Samuel Cahyawijaya, Ziwei Ji, and Pascale Fung. 2023. Negative Object Presence Evaluation (NOPE) to Measure Object Hallucination in Vision-Language Models. https://arxiv.org/abs/2310.05338 - Lu et al. (2023) Jiaying Lu, Jinmeng Rao, Kezhen Chen, Xiaoyuan Guo, Yawen Zhang, Baochen Sun, Carl Yang, and Jie Yang. 2023. Evaluation and Mitigation of Agnosia in Multimodal Large Language Models. ArXiv preprint abs/2309.04041 (2023). https://arxiv.org/abs/2309.04041 - Luo et al. (2020) Huaishao Luo, Lei Ji, Botian Shi, Haoyang Huang, Nan Duan, Tianrui Li, Jason Li, Taroon Bharti, and Ming Zhou. 2020. Univl: A unified video and language pre-training model for multimodal understanding and generation. ArXiv preprint abs/2002.06353 (2020). https://arxiv.org/abs/2002.06353 - Luo et al. (2023a) Junyu Luo, Cao Xiao, and Fenglong Ma. 2023a. Zero-Resource Hallucination Prevention for Large Language Models. ArXiv preprint abs/2309.02654 (2023). https://arxiv.org/abs/2309.02654 - Luo et al. (2023b) Zheheng Luo, Qianqian Xie, and Sophia Ananiadou. 2023b. Chatgpt as a factual inconsistency evaluator for text summarization. - Ma et al. (2023) Xinbei Ma, Yeyun Gong, Pengcheng He, Hai Zhao, and Nan Duan. 2023. Query Rewriting for Retrieval-Augmented Large Language Models. CoRR abs/2305.14283 (2023). https://doi.org/10.48550/ARXIV.2305.14283 arXiv:2305.14283 - Maaz et al. (2023) Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. 2023. Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models. ArXiv preprint abs/2306.05424 (2023). https://arxiv.org/abs/2306.05424 - Macpherson and Platchias (2013) Fiona Macpherson and Dimitris Platchias. 2013. Hallucination: Philosophy and psychology. MIT Press. https://books.google.com/books?hl=zh-CN&lr=&id=_bwtAAAAQBAJ&oi=fnd&pg=PR5&dq=Hallucination:+Philosophy+and+psychology&ots=2E62kf7_yC&sig=rH9HGXYacNkxOJNMVbw514aChZo - Malaviya et al. (2023) Chaitanya Malaviya, Subin Lee, Sihao Chen, Elizabeth Sieber, Mark Yatskar, and Dan Roth. 2023. ExpertQA: Expert-Curated Questions and Attributed Answers. arXiv:2309.07852 [cs.CL] - Mallen et al. (2023) Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (Eds.). Association for Computational Linguistics, 9802–9822. https://doi.org/10.18653/v1/2023.acl-long.546 - Manakul et al. (2023) Potsawee Manakul, Adian Liusie, and Mark J. F. Gales. 2023. SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models. ArXiv preprint abs/2303.08896 (2023). https://arxiv.org/abs/2303.08896 - Manber and Myers (1993) Udi Manber and Gene Myers. 1993. Suffix arrays: a new method for on-line string searches. siam Journal on Computing 22, 5 (1993), 935–948. - Mao et al. (2024) Shengyu Mao, Yong Jiang, Boli Chen, Xiao Li, Peng Wang, Xinyu Wang, Pengjun Xie, Fei Huang, Huajun Chen, and Ningyu Zhang. 2024. RaFe: Ranking Feedback Improves Query Rewriting for RAG. CoRR abs/2405.14431 (2024). https://doi.org/10.48550/ARXIV.2405.14431 arXiv:2405.14431 - Maynez et al. (2020) Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On Faithfulness and Factuality in Abstractive Summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Online, 1906–1919. https://doi.org/10.18653/v1/2020.acl-main.173 - Meister et al. (2020) Clara Meister, Ryan Cotterell, and Tim Vieira. 2020. If beam search is the answer, what was the question?. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Online, 2173–2185. https://doi.org/10.18653/v1/2020.emnlp-main.170 - Meng et al. (2022) Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and Editing Factual Associations in GPT. In NeurIPS. http://papers.nips.cc/paper_files/paper/2022/hash/6f1d43d5a82a37e89b0665b33bf3a182-Abstract-Conference.html - Meng et al. (2023) Kevin Meng, Arnab Sen Sharma, Alex J. Andonian, Yonatan Belinkov, and David Bau. 2023. Mass-Editing Memory in a Transformer. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net. https://openreview.net/pdf?id=MkbcAHIYgyS - Miao et al. (2021) Mengqi Miao, Fandong Meng, Yijin Liu, Xiao-Hua Zhou, and Jie Zhou. 2021. Prevent the Language Model from being Overconfident in Neural Machine Translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics, Online, 3456–3468. https://doi.org/10.18653/v1/2021.acl-long.268 - Miao et al. (2023) Ning Miao, Yee Whye Teh, and Tom Rainforth. 2023. Selfcheck: Using llms to zero-shot check their own step-by-step reasoning. ArXiv preprint abs/2308.00436 (2023). https://arxiv.org/abs/2308.00436 - Microsoft (2023) Microsoft. 2023. New Bing. https://www.bing.com/new - Min et al. (2023a) Sewon Min, Suchin Gururangan, Eric Wallace, Hannaneh Hajishirzi, Noah A. Smith, and Luke Zettlemoyer. 2023a. SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore. CoRR abs/2308.04430 (2023). https://doi.org/10.48550/ARXIV.2308.04430 arXiv:2308.04430 - Min et al. (2023b) Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023b. FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation. ArXiv preprint abs/2305.14251 (2023). https://arxiv.org/abs/2305.14251 - Mishra et al. (2021) Anshuman Mishra, Dhruvesh Patel, Aparna Vijayakumar, Xiang Lorraine Li, Pavan Kapanipathi, and Kartik Talamadupula. 2021. Looking Beyond Sentence-Level Natural Language Inference for Question Answering and Text Summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, Online, 1322–1336. https://doi.org/10.18653/v1/2021.naacl-main.104 - Mitchell et al. (2022a) Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D. Manning. 2022a. Fast Model Editing at Scale. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net. https://openreview.net/forum?id=0DcZxeWfOPt - Mitchell et al. (2022b) Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D. Manning, and Chelsea Finn. 2022b. Memory-Based Model Editing at Scale. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA (Proceedings of Machine Learning Research, Vol. 162), Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (Eds.). PMLR, 15817–15831. https://proceedings.mlr.press/v162/mitchell22a.html - Moschella et al. (2022) Luca Moschella, Valentino Maiorca, Marco Fumero, Antonio Norelli, Francesco Locatello, and Emanuele Rodola. 2022. Relative representations enable zero-shot latent space communication. ArXiv preprint abs/2209.15430 (2022). https://arxiv.org/abs/2209.15430 - Muennighoff et al. (2024) Niklas Muennighoff, Hongjin Su, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Amanpreet Singh, and Douwe Kiela. 2024. Generative Representational Instruction Tuning. CoRR abs/2402.09906 (2024). https://doi.org/10.48550/ARXIV.2402.09906 arXiv:2402.09906 - Muennighoff et al. (2023) Niklas Muennighoff, Nouamane Tazi, Loïc Magne, and Nils Reimers. 2023. MTEB: Massive Text Embedding Benchmark. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2023, Dubrovnik, Croatia, May 2-6, 2023, Andreas Vlachos and Isabelle Augenstein (Eds.). Association for Computational Linguistics, 2006–2029. https://doi.org/10.18653/V1/2023.EACL-MAIN.148 - Muhlgay et al. (2023) Dor Muhlgay, Ori Ram, Inbal Magar, Yoav Levine, Nir Ratner, Yonatan Belinkov, Omri Abend, Kevin Leyton-Brown, Amnon Shashua, and Yoav Shoham. 2023. Generating benchmarks for factuality evaluation of language models. ArXiv preprint abs/2307.06908 (2023). https://arxiv.org/abs/2307.06908 - Nair et al. (2023) Inderjeet Nair, Aparna Garimella, Balaji Vasan Srinivasan, Natwar Modani, Niyati Chhaya, Srikrishna Karanam, and Sumit Shekhar. 2023. A Neural CRF-based Hierarchical Approach for Linear Text Segmentation. In Findings of the Association for Computational Linguistics: EACL 2023, Dubrovnik, Croatia, May 2-6, 2023, Andreas Vlachos and Isabelle Augenstein (Eds.). Association for Computational Linguistics, 853–863. https://doi.org/10.18653/V1/2023.FINDINGS-EACL.65 - Nan et al. (2021) Feng Nan, Ramesh Nallapati, Zhiguo Wang, Cicero Nogueira dos Santos, Henghui Zhu, Dejiao Zhang, Kathleen McKeown, and Bing Xiang. 2021. Entity-level Factual Consistency of Abstractive Text Summarization. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume. Association for Computational Linguistics, Online, 2727–2733. https://doi.org/10.18653/v1/2021.eacl-main.235 - Narayanan Venkit et al. (2023) Pranav Narayanan Venkit, Sanjana Gautam, Ruchi Panchanadikar, Ting-Hao Huang, and Shomir Wilson. 2023. Nationality Bias in Text Generation. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics. Association for Computational Linguistics, Dubrovnik, Croatia, 116–122. https://aclanthology.org/2023.eacl-main.9 - Neeman et al. (2023) Ella Neeman, Roee Aharoni, Or Honovich, Leshem Choshen, Idan Szpektor, and Omri Abend. 2023. DisentQA: Disentangling Parametric and Contextual Knowledge with Counterfactual Question Answering. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (Eds.). Association for Computational Linguistics, 10056–10070. https://doi.org/10.18653/V1/2023.ACL-LONG.559 - Ni et al. (2024) Shiyu Ni, Keping Bi, Jiafeng Guo, and Xueqi Cheng. 2024. When Do LLMs Need Retrieval Augmentation? Mitigating LLMs’ Overconfidence Helps Retrieval Augmentation. CoRR abs/2402.11457 (2024). https://doi.org/10.48550/ARXIV.2402.11457 arXiv:2402.11457 - O’Brien and Lewis (2023) Sean O’Brien and Mike Lewis. 2023. Contrastive Decoding Improves Reasoning in Large Language Models. ArXiv preprint abs/2309.09117 (2023). https://arxiv.org/abs/2309.09117 - Onoe et al. (2022) Yasumasa Onoe, Michael Zhang, Eunsol Choi, and Greg Durrett. 2022. Entity Cloze By Date: What LMs Know About Unseen Entities. In Findings of the Association for Computational Linguistics: NAACL 2022. Association for Computational Linguistics, Seattle, United States, 693–702. https://doi.org/10.18653/v1/2022.findings-naacl.52 - OpenAI (2022) OpenAI. 2022. Introducing chatgpt. https://openai.com/blog/chatgpt - OpenAI (2023) OpenAI. 2023. GPT-4 Technical Report. ArXiv preprint abs/2303.08774 (2023). https://arxiv.org/abs/2303.08774 - Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In NeurIPS. http://papers.nips.cc/paper_files/paper/2022/hash/b1efde53be364a73914f58805a001731-Abstract-Conference.html - Ovadia et al. (2023) Oded Ovadia, Menachem Brief, Moshik Mishaeli, and Oren Elisha. 2023. Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs. CoRR abs/2312.05934 (2023). https://doi.org/10.48550/ARXIV.2312.05934 arXiv:2312.05934 - Pacchiardi et al. (2023) Lorenzo Pacchiardi, Alex J Chan, Sören Mindermann, Ilan Moscovitz, Alexa Y Pan, Yarin Gal, Owain Evans, and Jan Brauner. 2023. How to Catch an AI Liar: Lie Detection in Black-Box LLMs by Asking Unrelated Questions. ArXiv preprint abs/2309.15840 (2023). https://arxiv.org/abs/2309.15840 - Pagnoni et al. (2021) Artidoro Pagnoni, Vidhisha Balachandran, and Yulia Tsvetkov. 2021. Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, Online, 4812–4829. https://doi.org/10.18653/v1/2021.naacl-main.383 - Pan et al. (2023) Liangming Pan, Michael Saxon, Wenda Xu, Deepak Nathani, Xinyi Wang, and William Yang Wang. 2023. Automatically correcting large language models: Surveying the landscape of diverse self-correction strategies. ArXiv preprint abs/2308.03188 (2023). https://arxiv.org/abs/2308.03188 - Pan et al. (2024) Ruotong Pan, Boxi Cao, Hongyu Lin, Xianpei Han, Jia Zheng, Sirui Wang, Xunliang Cai, and Le Sun. 2024. Not All Contexts Are Equal: Teaching LLMs Credibility-aware Generation. CoRR abs/2404.06809 (2024). https://doi.org/10.48550/ARXIV.2404.06809 arXiv:2404.06809 - Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Philadelphia, Pennsylvania, USA, 311–318. https://doi.org/10.3115/1073083.1073135 - Parikh et al. (2020) Ankur P. Parikh, Xuezhi Wang, Sebastian Gehrmann, Manaal Faruqui, Bhuwan Dhingra, Diyi Yang, and Dipanjan Das. 2020. ToTTo: A Controlled Table-To-Text Generation Dataset. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (Eds.). Association for Computational Linguistics, 1173–1186. https://doi.org/10.18653/V1/2020.EMNLP-MAIN.89 - Paul et al. (2024) Debjit Paul, Robert West, Antoine Bosselut, and Boi Faltings. 2024. Making Reasoning Matter: Measuring and Improving Faithfulness of Chain-of-Thought Reasoning. CoRR abs/2402.13950 (2024). https://doi.org/10.48550/ARXIV.2402.13950 arXiv:2402.13950 - Paullada et al. (2021) Amandalynne Paullada, Inioluwa Deborah Raji, Emily M. Bender, Emily Denton, and Alex Hanna. 2021. Data and its (dis)contents: A survey of dataset development and use in machine learning research. Patterns 2, 11 (2021), 100336. https://doi.org/10.1016/J.PATTER.2021.100336 - Penedo et al. (2023) Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. 2023. The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only. ArXiv preprint abs/2306.01116 (2023). https://arxiv.org/abs/2306.01116 - Peng et al. (2023) Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. 2023. Instruction tuning with gpt-4. ArXiv preprint abs/2304.03277 (2023). https://arxiv.org/abs/2304.03277 - Perez et al. (2023) Ethan Perez, Sam Ringer, Kamile Lukosiute, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, Andy Jones, Anna Chen, Benjamin Mann, Brian Israel, Bryan Seethor, Cameron McKinnon, Christopher Olah, Da Yan, Daniela Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran-Johnson, Guro Khundadze, Jackson Kernion, James Landis, Jamie Kerr, Jared Mueller, Jeeyoon Hyun, Joshua Landau, Kamal Ndousse, Landon Goldberg, Liane Lovitt, Martin Lucas, Michael Sellitto, Miranda Zhang, Neerav Kingsland, Nelson Elhage, Nicholas Joseph, Noemí Mercado, Nova DasSarma, Oliver Rausch, Robin Larson, Sam McCandlish, Scott Johnston, Shauna Kravec, Sheer El Showk, Tamera Lanham, Timothy Telleen-Lawton, Tom Brown, Tom Henighan, Tristan Hume, Yuntao Bai, Zac Hatfield-Dodds, Jack Clark, Samuel R. Bowman, Amanda Askell, Roger Grosse, Danny Hernandez, Deep Ganguli, Evan Hubinger, Nicholas Schiefer, and Jared Kaplan. 2023. Discovering Language Model Behaviors with Model-Written Evaluations. In Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023, Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (Eds.). Association for Computational Linguistics, 13387–13434. https://doi.org/10.18653/V1/2023.FINDINGS-ACL.847 - Petroni et al. (2019) Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language Models as Knowledge Bases?. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). Association for Computational Linguistics, Hong Kong, China, 2463–2473. https://doi.org/10.18653/v1/D19-1250 - Press et al. (2022) Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. 2022. Measuring and narrowing the compositionality gap in language models. ArXiv preprint abs/2210.03350 (2022). https://arxiv.org/abs/2210.03350 - Qi et al. (2024) Jirui Qi, Gabriele Sarti, Raquel Fernández, and Arianna Bisazza. 2024. Model Internals-based Answer Attribution for Trustworthy Retrieval-Augmented Generation. CoRR abs/2406.13663 (2024). https://doi.org/10.48550/ARXIV.2406.13663 arXiv:2406.13663 - Qi et al. (2023) Zhixiao Qi, Yijiong Yu, Meiqi Tu, Junyi Tan, and Yongfeng Huang. 2023. FoodGPT: A Large Language Model in Food Testing Domain with Incremental Pre-training and Knowledge Graph Prompt. ArXiv preprint abs/2308.10173 (2023). https://arxiv.org/abs/2308.10173 - Qiao et al. (2022) Shuofei Qiao, Yixin Ou, Ningyu Zhang, Xiang Chen, Yunzhi Yao, Shumin Deng, Chuanqi Tan, Fei Huang, and Huajun Chen. 2022. Reasoning with language model prompting: A survey. ArXiv preprint abs/2212.09597 (2022). https://arxiv.org/abs/2212.09597 - Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. 2018. Improving language understanding by generative pre-training. (2018). - Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog 1, 8 (2019), 9. - Rafailov et al. (2023) Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. 2023. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. ArXiv preprint abs/2305.18290 (2023). https://arxiv.org/abs/2305.18290 - Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. J. Mach. Learn. Res. 21 (2020), 140:1–140:67. http://jmlr.org/papers/v21/20-074.html - Ram et al. (2023) Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. 2023. In-Context Retrieval-Augmented Language Models. ArXiv preprint abs/2302.00083 (2023). https://arxiv.org/abs/2302.00083 - Ranzato et al. (2016) Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2016. Sequence Level Training with Recurrent Neural Networks. In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, Yoshua Bengio and Yann LeCun (Eds.). http://arxiv.org/abs/1511.06732 - Ravaut et al. (2024) Mathieu Ravaut, Aixin Sun, Nancy F. Chen, and Shafiq Joty. 2024. On Context Utilization in Summarization with Large Language Models. arXiv:2310.10570 [cs.CL] - Rawte et al. (2023) Vipula Rawte, Amit P. Sheth, and Amitava Das. 2023. A Survey of Hallucination in Large Foundation Models. ArXiv preprint abs/2309.05922 (2023). https://arxiv.org/abs/2309.05922 - Reid et al. (2024) Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, Ioannis Antonoglou, Rohan Anil, Sebastian Borgeaud, Andrew Dai, Katie Millican, Ethan Dyer, Mia Glaese, Thibault Sottiaux, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, James Molloy, Jilin Chen, Michael Isard, Paul Barham, Tom Hennigan, Ross McIlroy, Melvin Johnson, Johan Schalkwyk, Eli Collins, Eliza Rutherford, Erica Moreira, Kareem Ayoub, Megha Goel, Clemens Meyer, Gregory Thornton, Zhen Yang, Henryk Michalewski, Zaheer Abbas, Nathan Schucher, Ankesh Anand, Richard Ives, James Keeling, Karel Lenc, Salem Haykal, Siamak Shakeri, Pranav Shyam, Aakanksha Chowdhery, Roman Ring, Stephen Spencer, Eren Sezener, Luke Vilnis, Oscar Chang, Nobuyuki Morioka, George Tucker, Ce Zheng, Oliver Woodman, Nithya Attaluri, Tomas Kocisky, Evgenii Eltyshev, Xi Chen, Timothy Chung, Vittorio Selo, Siddhartha Brahma, Petko Georgiev, Ambrose Slone, Zhenkai Zhu, James Lottes, Siyuan Qiao, Ben Caine, Sebastian Riedel, Alex Tomala, Martin Chadwick, Juliette Love, Peter Choy, Sid Mittal, Neil Houlsby, Yunhao Tang, Matthew Lamm, Libin Bai, Qiao Zhang, Luheng He, Yong Cheng, Peter Humphreys, Yujia Li, Sergey Brin, Albin Cassirer, Yingjie Miao, Lukas Zilka, Taylor Tobin, Kelvin Xu, Lev Proleev, Daniel Sohn, Alberto Magni, Lisa Anne Hendricks, Isabel Gao, Santiago Ontañón, Oskar Bunyan, Nathan Byrd, Abhanshu Sharma, Biao Zhang, Mario Pinto, Rishika Sinha, Harsh Mehta, Dawei Jia, Sergi Caelles, Albert Webson, Alex Morris, Becca Roelofs, Yifan Ding, Robin Strudel, Xuehan Xiong, Marvin Ritter, Mostafa Dehghani, Rahma Chaabouni, Abhijit Karmarkar, Guangda Lai, Fabian Mentzer, Bibo Xu, YaGuang Li, Yujing Zhang, Tom Le Paine, Alex Goldin, Behnam Neyshabur, Kate Baumli, Anselm Levskaya, Michael Laskin, Wenhao Jia, Jack W. Rae, Kefan Xiao, Antoine He, Skye Giordano, Lakshman Yagati, Jean-Baptiste Lespiau, Paul Natsev, Sanjay Ganapathy, Fangyu Liu, Danilo Martins, Nanxin Chen, Yunhan Xu, Megan Barnes, Rhys May, Arpi Vezer, Junhyuk Oh, Ken Franko, Sophie Bridgers, Ruizhe Zhao, Boxi Wu, Basil Mustafa, Sean Sechrist, Emilio Parisotto, Thanumalayan Sankaranarayana Pillai, Chris Larkin, Chenjie Gu, Christina Sorokin, Maxim Krikun, Alexey Guseynov, Jessica Landon, Romina Datta, Alexander Pritzel, Phoebe Thacker, Fan Yang, Kevin Hui, Anja Hauth, Chih-Kuan Yeh, David Barker, Justin Mao-Jones, Sophia Austin, Hannah Sheahan, Parker Schuh, James Svensson, Rohan Jain, Vinay Ramasesh, Anton Briukhov, Da-Woon Chung, Tamara von Glehn, Christina Butterfield, Priya Jhakra, Matthew Wiethoff, Justin Frye, Jordan Grimstad, Beer Changpinyo, Charline Le Lan, Anna Bortsova, Yonghui Wu, Paul Voigtlaender, Tara Sainath, Charlotte Smith, Will Hawkins, Kris Cao, James Besley, Srivatsan Srinivasan, Mark Omernick, Colin Gaffney, Gabriela Surita, Ryan Burnell, Bogdan Damoc, Junwhan Ahn, Andrew Brock, Mantas Pajarskas, Anastasia Petrushkina, Seb Noury, Lorenzo Blanco, Kevin Swersky, Arun Ahuja, Thi Avrahami, Vedant Misra, Raoul de Liedekerke, Mariko Iinuma, Alex Polozov, Sarah York, George van den Driessche, Paul Michel, Justin Chiu, Rory Blevins, Zach Gleicher, Adrià Recasens, Alban Rrustemi, Elena Gribovskaya, Aurko Roy, Wiktor Gworek, Séb Arnold, Lisa Lee, James Lee-Thorp, Marcello Maggioni, Enrique Piqueras, Kartikeya Badola, Sharad Vikram, Lucas Gonzalez, Anirudh Baddepudi, Evan Senter, Jacob Devlin, James Qin, Michael Azzam, Maja Trebacz, Martin Polacek, Kashyap Krishnakumar, Shuo yiin Chang, Matthew Tung, Ivo Penchev, Rishabh Joshi, Kate Olszewska, Carrie Muir, Mateo Wirth, Ale Jakse Hartman, Josh Newlan, Sheleem Kashem, Vijay Bolina, Elahe Dabir, Joost van Amersfoort, Zafarali Ahmed, James Cobon-Kerr, Aishwarya Kamath, Arnar Mar Hrafnkelsson, Le Hou, Ian Mackinnon, Alexandre Frechette, Eric Noland, Xiance Si, Emanuel Taropa, Dong Li, Phil Crone, Anmol Gulati, Sébastien Cevey, Jonas Adler, Ada Ma, David Silver, Simon Tokumine, Richard Powell, Stephan Lee, Michael Chang, Samer Hassan, Diana Mincu, Antoine Yang, Nir Levine, Jenny Brennan, Mingqiu Wang, Sarah Hodkinson, Jeffrey Zhao, Josh Lipschultz, Aedan Pope, Michael B. Chang, Cheng Li, Laurent El Shafey, Michela Paganini, Sholto Douglas, Bernd Bohnet, Fabio Pardo, Seth Odoom, Mihaela Rosca, Cicero Nogueira dos Santos, Kedar Soparkar, Arthur Guez, Tom Hudson, Steven Hansen, Chulayuth Asawaroengchai, Ravi Addanki, Tianhe Yu, Wojciech Stokowiec, Mina Khan, Justin Gilmer, Jaehoon Lee, Carrie Grimes Bostock, Keran Rong, Jonathan Caton, Pedram Pejman, Filip Pavetic, Geoff Brown, Vivek Sharma, Mario Lučić, Rajkumar Samuel, Josip Djolonga, Amol Mandhane, Lars Lowe Sjösund, Elena Buchatskaya, Elspeth White, Natalie Clay, Jiepu Jiang, Hyeontaek Lim, Ross Hemsley, Jane Labanowski, Nicola De Cao, David Steiner, Sayed Hadi Hashemi, Jacob Austin, Anita Gergely, Tim Blyth, Joe Stanton, Kaushik Shivakumar, Aditya Siddhant, Anders Andreassen, Carlos Araya, Nikhil Sethi, Rakesh Shivanna, Steven Hand, Ankur Bapna, Ali Khodaei, Antoine Miech, Garrett Tanzer, Andy Swing, Shantanu Thakoor, Zhufeng Pan, Zachary Nado, Stephanie Winkler, Dian Yu, Mohammad Saleh, Loren Maggiore, Iain Barr, Minh Giang, Thais Kagohara, Ivo Danihelka, Amit Marathe, Vladimir Feinberg, Mohamed Elhawaty, Nimesh Ghelani, Dan Horgan, Helen Miller, Lexi Walker, Richard Tanburn, Mukarram Tariq, Disha Shrivastava, Fei Xia, Chung-Cheng Chiu, Zoe Ashwood, Khuslen Baatarsukh, Sina Samangooei, Fred Alcober, Axel Stjerngren, Paul Komarek, Katerina Tsihlas, Anudhyan Boral, Ramona Comanescu, Jeremy Chen, Ruibo Liu, Dawn Bloxwich, Charlie Chen, Yanhua Sun, Fangxiaoyu Feng, Matthew Mauger, Xerxes Dotiwalla, Vincent Hellendoorn, Michael Sharman, Ivy Zheng, Krishna Haridasan, Gabe Barth-Maron, Craig Swanson, Dominika Rogozińska, Alek Andreev, Paul Kishan Rubenstein, Ruoxin Sang, Dan Hurt, Gamaleldin Elsayed, Renshen Wang, Dave Lacey, Anastasija Ilić, Yao Zhao, Lora Aroyo, Chimezie Iwuanyanwu, Vitaly Nikolaev, Balaji Lakshminarayanan, Sadegh Jazayeri, Raphaël Lopez Kaufman, Mani Varadarajan, Chetan Tekur, Doug Fritz, Misha Khalman, David Reitter, Kingshuk Dasgupta, Shourya Sarcar, Tina Ornduff, Javier Snaider, Fantine Huot, Johnson Jia, Rupert Kemp, Nejc Trdin, Anitha Vijayakumar, Lucy Kim, Christof Angermueller, Li Lao, Tianqi Liu, Haibin Zhang, David Engel, Somer Greene, Anaïs White, Jessica Austin, Lilly Taylor, Shereen Ashraf, Dangyi Liu, Maria Georgaki, Irene Cai, Yana Kulizhskaya, Sonam Goenka, Brennan Saeta, Kiran Vodrahalli, Christian Frank, Dario de Cesare, Brona Robenek, Harry Richardson, Mahmoud Alnahlawi, Christopher Yew, Priya Ponnapalli, Marco Tagliasacchi, Alex Korchemniy, Yelin Kim, Dinghua Li, Bill Rosgen, Zoe Ashwood, Kyle Levin, Jeremy Wiesner, Praseem Banzal, Praveen Srinivasan, Hongkun Yu, Çağlar Ünlü, David Reid, Zora Tung, Daniel Finchelstein, Ravin Kumar, Andre Elisseeff, Jin Huang, Ming Zhang, Rui Zhu, Ricardo Aguilar, Mai Giménez, Jiawei Xia, Olivier Dousse, Willi Gierke, Soheil Hassas Yeganeh, Damion Yates, Komal Jalan, Lu Li, Eri Latorre-Chimoto, Duc Dung Nguyen, Ken Durden, Praveen Kallakuri, Yaxin Liu, Matthew Johnson, Tomy Tsai, Alice Talbert, Jasmine Liu, Alexander Neitz, Chen Elkind, Marco Selvi, Mimi Jasarevic, Livio Baldini Soares, Albert Cui, Pidong Wang, Alek Wenjiao Wang, Xinyu Ye, Krystal Kallarackal, Lucia Loher, Hoi Lam, Josef Broder, Dan Holtmann-Rice, Nina Martin, Bramandia Ramadhana, Daniel Toyama, Mrinal Shukla, Sujoy Basu, Abhi Mohan, Nick Fernando, Noah Fiedel, Kim Paterson, Hui Li, Ankush Garg, Jane Park, DongHyun Choi, Diane Wu, Sankalp Singh, Zhishuai Zhang, Amir Globerson, Lily Yu, John Carpenter, Félix de Chaumont Quitry, Carey Radebaugh, Chu-Cheng Lin, Alex Tudor, Prakash Shroff, Drew Garmon, Dayou Du, Neera Vats, Han Lu, Shariq Iqbal, Alex Yakubovich, Nilesh Tripuraneni, James Manyika, Haroon Qureshi, Nan Hua, Christel Ngani, Maria Abi Raad, Hannah Forbes, Anna Bulanova, Jeff Stanway, Mukund Sundararajan, Victor Ungureanu, Colton Bishop, Yunjie Li, Balaji Venkatraman, Bo Li, Chloe Thornton, Salvatore Scellato, Nishesh Gupta, Yicheng Wang, Ian Tenney, Xihui Wu, Ashish Shenoy, Gabriel Carvajal, Diana Gage Wright, Ben Bariach, Zhuyun Xiao, Peter Hawkins, Sid Dalmia, Clement Farabet, Pedro Valenzuela, Quan Yuan, Chris Welty, Ananth Agarwal, Mia Chen, Wooyeol Kim, Brice Hulse, Nandita Dukkipati, Adam Paszke, Andrew Bolt, Elnaz Davoodi, Kiam Choo, Jennifer Beattie, Jennifer Prendki, Harsha Vashisht, Rebeca Santamaria-Fernandez, Luis C. Cobo, Jarek Wilkiewicz, David Madras, Ali Elqursh, Grant Uy, Kevin Ramirez, Matt Harvey, Tyler Liechty, Heiga Zen, Jeff Seibert, Clara Huiyi Hu, Mohamed Elhawaty, Andrey Khorlin, Maigo Le, Asaf Aharoni, Megan Li, Lily Wang, Sandeep Kumar, Alejandro Lince, Norman Casagrande, Jay Hoover, Dalia El Badawy, David Soergel, Denis Vnukov, Matt Miecnikowski, Jiri Simsa, Anna Koop, Praveen Kumar, Thibault Sellam, Daniel Vlasic, Samira Daruki, Nir Shabat, John Zhang, Guolong Su, Jiageng Zhang, Jeremiah Liu, Yi Sun, Evan Palmer, Alireza Ghaffarkhah, Xi Xiong, Victor Cotruta, Michael Fink, Lucas Dixon, Ashwin Sreevatsa, Adrian Goedeckemeyer, Alek Dimitriev, Mohsen Jafari, Remi Crocker, Nicholas FitzGerald, Aviral Kumar, Sanjay Ghemawat, Ivan Philips, Frederick Liu, Yannie Liang, Rachel Sterneck, Alena Repina, Marcus Wu, Laura Knight, Marin Georgiev, Hyo Lee, Harry Askham, Abhishek Chakladar, Annie Louis, Carl Crous, Hardie Cate, Dessie Petrova, Michael Quinn, Denese Owusu-Afriyie, Achintya Singhal, Nan Wei, Solomon Kim, Damien Vincent, Milad Nasr, Christopher A. Choquette-Choo, Reiko Tojo, Shawn Lu, Diego de Las Casas, Yuchung Cheng, Tolga Bolukbasi, Katherine Lee, Saaber Fatehi, Rajagopal Ananthanarayanan, Miteyan Patel, Charbel Kaed, Jing Li, Jakub Sygnowski, Shreyas Rammohan Belle, Zhe Chen, Jaclyn Konzelmann, Siim Põder, Roopal Garg, Vinod Koverkathu, Adam Brown, Chris Dyer, Rosanne Liu, Azade Nova, Jun Xu, Slav Petrov, Demis Hassabis, Koray Kavukcuoglu, Jeffrey Dean, and Oriol Vinyals. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv:2403.05530 [cs.CL] - Ren et al. (2023a) Ruiyang Ren, Yuhao Wang, Yingqi Qu, Wayne Xin Zhao, Jing Liu, Hao Tian, Hua Wu, Ji-Rong Wen, and Haifeng Wang. 2023a. Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval Augmentation. ArXiv preprint abs/2307.11019 (2023). https://arxiv.org/abs/2307.11019 - Ren et al. (2023b) Ruiyang Ren, Yuhao Wang, Yingqi Qu, Wayne Xin Zhao, Jing Liu, Hao Tian, Hua Wu, Ji-Rong Wen, and Haifeng Wang. 2023b. Investigating the factual knowledge boundary of large language models with retrieval augmentation. ArXiv preprint abs/2307.11019 (2023). https://arxiv.org/abs/2307.11019 - Reuters (2023) Reuters. 2023. U.S. Copyright Office says some AI-assisted works may be copyrighted. https://www.reuters.com/world/us/us-copyright-office-says-some-ai-assisted-works-may-be-copyrighted-2023-03-15/ - Rimsky (2023a) Nina Rimsky. 2023a. Modulating sycophancy in an RLHF model via activation steering. https://www.alignmentforum.org/posts/zt6hRsDE84HeBKh7E/reducing-sycophancy-and-improving-honesty-via-activation - Rimsky (2023b) Nina Rimsky. 2023b. Reducing sycophancy and improving honesty via activation steering. https://www.alignmentforum.org/posts/zt6hRsDE84HeBKh7E/reducing-sycophancy-and-improving-honesty-via-activation - Sadasivan et al. (2023) Vinu Sankar Sadasivan, Aounon Kumar, Sriram Balasubramanian, Wenxiao Wang, and Soheil Feizi. 2023. Can AI-Generated Text be Reliably Detected? CoRR abs/2303.11156 (2023). https://doi.org/10.48550/ARXIV.2303.11156 arXiv:2303.11156 - Santhanam et al. (2021) Sashank Santhanam, Behnam Hedayatnia, Spandana Gella, Aishwarya Padmakumar, Seokhwan Kim, Yang Liu, and Dilek Hakkani-Tur. 2021. Rome was built in 1776: A case study on factual correctness in knowledge-grounded response generation. ArXiv preprint abs/2110.05456 (2021). https://arxiv.org/abs/2110.05456 - Sarthi et al. (2024) Parth Sarthi, Salman Abdullah, Aditi Tuli, Shubh Khanna, Anna Goldie, and Christopher D. Manning. 2024. RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval. CoRR abs/2401.18059 (2024). https://doi.org/10.48550/ARXIV.2401.18059 arXiv:2401.18059 - Saunders et al. (2022) William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. 2022. Self-critiquing models for assisting human evaluators. ArXiv preprint abs/2206.05802 (2022). https://arxiv.org/abs/2206.05802 - Schulman (2023) John Schulman. 2023. Reinforcement Learning from Human Feedback: Progress and Challenges. Berkeley EECS. https://www.youtube.com/watch?v=hhiLw5Q_UFg - Schulman et al. (2017) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal Policy Optimization Algorithms. ArXiv preprint abs/1707.06347 (2017). https://arxiv.org/abs/1707.06347 - Scialom et al. (2021) Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano, Alex Wang, and Patrick Gallinari. 2021. QuestEval: Summarization Asks for Fact-based Evaluation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 6594–6604. https://doi.org/10.18653/v1/2021.emnlp-main.529 - Shao et al. (2024) Yijia Shao, Yucheng Jiang, Theodore A. Kanell, Peter Xu, Omar Khattab, and Monica S. Lam. 2024. Assisting in Writing Wikipedia-like Articles From Scratch with Large Language Models. CoRR abs/2402.14207 (2024). https://doi.org/10.48550/ARXIV.2402.14207 arXiv:2402.14207 - Shao et al. (2023) Zhihong Shao, Yeyun Gong, Yelong Shen, Minlie Huang, Nan Duan, and Weizhu Chen. 2023. Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergy. ArXiv preprint abs/2305.15294 (2023). https://arxiv.org/abs/2305.15294 - Sharma et al. (2023) Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R. Bowman, Newton Cheng, Esin Durmus, Zac Hatfield-Dodds, Scott R. Johnston, Shauna Kravec, Timothy Maxwell, Sam McCandlish, Kamal Ndousse, Oliver Rausch, Nicholas Schiefer, Da Yan, Miranda Zhang, and Ethan Perez. 2023. Towards Understanding Sycophancy in Language Models. ArXiv preprint abs/2310.13548 (2023). https://arxiv.org/abs/2310.13548 - Shi et al. (2023a) Weijia Shi, Xiaochuang Han, Mike Lewis, Yulia Tsvetkov, Luke Zettlemoyer, and Scott Wen-tau Yih. 2023a. Trusting Your Evidence: Hallucinate Less with Context-aware Decoding. ArXiv preprint abs/2305.14739 (2023). https://arxiv.org/abs/2305.14739 - Shi et al. (2023b) Weijia Shi, Sewon Min, Maria Lomeli, Chunting Zhou, Margaret Li, Xi Victoria Lin, Noah A. Smith, Luke Zettlemoyer, Scott Yih, and Mike Lewis. 2023b. In-Context Pretraining: Language Modeling Beyond Document Boundaries. ArXiv preprint abs/2310.10638 (2023). https://arxiv.org/abs/2310.10638 - Shi et al. (2023c) Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. 2023c. REPLUG: Retrieval-Augmented Black-Box Language Models. CoRR abs/2301.12652 (2023). https://doi.org/10.48550/ARXIV.2301.12652 arXiv:2301.12652 - Shuster et al. (2021) Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. 2021. Retrieval Augmentation Reduces Hallucination in Conversation. In Findings of the Association for Computational Linguistics: EMNLP 2021. Association for Computational Linguistics, Punta Cana, Dominican Republic, 3784–3803. https://doi.org/10.18653/v1/2021.findings-emnlp.320 - Singhal et al. (2023) Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Le Hou, Kevin Clark, Stephen Pfohl, Heather Cole-Lewis, Darlene Neal, Mike Schaekermann, Amy Wang, Mohamed Amin, Sami Lachgar, Philip Andrew Mansfield, Sushant Prakash, Bradley Green, Ewa Dominowska, Blaise Agüera y Arcas, Nenad Tomasev, Yun Liu, Renee Wong, Christopher Semturs, S. Sara Mahdavi, Joelle K. Barral, Dale R. Webster, Gregory S. Corrado, Yossi Matias, Shekoofeh Azizi, Alan Karthikesalingam, and Vivek Natarajan. 2023. Towards Expert-Level Medical Question Answering with Large Language Models. ArXiv preprint abs/2305.09617 (2023). https://arxiv.org/abs/2305.09617 - Sinitsin et al. (2020) Anton Sinitsin, Vsevolod Plokhotnyuk, Dmitriy Pyrkin, Sergei Popov, and Artem Babenko. 2020. Editable Neural Networks. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net. https://openreview.net/forum?id=HJedXaEtvS - Slobodkin et al. (2023) Aviv Slobodkin, Omer Goldman, Avi Caciularu, Ido Dagan, and Shauli Ravfogel. 2023. The Curious Case of Hallucinatory Unanswerablity: Finding Truths in the Hidden States of Over-Confident Large Language Models. ArXiv preprint abs/2310.11877 (2023). https://arxiv.org/abs/2310.11877 - Slobodkin et al. (2024) Aviv Slobodkin, Eran Hirsch, Arie Cattan, Tal Schuster, and Ido Dagan. 2024. Attribute First, then Generate: Locally-attributable Grounded Text Generation. CoRR abs/2403.17104 (2024). https://doi.org/10.48550/ARXIV.2403.17104 arXiv:2403.17104 - Stahlberg and Byrne (2019) Felix Stahlberg and Bill Byrne. 2019. On NMT Search Errors and Model Errors: Cat Got Your Tongue?. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). Association for Computational Linguistics, Hong Kong, China, 3356–3362. https://doi.org/10.18653/v1/D19-1331 - Stelmakh et al. (2022) Ivan Stelmakh, Yi Luan, Bhuwan Dhingra, and Ming-Wei Chang. 2022. ASQA: Factoid Questions Meet Long-Form Answers. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Abu Dhabi, United Arab Emirates, 8273–8288. https://aclanthology.org/2022.emnlp-main.566 - Stiennon et al. (2020) Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F. Christiano. 2020. Learning to summarize with human feedback. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (Eds.). https://proceedings.neurips.cc/paper/2020/hash/1f89885d556929e98d3ef9b86448f951-Abstract.html - Su et al. (2024b) Hongjin Su, Howard Yen, Mengzhou Xia, Weijia Shi, Niklas Muennighoff, Han yu Wang, Haisu Liu, Quan Shi, Zachary S. Siegel, Michael Tang, Ruoxi Sun, Jinsung Yoon, Sercan O. Arik, Danqi Chen, and Tao Yu. 2024b. BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrieval. arXiv:2407.12883 [cs.CL] https://arxiv.org/abs/2407.12883 - Su et al. (2021) Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. 2021. RoFormer: Enhanced Transformer with Rotary Position Embedding. CoRR abs/2104.09864 (2021). arXiv:2104.09864 https://arxiv.org/abs/2104.09864 - Su et al. (2024a) Weihang Su, Yichen Tang, Qingyao Ai, Zhijing Wu, and Yiqun Liu. 2024a. DRAGIN: Dynamic Retrieval Augmented Generation based on the Real-time Information Needs of Large Language Models. CoRR abs/2403.10081 (2024). https://doi.org/10.48550/ARXIV.2403.10081 arXiv:2403.10081 - Subramani et al. (2022) Nishant Subramani, Nivedita Suresh, and Matthew Peters. 2022. Extracting Latent Steering Vectors from Pretrained Language Models. In Findings of the Association for Computational Linguistics: ACL 2022. Association for Computational Linguistics, Dublin, Ireland, 566–581. https://doi.org/10.18653/v1/2022.findings-acl.48 - Sun et al. (2023) Kai Sun, Yifan Ethan Xu, Hanwen Zha, Yue Liu, and Xin Luna Dong. 2023. Head-to-Tail: How Knowledgeable are Large Language Models (LLM)? A.K.A. Will LLMs Replace Knowledge Graphs? CoRR abs/2308.10168 (2023). https://doi.org/10.48550/ARXIV.2308.10168 arXiv:2308.10168 - Sutskever (2023) Ilya Sutskever. 2023. An observation on Generalization. Youtube. https://www.youtube.com/watch?v=AKMuA_TVz3A&t=5s - Tan et al. (2024b) Chenmien Tan, Ge Zhang, and Jie Fu. 2024b. Massive Editing for Large Language Model via Meta Learning. In The Twelfth International Conference on Learning Representations. https://openreview.net/forum?id=L6L1CJQ2PE - Tan et al. (2024a) Hexiang Tan, Fei Sun, Wanli Yang, Yuanzhuo Wang, Qi Cao, and Xueqi Cheng. 2024a. Blinded by Generated Contexts: How Language Models Merge Generated and Retrieved Contexts for Open-Domain QA? CoRR abs/2401.11911 (2024). https://doi.org/10.48550/ARXIV.2401.11911 arXiv:2401.11911 - Tang et al. (2023) Raphael Tang, Xinyu Zhang, Xueguang Ma, Jimmy Lin, and Ferhan Ture. 2023. Found in the Middle: Permutation Self-Consistency Improves Listwise Ranking in Large Language Models. CoRR abs/2310.07712 (2023). https://doi.org/10.48550/ARXIV.2310.07712 arXiv:2310.07712 - Thakur et al. (2021) Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021. BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models. CoRR abs/2104.08663 (2021). arXiv:2104.08663 https://arxiv.org/abs/2104.08663 - Tian et al. (2019) Ran Tian, Shashi Narayan, Thibault Sellam, and Ankur P Parikh. 2019. Sticking to the facts: Confident decoding for faithful data-to-text generation. ArXiv preprint abs/1910.08684 (2019). https://arxiv.org/abs/1910.08684 - Tong et al. (2024) Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. 2024. Eyes wide shut? exploring the visual shortcomings of multimodal llms. arXiv preprint arXiv:2401.06209 (2024). - Tonmoy et al. (2024) S. M. Towhidul Islam Tonmoy, S. M. Mehedi Zaman, Vinija Jain, Anku Rani, Vipula Rawte, Aman Chadha, and Amitava Das. 2024. A Comprehensive Survey of Hallucination Mitigation Techniques in Large Language Models. CoRR abs/2401.01313 (2024). https://doi.org/10.48550/ARXIV.2401.01313 arXiv:2401.01313 - Touvron et al. (2023a) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023a. LLaMA: Open and Efficient Foundation Language Models. CoRR abs/2302.13971 (2023). https://doi.org/10.48550/ARXIV.2302.13971 arXiv:2302.13971 - Touvron et al. (2023b) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023b. Llama 2: Open Foundation and Fine-Tuned Chat Models. ArXiv preprint abs/2307.09288 (2023). https://arxiv.org/abs/2307.09288 - Trivedi et al. (2023) Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2023. Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (Eds.). Association for Computational Linguistics, 10014–10037. https://aclanthology.org/2023.acl-long.557 - Turpin et al. (2023) Miles Turpin, Julian Michael, Ethan Perez, and Samuel R. Bowman. 2023. Language Models Don’t Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting. ArXiv preprint abs/2305.04388 (2023). https://arxiv.org/abs/2305.04388 - Umapathi et al. (2023) Logesh Kumar Umapathi, Ankit Pal, and Malaikannan Sankarasubbu. 2023. Med-halt: Medical domain hallucination test for large language models. ArXiv preprint abs/2307.15343 (2023). https://arxiv.org/abs/2307.15343 - van den Oord et al. (2018) Aäron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation Learning with Contrastive Predictive Coding. CoRR abs/1807.03748 (2018). arXiv:1807.03748 http://arxiv.org/abs/1807.03748 - van der Poel et al. (2022) Liam van der Poel, Ryan Cotterell, and Clara Meister. 2022. Mutual Information Alleviates Hallucinations in Abstractive Summarization. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Abu Dhabi, United Arab Emirates, 5956–5965. https://aclanthology.org/2022.emnlp-main.399 - Varshney et al. (2023) Neeraj Varshney, Wenlin Yao, Hongming Zhang, Jianshu Chen, and Dong Yu. 2023. A Stitch in Time Saves Nine: Detecting and Mitigating Hallucinations of LLMs by Validating Low-Confidence Generation. ArXiv preprint abs/2307.03987 (2023). https://arxiv.org/abs/2307.03987 - Voita et al. (2019) Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. 2019. Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Florence, Italy, 5797–5808. https://doi.org/10.18653/v1/P19-1580 - Vu et al. (2023) Tu Vu, Mohit Iyyer, Xuezhi Wang, Noah Constant, Jerry Wei, Jason Wei, Chris Tar, Yun-Hsuan Sung, Denny Zhou, Quoc Le, and Thang Luong. 2023. FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation. arXiv:2310.03214 [cs.CL] - Wan et al. (2023) David Wan, Mengwen Liu, Kathleen McKeown, Markus Dreyer, and Mohit Bansal. 2023. Faithfulness-Aware Decoding Strategies for Abstractive Summarization. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics. Association for Computational Linguistics, Dubrovnik, Croatia, 2864–2880. https://aclanthology.org/2023.eacl-main.210 - Wang et al. (2020a) Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020a. Asking and Answering Questions to Evaluate the Factual Consistency of Summaries. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Online, 5008–5020. https://doi.org/10.18653/v1/2020.acl-main.450 - Wang et al. (2023b) Binjie Wang, Ethan Chern, and Pengfei Liu. 2023b. ChineseFactEval: A Factuality Benchmark for Chinese LLMs. - Wang et al. (2023e) Cunxiang Wang, Xiaoze Liu, Yuanhao Yue, Xiangru Tang, Tianhang Zhang, Jiayang Cheng, Yunzhi Yao, Wenyang Gao, Xuming Hu, Zehan Qi, Yidong Wang, Linyi Yang, Jindong Wang, Xing Xie, Zheng Zhang, and Yue Zhang. 2023e. Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificity. ArXiv preprint abs/2310.07521 (2023). https://arxiv.org/abs/2310.07521 - Wang and Sennrich (2020) Chaojun Wang and Rico Sennrich. 2020. On Exposure Bias, Hallucination and Domain Shift in Neural Machine Translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Online, 3544–3552. https://doi.org/10.18653/v1/2020.acl-main.326 - Wang et al. (2023d) Jiaan Wang, Yunlong Liang, Fandong Meng, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023d. Is chatgpt a good nlg evaluator? a preliminary study. ArXiv preprint abs/2303.04048 (2023). https://arxiv.org/abs/2303.04048 - Wang et al. (2023g) Junyang Wang, Yuhang Wang, Guohai Xu, Jing Zhang, Yukai Gu, Haitao Jia, Ming Yan, Ji Zhang, and Jitao Sang. 2023g. An llm-free multi-dimensional benchmark for mllms hallucination evaluation. arXiv preprint arXiv:2311.07397 (2023). - Wang et al. (2024a) Lei Wang, Jiabang He, Shenshen Li, Ning Liu, and Ee-Peng Lim. 2024a. Mitigating fine-grained hallucination by fine-tuning large vision-language models with caption rewrites. In International Conference on Multimedia Modeling. Springer, 32–45. - Wang et al. (2023h) Liang Wang, Nan Yang, and Furu Wei. 2023h. Query2doc: Query Expansion with Large Language Models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, 9414–9423. https://doi.org/10.18653/V1/2023.EMNLP-MAIN.585 - Wang et al. (2023f) Peifeng Wang, Zhengyang Wang, Zheng Li, Yifan Gao, Bing Yin, and Xiang Ren. 2023f. SCOTT: Self-consistent chain-of-thought distillation. ArXiv preprint abs/2305.01879 (2023). https://arxiv.org/abs/2305.01879 - Wang et al. (2024b) Shuting Wang, Xin Yu, Mang Wang, Weipeng Chen, Yutao Zhu, and Zhicheng Dou. 2024b. RichRAG: Crafting Rich Responses for Multi-faceted Queries in Retrieval-Augmented Generation. CoRR abs/2406.12566 (2024). https://doi.org/10.48550/ARXIV.2406.12566 arXiv:2406.12566 - Wang et al. (2023j) Song Wang, Yaochen Zhu, Haochen Liu, Zaiyi Zheng, Chen Chen, and Jundong Li. 2023j. Knowledge Editing for Large Language Models: A Survey. arXiv:2310.16218 [cs.CL] - Wang et al. (2023c) Yile Wang, Peng Li, Maosong Sun, and Yang Liu. 2023c. Self-Knowledge Guided Retrieval Augmentation for Large Language Models. In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, 10303–10315. https://doi.org/10.18653/V1/2023.FINDINGS-EMNLP.691 - Wang et al. (2023i) Yufei Wang, Wanjun Zhong, Liangyou Li, Fei Mi, Xingshan Zeng, Wenyong Huang, Lifeng Shang, Xin Jiang, and Qun Liu. 2023i. Aligning Large Language Models with Human: A Survey. CoRR abs/2307.12966 (2023). https://doi.org/10.48550/ARXIV.2307.12966 arXiv:2307.12966 - Wang et al. (2023a) Zhiruo Wang, Jun Araki, Zhengbao Jiang, Md. Rizwan Parvez, and Graham Neubig. 2023a. Learning to Filter Context for Retrieval-Augmented Generation. CoRR abs/2311.08377 (2023). https://doi.org/10.48550/ARXIV.2311.08377 arXiv:2311.08377 - Wang et al. (2020b) Zhenyi Wang, Xiaoyang Wang, Bang An, Dong Yu, and Changyou Chen. 2020b. Towards Faithful Neural Table-to-Text Generation with Content-Matching Constraints. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Online, 1072–1086. https://doi.org/10.18653/v1/2020.acl-main.101 - Wang et al. (2022) Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. 2022. SimVLM: Simple Visual Language Model Pretraining with Weak Supervision. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net. https://openreview.net/forum?id=GUrhfTuf_3 - Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems 35 (2022), 24824–24837. - Wei et al. (2023) Jerry W. Wei, Da Huang, Yifeng Lu, Denny Zhou, and Quoc V. Le. 2023. Simple synthetic data reduces sycophancy in large language models. ArXiv preprint abs/2308.03958 (2023). https://arxiv.org/abs/2308.03958 - Weidinger et al. (2021) Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, Zac Kenton, Sasha Brown, Will Hawkins, Tom Stepleton, Courtney Biles, Abeba Birhane, Julia Haas, Laura Rimell, Lisa Anne Hendricks, William Isaac, Sean Legassick, Geoffrey Irving, and Iason Gabriel. 2021. Ethical and social risks of harm from Language Models. ArXiv preprint abs/2112.04359 (2021). https://arxiv.org/abs/2112.04359 - Wen et al. (2023) Yilin Wen, Zifeng Wang, and Jimeng Sun. 2023. MindMap: Knowledge Graph Prompting Sparks Graph of Thoughts in Large Language Models. ArXiv preprint abs/2308.09729 (2023). https://arxiv.org/abs/2308.09729 - Wu et al. (2024a) Di Wu, Jia-Chen Gu, Fan Yin, Nanyun Peng, and Kai-Wei Chang. 2024a. Synchronous Faithfulness Monitoring for Trustworthy Retrieval-Augmented Generation. CoRR abs/2406.13692 (2024). https://doi.org/10.48550/ARXIV.2406.13692 arXiv:2406.13692 - Wu et al. (2024b) Kevin Wu, Eric Wu, and James Zou. 2024b. ClashEval: Quantifying the tug-of-war between an LLM’s internal prior and external evidence. arXiv:2404.10198 [cs.CL] https://arxiv.org/abs/2404.10198 - Xiao et al. (2023) Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighof. 2023. C-Pack: Packaged Resources To Advance General Chinese Embedding. CoRR abs/2309.07597 (2023). https://doi.org/10.48550/ARXIV.2309.07597 arXiv:2309.07597 - Xiao and Wang (2021) Yijun Xiao and William Yang Wang. 2021. On Hallucination and Predictive Uncertainty in Conditional Language Generation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume. Association for Computational Linguistics, Online, 2734–2744. https://doi.org/10.18653/v1/2021.eacl-main.236 - Xie et al. (2023) Jian Xie, Kai Zhang, Jiangjie Chen, Renze Lou, and Yu Su. 2023. Adaptive Chameleon or Stubborn Sloth: Unraveling the Behavior of Large Language Models in Knowledge Clashes. ArXiv preprint abs/2305.13300 (2023). https://arxiv.org/abs/2305.13300 - Xiong et al. (2023) Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. 2023. Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs. ArXiv preprint abs/2306.13063 (2023). https://arxiv.org/abs/2306.13063 - Xu et al. (2023b) Fangyuan Xu, Weijia Shi, and Eunsol Choi. 2023b. RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentation. ArXiv preprint abs/2310.04408 (2023). https://arxiv.org/abs/2310.04408 - Xu et al. (2020) Jiacheng Xu, Shrey Desai, and Greg Durrett. 2020. Understanding Neural Abstractive Summarization Models via Uncertainty. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Online, 6275–6281. https://doi.org/10.18653/v1/2020.emnlp-main.508 - Xu et al. (2024) Jundong Xu, Hao Fei, Liangming Pan, Qian Liu, Mong-Li Lee, and Wynne Hsu. 2024. Faithful Logical Reasoning via Symbolic Chain-of-Thought. CoRR abs/2405.18357 (2024). https://doi.org/10.48550/ARXIV.2405.18357 arXiv:2405.18357 - Xu et al. (2023a) Shicheng Xu, Danyang Hou, Liang Pang, Jingcheng Deng, Jun Xu, Huawei Shen, and Xueqi Cheng. 2023a. AI-Generated Images Introduce Invisible Relevance Bias to Text-Image Retrieval. CoRR abs/2311.14084 (2023). https://doi.org/10.48550/ARXIV.2311.14084 arXiv:2311.14084 - Yang et al. (2023b) Shiping Yang, Renliang Sun, and Xiaojun Wan. 2023b. A New Benchmark and Reverse Validation Method for Passage-level Hallucination Detection. ArXiv preprint abs/2310.06498 (2023). https://arxiv.org/abs/2310.06498 - Yang et al. (2023a) Yuqing Yang, Ethan Chern, Xipeng Qiu, Graham Neubig, and Pengfei Liu. 2023a. Alignment for Honesty. CoRR abs/2312.07000 (2023). https://doi.org/10.48550/ARXIV.2312.07000 arXiv:2312.07000 - Yang et al. (2018a) Zhilin Yang, Zihang Dai, Ruslan Salakhutdinov, and William W. Cohen. 2018a. Breaking the Softmax Bottleneck: A High-Rank RNN Language Model. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net. https://openreview.net/forum?id=HkwZSG-CZ - Yang et al. (2019) Zhilin Yang, Thang Luong, Ruslan Salakhutdinov, and Quoc V. Le. 2019. Mixtape: Breaking the Softmax Bottleneck Efficiently. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d’Alché-Buc, Emily B. Fox, and Roman Garnett (Eds.). 15922–15930. https://proceedings.neurips.cc/paper/2019/hash/512fc3c5227f637e41437c999a2d3169-Abstract.html - Yang et al. (2018b) Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018b. HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Brussels, Belgium, 2369–2380. https://doi.org/10.18653/v1/D18-1259 - Yao et al. (2023a) Jia-Yu Yao, Kun-Peng Ning, Zhen-Hui Liu, Mu-Nan Ning, and Li Yuan. 2023a. LLM Lies: Hallucinations are not Bugs, but Features as Adversarial Examples. ArXiv preprint abs/2310.01469 (2023). https://arxiv.org/abs/2310.01469 - Yao et al. (2022) Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. ArXiv preprint abs/2210.03629 (2022). https://arxiv.org/abs/2210.03629 - Yao et al. (2023b) Yunzhi Yao, Peng Wang, Bozhong Tian, Siyuan Cheng, Zhoubo Li, Shumin Deng, Huajun Chen, and Ningyu Zhang. 2023b. Editing Large Language Models: Problems, Methods, and Opportunities. ArXiv preprint abs/2305.13172 (2023). https://arxiv.org/abs/2305.13172 - Ye et al. (2023) Xi Ye, Ruoxi Sun, Sercan Ö. Arik, and Tomas Pfister. 2023. Effective Large Language Model Adaptation for Improved Grounding. CoRR abs/2311.09533 (2023). https://doi.org/10.48550/ARXIV.2311.09533 arXiv:2311.09533 - Yin et al. (2023a) Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang, Dianbo Sui, Yunhang Shen, Ke Li, Xing Sun, and Enhong Chen. 2023a. Woodpecker: Hallucination Correction for Multimodal Large Language Models. https://arxiv.org/abs/2310.16045 - Yin et al. (2023b) Zhangyue Yin, Qiushi Sun, Qipeng Guo, Jiawen Wu, Xipeng Qiu, and Xuanjing Huang. 2023b. Do Large Language Models Know What They Don’t Know?. In Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023, Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (Eds.). Association for Computational Linguistics, 8653–8665. https://doi.org/10.18653/V1/2023.FINDINGS-ACL.551 - Yoon et al. (2024) Chanwoong Yoon, Gangwoo Kim, Byeongguk Jeon, Sungdong Kim, Yohan Jo, and Jaewoo Kang. 2024. Ask Optimal Questions: Aligning Large Language Models with Retriever’s Preference in Conversational Search. CoRR abs/2402.11827 (2024). https://doi.org/10.48550/ARXIV.2402.11827 arXiv:2402.11827 - Yoran et al. (2023) Ori Yoran, Tomer Wolfson, Ori Ram, and Jonathan Berant. 2023. Making Retrieval-Augmented Language Models Robust to Irrelevant Context. CoRR abs/2310.01558 (2023). https://doi.org/10.48550/ARXIV.2310.01558 arXiv:2310.01558 - Yu et al. (2022) Fangyi Yu, Lee Quartey, and Frank Schilder. 2022. Legal Prompting: Teaching a Language Model to Think Like a Lawyer. ArXiv preprint abs/2212.01326 (2022). https://arxiv.org/abs/2212.01326 - Yu et al. (2023b) Fei Yu, Hongbo Zhang, and Benyou Wang. 2023b. Nature language reasoning, a survey. ArXiv preprint abs/2303.14725 (2023). https://arxiv.org/abs/2303.14725 - Yu et al. (2021) Weijiang Yu, Jian Liang, Lei Ji, Lu Li, Yuejian Fang, Nong Xiao, and Nan Duan. 2021. Hybrid reasoning network for video-based commonsense captioning. In Proceedings of the 29th ACM international conference on multimedia. 5213–5221. - Yu et al. (2023a) Weijiang Yu, Haofan Wang, Guohao Li, Nong Xiao, and Bernard Ghanem. 2023a. Knowledge-aware Global Reasoning for Situation Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence (2023). - Yu et al. (2023d) Wenhao Yu, Hongming Zhang, Xiaoman Pan, Kaixin Ma, Hongwei Wang, and Dong Yu. 2023d. Chain-of-Note: Enhancing Robustness in Retrieval-Augmented Language Models. CoRR abs/2311.09210 (2023). https://doi.org/10.48550/ARXIV.2311.09210 arXiv:2311.09210 - Yu et al. (2023c) Wenhao Yu, Zhihan Zhang, Zhenwen Liang, Meng Jiang, and Ashish Sabharwal. 2023c. Improving Language Models via Plug-and-Play Retrieval Feedback. ArXiv preprint abs/2305.14002 (2023). https://arxiv.org/abs/2305.14002 - Yuan et al. (2021) Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. BARTScore: Evaluating Generated Text as Text Generation. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, Marc’Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (Eds.). 27263–27277. https://proceedings.neurips.cc/paper/2021/hash/e4d2b6e6fdeca3e60e0f1a62fee3d9dd-Abstract.html - Zellers et al. (2019) Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. From Recognition to Cognition: Visual Commonsense Reasoning. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019. Computer Vision Foundation / IEEE, 6720–6731. https://doi.org/10.1109/CVPR.2019.00688 - Zhai et al. (2023) Bohan Zhai, Shijia Yang, Chenfeng Xu, Sheng Shen, Kurt Keutzer, and Manling Li. 2023. Halle-switch: Controlling object hallucination in large vision language models. arXiv e-prints (2023), arXiv–2310. - Zhang et al. (2023a) Hanning Zhang, Shizhe Diao, Yong Lin, Yi R. Fung, Qing Lian, Xingyao Wang, Yangyi Chen, Heng Ji, and Tong Zhang. 2023a. R-Tuning: Teaching Large Language Models to Refuse Unknown Questions. CoRR abs/2311.09677 (2023). https://doi.org/10.48550/ARXIV.2311.09677 arXiv:2311.09677 - Zhang et al. (2021) Hugh Zhang, Daniel Duckworth, Daphne Ippolito, and Arvind Neelakantan. 2021. Trading Off Diversity and Quality in Natural Language Generation. In Proceedings of the Workshop on Human Evaluation of NLP Systems (HumEval). Association for Computational Linguistics, Online, 25–33. https://aclanthology.org/2021.humeval-1.3 - Zhang et al. (2023e) Jiaxin Zhang, Zhuohang Li, Kamalika Das, Bradley A. Malin, and Sricharan Kumar. 2023e. SAC 3: Reliable Hallucination Detection in Black-Box Language Models via Semantic-aware Cross-check Consistency. arXiv:2311.01740 [cs.CL] - Zhang et al. (2018) Jiajun Zhang, Yang Zhao, Haoran Li, and Chengqing Zong. 2018. Attention with sparsity regularization for neural machine translation and summarization. IEEE/ACM Transactions on Audio, Speech, and Language Processing 27, 3 (2018), 507–518. - Zhang et al. (2020) Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J. Liu. 2020. PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event (Proceedings of Machine Learning Research, Vol. 119). PMLR, 11328–11339. http://proceedings.mlr.press/v119/zhang20ae.html - Zhang et al. (2024c) Mingtian Zhang, Shawn Lan, Peter Hayes, and David Barber. 2024c. Mafin: Enhancing Black-Box Embeddings with Model Augmented Fine-Tuning. arXiv:2402.12177 [cs.LG] - Zhang et al. (2023g) Muru Zhang, Ofir Press, William Merrill, Alisa Liu, and Noah A Smith. 2023g. How language model hallucinations can snowball. ArXiv preprint abs/2305.13534 (2023). https://arxiv.org/abs/2305.13534 - Zhang et al. (2024e) Ningyu Zhang, Yunzhi Yao, Bozhong Tian, Peng Wang, Shumin Deng, Mengru Wang, Zekun Xi, Shengyu Mao, Jintian Zhang, Yuansheng Ni, Siyuan Cheng, Ziwen Xu, Xin Xu, Jia-Chen Gu, Yong Jiang, Pengjun Xie, Fei Huang, Lei Liang, Zhiqiang Zhang, Xiaowei Zhu, Jun Zhou, and Huajun Chen. 2024e. A Comprehensive Study of Knowledge Editing for Large Language Models. arXiv:2401.01286 [cs.CL] - Zhang et al. (2023b) Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, et al. 2023b. Instruction Tuning for Large Language Models: A Survey. ArXiv preprint abs/2308.10792 (2023). https://arxiv.org/abs/2308.10792 - Zhang et al. (2023f) Shuo Zhang, Liangming Pan, Junzhou Zhao, and William Yang Wang. 2023f. Mitigating Language Model Hallucination with Interactive Question-Knowledge Alignment. ArXiv preprint abs/2305.13669 (2023). https://arxiv.org/abs/2305.13669 - Zhang et al. (2022) Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona T. Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022. OPT: Open Pre-trained Transformer Language Models. ArXiv preprint abs/2205.01068 (2022). https://arxiv.org/abs/2205.01068 - Zhang et al. (2023c) Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori B Hashimoto. 2023c. Benchmarking large language models for news summarization. ArXiv preprint abs/2301.13848 (2023). https://arxiv.org/abs/2301.13848 - Zhang et al. (2023h) Tianhang Zhang, Lin Qiu, Qipeng Guo, Cheng Deng, Yue Zhang, Zheng Zhang, Chenghu Zhou, Xinbing Wang, and Luoyi Fu. 2023h. Enhancing Uncertainty-Based Hallucination Detection with Stronger Focus. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, 915–932. https://aclanthology.org/2023.emnlp-main.58 - Zhang et al. (2024d) Xiaoying Zhang, Baolin Peng, Ye Tian, Jingyan Zhou, Lifeng Jin, Linfeng Song, Haitao Mi, and Helen Meng. 2024d. Self-Alignment for Factuality: Mitigating Hallucinations in LLMs via Self-Evaluation. CoRR abs/2402.09267 (2024). https://doi.org/10.48550/ARXIV.2402.09267 arXiv:2402.09267 - Zhang et al. (2023d) Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, Longyue Wang, Anh Tuan Luu, Wei Bi, Freda Shi, and Shuming Shi. 2023d. Siren’s Song in the AI Ocean: A Survey on Hallucination in Large Language Models. ArXiv preprint abs/2309.01219 (2023). https://arxiv.org/abs/2309.01219 - Zhang et al. (2024a) Zhenyu Zhang, Runjin Chen, Shiwei Liu, Zhewei Yao, Olatunji Ruwase, Beidi Chen, Xiaoxia Wu, and Zhangyang Wang. 2024a. Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encoding. arXiv:2403.04797 [cs.CL] - Zhang et al. (2024b) Zihan Zhang, Meng Fang, and Ling Chen. 2024b. RetrievalQA: Assessing Adaptive Retrieval-Augmented Generation for Short-form Open-Domain Question Answering. arXiv:2402.16457 [cs.CL] - Zhao et al. (2024a) Linxi Zhao, Yihe Deng, Weitong Zhang, and Quanquan Gu. 2024a. Mitigating Object Hallucination in Large Vision-Language Models via Classifier-Free Guidance. arXiv preprint arXiv:2402.08680 (2024). - Zhao et al. (2023a) Liang Zhao, Xiaocheng Feng, Xiachong Feng, Bing Qin, and Ting Liu. 2023a. Length Extrapolation of Transformers: A Survey from the Perspective of Position Encoding. CoRR abs/2312.17044 (2023). https://doi.org/10.48550/ARXIV.2312.17044 arXiv:2312.17044 - Zhao et al. (2023b) Ruochen Zhao, Xingxuan Li, Shafiq Joty, Chengwei Qin, and Lidong Bing. 2023b. Verify-and-Edit: A Knowledge-Enhanced Chain-of-Thought Framework. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (Eds.). Association for Computational Linguistics, 5823–5840. https://doi.org/10.18653/v1/2023.acl-long.320 - Zhao et al. (2024b) Wayne Xin Zhao, Jing Liu, Ruiyang Ren, and Ji-Rong Wen. 2024b. Dense text retrieval based on pretrained language models: A survey. ACM Transactions on Information Systems 42, 4 (2024), 1–60. - Zhao et al. (2023d) Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023d. A survey of large language models. ArXiv preprint abs/2303.18223 (2023). https://arxiv.org/abs/2303.18223 - Zhao et al. (2023c) Yukun Zhao, Lingyong Yan, Weiwei Sun, Guoliang Xing, Chong Meng, Shuaiqiang Wang, Zhicong Cheng, Zhaochun Ren, and Dawei Yin. 2023c. Knowing What LLMs DO NOT Know: A Simple Yet Effective Self-Detection Method. ArXiv preprint abs/2310.17918 (2023). https://arxiv.org/abs/2310.17918 - Zheng et al. (2024) Danna Zheng, Mirella Lapata, and Jeff Z. Pan. 2024. Large Language Models as Reliable Knowledge Bases? arXiv:2407.13578 [cs.CL] https://arxiv.org/abs/2407.13578 - Zheng et al. (2023) Shen Zheng, Jie Huang, and Kevin Chen-Chuan Chang. 2023. Why Does ChatGPT Fall Short in Answering Questions Faithfully? ArXiv preprint abs/2304.10513 (2023). https://arxiv.org/abs/2304.10513 - Zhong et al. (2023) Weihong Zhong, Mao Zheng, Duyu Tang, Xuan Luo, Heng Gong, Xiaocheng Feng, and Bing Qin. 2023. STOA-VLP: Spatial-Temporal Modeling of Object and Action for Video-Language Pre-Training. In Proceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence and Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence and Thirteenth Symposium on Educational Advances in Artificial Intelligence, Vol. 37. 3715–3723. https://doi.org/10.1609/aaai.v37i3.25483 - Zhou et al. (2023b) Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. 2023b. Lima: Less is more for alignment. ArXiv preprint abs/2305.11206 (2023). https://arxiv.org/abs/2305.11206 - Zhou et al. (2021) Chunting Zhou, Graham Neubig, Jiatao Gu, Mona Diab, Francisco Guzmán, Luke Zettlemoyer, and Marjan Ghazvininejad. 2021. Detecting Hallucinated Content in Conditional Neural Sequence Generation. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021. Association for Computational Linguistics, Online, 1393–1404. https://doi.org/10.18653/v1/2021.findings-acl.120 - Zhou et al. (2023c) Wenxuan Zhou, Sheng Zhang, Hoifung Poon, and Muhao Chen. 2023c. Context-faithful Prompting for Large Language Models. In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, 14544–14556. https://aclanthology.org/2023.findings-emnlp.968 - Zhou et al. (2023a) Yiyang Zhou, Chenhang Cui, Jaehong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao. 2023a. Analyzing and Mitigating Object Hallucination in Large Vision-Language Models. https://arxiv.org/abs/2310.00754 - Zhu et al. (2023a) Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023a. Minigpt-4: Enhancing vision-language understanding with advanced large language models. ArXiv preprint abs/2304.10592 (2023). https://arxiv.org/abs/2304.10592 - Zhu et al. (2023b) Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Lingpeng Kong, Jiajun Chen, Lei Li, and Shujian Huang. 2023b. Multilingual machine translation with large language models: Empirical results and analysis. ArXiv preprint abs/2304.04675 (2023). https://arxiv.org/abs/2304.04675 - Zhu et al. (2023c) Yutao Zhu, Huaying Yuan, Shuting Wang, Jiongnan Liu, Wenhan Liu, Chenlong Deng, Zhicheng Dou, and Ji-Rong Wen. 2023c. Large Language Models for Information Retrieval: A Survey. CoRR abs/2308.07107 (2023). https://doi.org/10.48550/ARXIV.2308.07107 arXiv:2308.07107 - Zong et al. (2023) Yongshuo Zong, Tingyang Yu, Bingchen Zhao, Ruchika Chavhan, and Timothy Hospedales. 2023. Fool Your (Vision and) Language Model With Embarrassingly Simple Permutations. ArXiv preprint abs/2310.01651 (2023). https://arxiv.org/abs/2310.01651