2410.21360

# A Survey on Automatic Credibility Assessment Using Textual Credibility Signals in the Era of Large Language Models **Authors**: Ivan Srba, Olesya Razuvayevskaya, João A. Leite, Robert Moro, Ipek Baris Schlicht, Sara Tonelli, Francisco Moreno García, Santiago Barrio Lottmann, Denis Teyssou, Valentin Porcellini, Carolina Scarton, Kalina Bontcheva, Maria Bielikova > ivan.srba@kinit.sk0000-0003-3511-5337Kempelen Institute of Intelligent TechnologiesBratislavaSlovakia > o.razuvayevskaya@sheffield.ac.uk0000-0002-7922-7982The University of SheffieldSheffieldUK > jaleite1@sheffield.ac.uk0000-0002-3587-853XThe University of SheffieldSheffieldUK > robert.moro@kinit.sk0000-0002-3052-8290Kempelen Institute of Intelligent TechnologiesBratislavaSlovakia > ipek.baris-schlicht@dw.com0000-0002-5037-2203Deutsche WelleBonn/BerlinGermanyUniversitat Politècnica de ValènciaValenciaSpain > satonelli@fbk.eu0000-0001-8010-6689Fondazione Bruno KesslerTrentoItaly > fran.moreno@upm.es0000-0001-7883-4812Universidad Politécnica de MadridMadridSpain > s.barrio@upm.es0000-0003-1881-7871Universidad Politécnica de MadridMadridSpain > Denis.TEYSSOU@afp.com0000-0003-0505-6356Agence France-PresseParisFrance > Valentin.PORCELLINI@afp.com0000-0001-7909-8853Agence France-PresseParisFrance > c.scarton@sheffield.ac.uk0000-0002-0103-4072The University of SheffieldSheffieldUK > k.bontcheva@sheffield.ac.uk0000-0001-6152-9600The University of SheffieldSheffieldUK > maria.bielikova@kinit.sk0000-0003-4105-3494Kempelen Institute of Intelligent TechnologiesBratislavaSlovakia (2025) Abstract. In the age of social media and generative AI, the ability to automatically assess the credibility of online content has become increasingly critical, complementing traditional approaches to false information detection. Credibility assessment relies on aggregating diverse credibility signals – small units of information, such as content subjectivity, bias, or a presence of persuasion techniques – into a final credibility label/score. However, current research in automatic credibility assessment and credibility signals detection remains highly fragmented, with many signals studied in isolation and lacking integration. Notably, there is a scarcity of approaches that detect and aggregate multiple credibility signals simultaneously. These challenges are further exacerbated by the absence of a comprehensive and up-to-date overview of research works that connects these research efforts under a common framework and identifies shared trends, challenges, and open problems. In this survey, we address this gap by presenting a systematic and comprehensive literature review of 175 research papers, focusing on textual credibility signals within the field of Natural Language Processing (NLP), which undergoes a rapid transformation due to advancements in Large Language Models (LLMs). While positioning the NLP research into the the broader multidisciplinary landscape, we examine both automatic credibility assessment methods as well as the detection of nine categories of credibility signals. We provide an in-depth analysis of three key categories: 1) factuality, subjectivity and bias, 2) persuasion techniques and logical fallacies, and 3) check-worthy and fact-checked claims. In addition to summarising existing methods, datasets, and tools, we outline future research direction and emerging opportunities, with particular attention to evolving challenges posed by generative AI. Credibility Assessment, Credibility Signals, Natural Language Processing, NLP, Literature Survey, Large Language Models copyright: acmlicensed journalyear: 2025 doi: 10.1145/3770077 journal: TIST journalvolume: 0 journalnumber: 0 article: 0 publicationmonth: 0 ccs: General and reference Surveys and overviews ccs: Computing methodologies Natural language processing ccs: Human-centered computing Social media 1. Introduction Tackling false information (i.e., disinformation and misinformation) has attracted a substantial attention in recent years from researchers, media professionals, AI and social media practitioners as well as the general public. Researchers have approached false information detection through a variety of classification tasks, leveraging AI technologies such as machine learning, natural language processing, computer vision, or social network analysis (Sharma et al., 2019). The primary stream of research works on false information detection, commonly also denoted as a fake news detection (Shu et al., 2017; Zhou and Zafarani, 2020; Hu et al., 2024), assesses the veracity (i.e., factual accuracy) of information by means of a single-label prediction, typically a binary one (i.e., false/true content) (Shu et al., 2017). Some works go beyond a binary classification and predict multiple classes, such as veracity levels, commonly used by fact-checking organizations (e.g., true, mostly true, mixture, mostly false, false). However, veracity alone offers only a limited perspective on the potential harm of online content. In practice, it should be considered alongside credibility. By proceeding from existing (quite heterogeneous) definitions (Hellmueller and Trilling, 2012; Qureshi et al., 2021; Alrubaian et al., 2019; Rieh and Danielson, 2007; Self and Roberts, 2019), under the credibility we understand the perceived trustworthiness, accuracy, and reliability of information or its source. It reflects the extent to which content is believed to be factual, unbiased, and free from manipulation or deception. False content that is perceived as highly credible by its audience can lead to significantly greater harm than false content that is clearly perceived as lacking credibility. Credibility assessment therefore provides an interesting potential to extend and complement the existing stream of works on false information detection. It typically follows a two-step process. First, granular credibility signals (also referred to as credibility indicators) are detected. Second, these signals (serving as evidence) are aggregated into a single ordinal credibility label or a numerical credibility score. Common examples of credibility signals include subjectivity and bias present in the content, persuasion techniques, logical fallacies or presence of machine-generated content, which has become increasingly relevant with the rise of generative AI. While credibility assessment can be conducted manually by the end users, its full potential lies in automated detection of credibility signals and their consequent aggregation by a credibility assessment algorithm. These automated approaches may range from simple (e.g., rule-based or heuristic) methods to more advanced systems powered by AI, capable of nuanced and context-aware content analysis. Such automation not only improves scalability but also enables real-time evaluation of large volumes of online content. The benefit of credibility signals is their flexibility and wide-applicability. Besides their primary usage, to be aggregated into a credibility label/score, they can also be utilized and interpreted by a user directly (e.g., as an “information nutrition label” (Fuhr et al., 2018)), provide valuable inputs to information retrieval engines or recommender systems in order to prefer content/sources associated with a high level of credibility (Weerkamp and De Rijke, 2008), or even serve as features in subsequent classification tasks (including false information detection itself (Sitaula et al., 2020)). Credibility assessment and credibility signals are, furthermore, especially useful in cases in which we cannot easily ascertain that something is true or false (on a single veracity dimension) – concept of credibility provides a necessary level of granularity to represent a potentially complex information from multiple perspectives. Credibility signals are also more easily interpretable and naturally support semi-automatic human-centered AI approaches that attempt supporting instead of replacing a human expert or a lay person (individual credibility signals can be automatically detected and their interpretation can be subsequently done by a human). Study by Lu et al. (2022) showed that even if people are influenced by others when judging the veracity of online content, providing accurate AI-based credibility indicators can effectively improve people’s ability in detecting false information. Last but not least, they are also aligned with journalistic/fact-checking workflows for identifying trustworthy information (to be cited) or potential false information (to be fact-checked). In contrast to other related research areas (including false information detection), research works on automatic credibility assessment are mostly carried out in isolation. As we show in this survey, there is a lack of works addressing multiple categories of credibility signals at once and, moreover, many research works do not explicitly state that the outcome of their solution can actually serve as a credibility signal. Getting familiar with the existing works on automatic credibility assessment and detection of credibility signals is challenging due to an ambiguous and inconsistent terminology used by researchers, a lack of clear definition of credibility and credibility signals as well as due to a missing standardized taxonomy of various credibility signals’ categories. At the same time, automatic credibility assessment can significantly benefit from a holistic approach. For example, due to common underlying similarities, there is a big potential of transfer or multi-task learning. While there is a wide range of possible credibility signals, we specifically focus on such signals that relate to textual content and can be automatically detected by Natural Language Processing (NLP) methods. This area is currently at the spotlight due to the recent advancements of Large Language Models (LLMs). Their rapid development has a revolutionizing effect on many text classification tasks, automatic credibility assessment and detection of credibility signals not being an exception. Despite multiple surveys addressing false information detection (e.g., (Hu et al., 2024; Tufchi et al., 2023; A.b. et al., 2023; Zhou and Zafarani, 2020; Shu et al., 2017)), to the best of our knowledge, there is no survey on automatic credibility assessment and detection of credibility signals from the NLP perspective. In this paper, we therefore provide the first comprehensive and systematic literature overview of credibility signal detection with the focus on NLP and LLMs. The main contributions are as follows: 1. We address the fragmentation of existing works on automatic credibility assessment and credibility signals detection. We connect these research efforts under a common framework and identify shared trends, challenges, and open problems. In this way, we motivate future research in connecting and integrating individual outcomes (i.e., to move from currently isolated individual signals to credibility assessment effectively combining them). 1. We provide a necessary background overview of definitions and dimensions of credibility assessment (including a unified categorization of credibility signals based on their various taxonomies and list used in the existing literature) to setup the common understanding that is currently missing in the existing works. 1. We analyse and describe a total of 175 NLP works tackling the automatic credibility assessment and detection of credibility signals. We provide an in-depth description of 3 key categories of credibility signals, which have been selected due to a considerable NLP research interest and their recognition as important by end users, namely: (i) factuality, subjectivity and bias; (ii) persuasion techniques and logical fallacies; and (iii) check-worthy and fact-checked claims. This analysis is complemented with an overview of additional 6 categories of credibility signals. We also thoroughly position such NLP research in the context of other modalities as well as multidisciplinary works. 1. We identify and critically analyse key gaps in the existing literature, highlighting underexplored areas and methodological limitations in the current approaches to credibility assessment. Furthermore, we outline future research challenges and opportunities, with particular emphasis on the rapidly evolving landscape shaped by generative AI, especially Large Language Models (LLMs). We discuss how these advancements both pose new threats, such as the large-scale generation of highly credible yet false content, and offer new capabilities for enhancing the detection and aggregation of credibility signals. Our analysis aims to guide future research toward more comprehensive, robust and explainable credibility assessment approaches. Besides the review of published research works, this survey builds on extensive past research activities and acquired knowledge of its authors in the target area. A unique composition of authors consisting of experts not only on the primary NLP area, but also on other related research domains (such as computer vision, social network analysis) as well as media professionals, provides a novel holistic perspective on the addressed topic. Such diversity contributes, besides others, to identification of open problems and future challenges, in which an application of research in practice plays a prominent role. This survey is structured into 10 sections. Section 2 provides definitions of core concepts related to credibility, describes various dimensions of credibility assessment as well as defines the scope of this survey while situating it into the context of other existing surveys. In Section 3, the methodology employed in the survey process is thoroughly described. Section 4 focuses on research papers that address automatic credibility assessment (i.e., approaches addressing both steps — detection of credibility signals and their aggregation into a credibility label/score). Subsequently, Sections 5-7 address an automatic detection of 3 selected key categories of credibility signals. Section 8 supplements this analysis with a brief overview of additional 6 categories of credibility signals, including the complementary perspective of credibility signals addressed in the non-NLP research. In Section 9, we provide an orthogonal analysis of challenges and open problems that characterize the state of the art in this area. Finally, conclusions are drawn in Section 10. 2. Background 2.1. Definitions The construct of credibility has received more scholarly attention than most other communication variables, with foundational research dating back to 1951, when Hovland and Weiss (1951) demonstrated that the effectiveness of communication is heavily influenced by the perceived credibility of its source. A systematic meta-analysis of leading international communication journals spanning 1951 to 2011 (Hellmueller and Trilling, 2012) examined how source, message, and media credibility have been conceptualized and measured over time. The findings revealed significant inconsistencies across credibility scales, a lack of operational precision, and limited replication and validation of the credibility construct. These discrepancies largely stem from the fact that credibility is a complex, multidimensional concept that lacks a single, unified, and widely accepted definition. Existing definitions vary especially across various disciplines and fields, which study credibility from different perspectives, such as information science, psychology, marketing or human-computer interaction (HCI) (Rieh and Danielson, 2007). It is commonly defined as a high-level construct with a help of related concepts, such as believability (which is considered to be roughly a synonym with credibility), trust/trusthworthiness, reliability or their various combinations (Qureshi et al., 2021; Alrubaian et al., 2019; Rieh and Danielson, 2007; Self and Roberts, 2019). Credibility is always tied to a target entity (an object of assessment), that can be either a piece of content or a source spreading such content (Kiousis, 2001). Due to ambiguity of terminology, in this survey we opt for community-contributed definitions (Hawke, [n. d.]) gathered by the Credible Web Community Group https://credweb.org/. It defines credibility as a degree to which information is credible (believable) and to which information appears non-misleading and useful (for the given audience). Credibility assessment is a process of ascertaining some level/degree of credibility to a target entity. Practically, credibility assessment can be formulated as (Qureshi et al., 2021): (i) an ordinal classification problem (i.e., assigning a label from pre-defined categories, such as a low/high level of credibility); or as (ii) a regression/scoring problem (i.e., assigning a numerical score). In both cases, there are no standard scales adopted yet. Credible Web Community Group distinguishes two kinds of credibility assessment acts (Hawke, [n. d.]): 1. Making credibility decisions is an act of adhoc deciding for oneself what to believe, which is often informal, immediate, and unconscious. Doing this incorrectly often leads to being misled, although it may not be practical to do it correctly at all times. 1. Credibility analysis is an act of systematic gathering, organizing, and analysing evidence to help people make credibility decisions about a particular information item. A credibility analysis process (performed manually by people or automatically by machines) might produce some kind of report which might itself be called a credibility assessment and might include a credibility score. In the following text, we will focus especially on this act. Consequently, a credibility signal (Hawke, [n. d.]) is a small unit of information used in making a credibility assessment as an evidence. This can be a measurable feature of the information being assessed for credibility, or information about it (metadata), or information about entities which relate to it in various ways, such as the entity who provided it. Credibility indicator (Hawke, [n. d.]) is commonly used interchangeably with credibility signal. In some communities, credibility signal is used for inputs to credibility assessment algorithms and credibility indicator is used for the display features added to the output, to communicate (explain) results to an end user. In this survey, we prefer to use credibility signal, while we would like to emphasize that in practice many signals can be used directly also as indicators communicated to end users. Finally, credibility assessment tools (Hawke, [n. d.]) are software features or applications which perform credibility assessment or help people do so. Such tools can either derive credibility signals, or implement also a credibility assessment algorithm to aggregate such signals into a credibility label/score. Credibility and credibility assessment is inherently very close to false information and the task of false information detection. Nevertheless, credibility (as also defined above) is rather a parallel/complementary concept. While they tend to correlate (low credibility may indicate false content), non-credible content does not necessarily need to be false only and vice versa. We would like to emphasize that credibility signals have been already recognized in false information research as a possible solution towards more accurate and explainable methods. The work by Grieve and Woodfield (2023) delve into the complexities of identifying and analysing disinformation deceptive news. The authors study the way language adapts to different contexts and purposes, arguing that the linguistic choices in deceptive news differ systematically from those in genuine news. The difference in intent – to deceive versus to inform – leads to detectable variations in language use. The authors illustrate this with the famous case of fake news involving Jayson Blair of The New York Times, where false news were less informationally dense and less confident than real ones. The pressure to invent news and the lack of factual grounding influenced writing style, making it less precise and authoritative. The authors propose that this type of register variation could be a broader indicator of fake news. They call for continued research to understand the linguistic patterns of fake news and to develop strategies to combat its spread. Although the above framework is only based on the analysis of one reporter’s articles, this approach may be extended to corpora of non-credible/fake news and credible/real news on various topics to try to identify credibility signals/register variation between the two. To complement this recent case, we can also recall that the idea of analysing the language of disinformation and propaganda has its source in the pioneering work of the German linguist and philologist Viktor Klemperer (Klemperer and Brady, 2006). Klemperer showed how hate speech and conspiracy theories were used by the Nazis to justify the persecution of Jews. He also showed how these discourses were repeated and amplified by the media and institutions, until they became the norm. 2.2. Dimensions of credibility assessment In the past research works, a wide range of approaches studying the concept of credibility and credibility assessment was developed. In the following overview (see also Figure 1), we provide various dimensions adopted to study these concepts. \pgfqpointtransformed \pgfqpointtransformed \pgfqpointtransformed \pgfqpointtransformed \pgfqpointtransformed \pgfqpointtransformed \pgfqpointtransformed \pgfqpointtransformed \pgfqpointtransformed \pgfqpointtransformed \pgfqpointtransformed \pgfqpointtransformed | Dimensions of credibility assessment | | --- | Approaches to credibility assessment Automation-based approaches Human-based approaches Hybrid approaches Levels of credibility assessment Content level User/source level Topic/event level Media level Hybrid level Taxonomies of credibility signals Information Source Content-based signals Context-based signals Polarity Predictors of high credibility Predictors of low credibility Subject Type Text-based subjects: Claim, Text (in general), Article, Title Multimedia-based subjects: Image, Audio, Video Metadata subjects: Web page, Website, Aggregation, Venue, Provider, Creator, Person, Organization Figure 1. Dimensions of credibility assessment. Bold font highlights such aspects which are within the scope of this survey (see Section 2.4 for more information). Please note that Content level under the dimension of Levels of credibility assessment and Content-based signals under the dimension of Taxonomies of credibility signals refer to two different concepts – the former to the level of analysis, and the latter to the nature of the source data from which the credibility signals are detected. Approaches to credibility assessment. First, the credibility assessment methods can be broadly divided into three major groups (Alrubaian et al., 2019): 1. A large group of automation-based approaches focuses either on automatically detecting the credibility signals or aggregating them into the credibility indicators. The complexity of such automation can be very diverse, ranging from simple rule- and heuristic-based methods, through weighted and information retrieval (IR) approaches to supervised/unsupervised machine learning. This group also covers research works focusing on building datasets appropriate to create such methods/models; as well as systems (web applications, browser extensions, etc.) built on the top of such automated methods. 1. The second group corresponds to human-based approaches, in which various aspects of credibility are studied from the perspective of end-users. The human-based credibility assessment approaches can range from voting methods, cognitive and perception approaches to manual verification approaches. 1. Finally, hybrid approaches combine and utilize the advantages of both the automation- and human-based approaches. Levels of credibility assessment. Following a type of a target entity, which is a subject of credibility assessment, we can distinguish five levels of credibility assessment as follows (Alrubaian et al., 2019; Qureshi et al., 2021): 1. Content level. At the content level, the task is to analyse the content attributes typically of a social media post or a news article. It is the most fundamental and important type of assessment, which is commonly used to determine the credibility at other levels (e.g., if a post is assessed to be credible, also a corresponding user/source or an associated topic/event is considered to be credible). 1. User/source level. This level of credibility assessment depends on features extracted from user accounts (e.g., age, education, profile image) and a history of user-generated content. Besides a specific user, it can relate to a broader source (e.g., a news portal, an organization). 1. Topic/event level. At this level, a credibility is assessed by proceeding from a cluster of posts falling under a specific trending or potentially high-impact topic or event (e.g., elections, societal crisis situations). 1. Media level. At the media level, the medium used to communicate and spread information is a target of credibility analysis (e.g., an online social network). This level typically encompasses credibility analysis of authors, spreaders as well as posts themselves. 1. Hybrid level. To optimally utilize the advantages of individual levels and take advantage of high correlation between them, researchers commonly apply a hybrid approach in which the credibility is assessed at multiple levels at the same time. Taxonomies of credibility signals. Credibility signals, as a core unit in the credibility assessment process, can be categorized by various taxonomies, following different views. First, reflecting the source of information, many works broadly distinguish between content- and context-based credibility signals (Zhang et al., 2018; Kang et al., 2012; Bhuiyan et al., 2020; Wu et al., 2020; El Ballouli et al., 2017; Esteves et al., 2018; Qureshi and Malick, 2023; Piccolo et al., 2021; Jahanbakhsh and Karger, 2024; Chang et al., 2021; Ji et al., 2023b; Gao et al., 2015; Choi et al., 2023). Content-based signals are derived from the content itself, such as partisanship, emotional appeal, or persuasion techniques. On the other hand, context-based signals, also referred to as provenance- (Feng et al., 2023) or meta-information, take into account contextual (e.g., user, source) cues, such as user statistics, location, domain reputation, or the number of shares. In practice, when a content-level credibility (e.g., a credibility of a social media post) is about to be assessed, a combination of both – content-based credibility signals (e.g., a presence of persuasion techniques) as well as context-based credibility signals (e.g., the same claim was already fact-checked) – may be utilized. Second, the credibility signals differ in their polarity. In the existing works, some signals are formulated as predictors of high credibility (e.g., high objectivity), while other signals are formulated as predictors of low credibility (e.g., presence of logical flaws). When interpreting or aggregating individual credibility signals (either by an end user or a credibility assessment algorithm), such polarity must be explicitly taken into consideration (e.g., in (Podgurski et al., 2022; Leite et al., 2024)). Third, Credible Web Community Group (Multiple authors and Hawke, 2019) structured credibility signals by a subject type (a part of target entity they are derived from): claim, text, image, audio, video, article, title, web page, website, aggregation (e.g., Really Simple Syndication - RSS), venue, provider, creator, person, or organization. Despite many commendable efforts, there is no standardized unified list and categorization of credibility signals. As a result, research works commonly utilize various and very diverse lists of credibility signals. For example, Molina et al. (2021) proposed separate lists of credibility signals (or features as they have been denoted by the authors) to distinguish between eight types of online content, such as real news, fabricated news, or satire. Each list grouped signals into 4 main categories roughly corresponding to levels of credibility assessment: (i) message and linguistic, (ii) sources and intentions, (ii) structural, and (iv) network. An alternative approach was used by the Credibility Coalition https://credibilitycoalition.org/, which organized weekly remote sessions during which participants drafted about 100 credibility signals specifically aimed at the credibility of web pages (Zhang et al., 2018). They have been later coalesced into 12 major categories, including reader behaviour, revenue models, publication metadata, and inbound and outbound references. In 2017, the Credibility Coalition formed the World Wide Web Consortium (W3C) Credible Web Community Group, which continued in this initiative. It crowdsourced from human experts (researchers as well practitioners) an extensive list containing more than 200 credibility signals, commonly referred to as W3C signals (Multiple authors and Hawke, 2019). This list introduces a number of signals categorized by subject types. While such a list still remains an informal and incomplete draft and has not been standardized yet, it has been adopted and served as a foundation to several research works (e.g., (Piccolo et al., 2021; Podgurski et al., 2022)). We can conclude that no standard or widely used and recognized list or categorization of credibility signals currently exists that could serve as a solid basis for our literature survey. Categorization by means of subject types as proposed by W3C signals (Multiple authors and Hawke, 2019) is probably the most comprehensive and elaborated one, nevertheless, it mixes at the top level different modalities (text, video, audio, etc.) and content types (article, title, claim etc.). More specifically, the signals relevant to our survey are not optimally classified under the text modality or across various individual content types. Therefore, we thoroughly analysed the relevant existing works (Molina et al., 2021; Zhang et al., 2018; Multiple authors and Hawke, 2019; Podgurski et al., 2022; Nakov et al., 2024) and proposed a novel unified categorization of textual credibility signals (see Figure 2). We adopted a bottom-up approach as follows: 1. First, out of mentioned credibility signals, we identified specifically textual credibility signals (i.e., signals that can be derived from textual content by means of diverse NLP methods). 1. We grouped the resulting credibility signals according to their semantic similarity and shared datasets and methods applied for their detection. To this end, we employed three grouping strategies. At first, we considered the existing categorizations that have been proposed by the relevant works, especially individual (sub)categories from the above-mentioned W3C signals (Multiple authors and Hawke, 2019). In this way, we grouped all signals related to a title into a single category named Clickbaits and title representativeness, all signals related to claims into a category of Check-worthy and fact-checked claims as well as reused a subcategory of References and citations. Second, we grouped together signals which refer to the same or highly-similar concepts and just use a different terminology, e.g., grammar and spelling errors, grammar, spelling or punctuation mistakes, punctuation naturally fall under the category of Text quality; or profanity, incivility, impoliteness and hate speech under the category of Offensive language. In some cases, the signals refer to the same concept, but in an opposite meaning, like originality and attribution of non-original content belong to the same category of Originality and content reuse. In some cases, the signals refer to the same concepts but are defined on different granularity levels, e.g., logical fallacies, logical flaws, ad-hominem attacks all together belong under Persuasion techniques and logical fallacies. Third, we grouped together signals where the borders between them are not clearly defined and in practice they correlate with each other, e.g., subjectivity, political bias, confirmation bias, emotional tone were grouped under Factuality, subjectivity and bias. We also jointly combined these strategies, especially the category of Factuality, subjectivity and bias groups signals that are closely related, commonly refer to the same concepts but from opposite angles, or are at different levels of abstractions. 1. In this way, we finally proposed 8 categories of credibility signals (see also Figure 2; more detailed description of each category is provided in the later sections): (i) Factuality, subjectivity and bias; (ii) Persuasion techniques and logical fallacies; (iii) Check-worthy and fact-checked claims; (iv) Text quality; (v) References and citations; (vi) Clickbaits and title representativeness; (vii) Originality and content reuse; and (viii) Offensive language. 1. In addition, we recognized one more category of signals – (ix) Machine-generated text – that has become very relevant only recently with the emergence of LLMs and, therefore, it did not appear in the existing lists/taxonomies. At the same time, it is a strong predictor of credibility as fully generated text can contain many factual errors either as a result of unintended hallucinations of LLMs or due to their intended misuse to generate false content (the previous works demonstrated significant vulnerabilities of LLMs to generate disinformation (Vykopal et al., 2024), even personalized one (Zugecova et al., 2025)). | Textual Credibility Signals | | --- | Factuality, subjectivity and bias Factual accuracy (Nakov et al., 2024; Molina et al., 2021), Event factuality (Li et al., 2020b; Cao et al., 2021; Rovera et al., 2025), Subjectivity (Podgurski et al., 2022), Selection and presentation bias (Nakov et al., 2024), Hyperpartisanship / political bias (Multiple authors and Hawke, 2019), Confirmation bias (Multiple authors and Hawke, 2019), Framing (Nakov et al., 2024), Emotional tone (Zhang et al., 2018), Formal / Informal tone (Multiple authors and Hawke, 2019), Emotional, sensational or affective language (Podgurski et al., 2022), Emotionally charged (Molina et al., 2021; Multiple authors and Hawke, 2019), Emotional Valence (Multiple authors and Hawke, 2019) Use of hyperboles (Molina et al., 2021), Impartial reporting (Molina et al., 2021), One-sided reporting (Molina et al., 2021), Discrepancies or omissions (Molina et al., 2021) Persuasion techniques and logical fallacies Logical fallacies (Zhang et al., 2018), Logical flaws (Molina et al., 2021), Inference (Zhang et al., 2018), Ad-hominem attacks (Molina et al., 2021), Conspirational reasoning (Molina et al., 2021), Arguments from authority (Molina et al., 2021), Common man appeals (Molina et al., 2021), Call to Action (Political) (Multiple authors and Hawke, 2019) Check-worthy and fact-checked claims Fact-checked (Zhang et al., 2018; Molina et al., 2021), Fact-check status of claim (Multiple authors and Hawke, 2019), Article has a central claim (Multiple authors and Hawke, 2019) Text quality Grammar and spelling errors (Podgurski et al., 2022), Grammar, spelling or punctuation mistakes (Molina et al., 2021; Multiple authors and Hawke, 2019), Readability (Podgurski et al., 2022), Punctuation (Podgurski et al., 2022), Vocabulary or reading level (Multiple authors and Hawke, 2019) References and citations Quotes from outside experts (Zhang et al., 2018), Citation of organizations and studies (Zhang et al., 2018), Representative citations (Zhang et al., 2018), External links (Podgurski et al., 2022), No quotes or made-up quotes (Molina et al., 2021), No source attribution (Molina et al., 2021; Multiple authors and Hawke, 2019), Uses standardized references or citations (Multiple authors and Hawke, 2019), Few to zero references or citations (Multiple authors and Hawke, 2019) Clickbaits and title representativeness Clickbait title (Zhang et al., 2018; Podgurski et al., 2022; Multiple authors and Hawke, 2019), Misleading and clickbait headlines (Molina et al., 2021), Title representativeness (Zhang et al., 2018; Multiple authors and Hawke, 2019) Originality and content reuse Originality (Zhang et al., 2018), Originality types (Multiple authors and Hawke, 2019), Attribution of Non-Original Content (Multiple authors and Hawke, 2019) Offensive language Profanity (Podgurski et al., 2022), Incivility and impoliteness (Multiple authors and Hawke, 2019), Hate speech (Multiple authors and Hawke, 2019) Machine-generated text No appearances in the existing lists/categorizations yet Figure 2. Unified categorization of textual credibility signals. Light blue nodes consist of credibility signals depicted in the existing taxonomies and categorizations. Bold font highlights categories of credibility signals on which this survey put a focus on (see Section 2.4 for more information). Importance and effects of credibility signals on end users. Due to a broad definition of credibility, as well as rich taxonomies of credibility signals, a number of human-based studies have analysed the importance of individual credibility signals and their effects on the perceived credibility, in general as well as for specific groups of end users (according to their expertise, age, education etc.). First, these studies address various content types and platforms, such as social media posts (Shusas and Forte, 2024; Al-Eidan et al., 2009), news articles (Lu et al., 2022; Huang et al., 2019; Heuer and Glassman, 2024), videos and video sharing platforms (Hughes et al., 2024; Feng et al., 2023), images (Feng et al., 2023), search engines (Wang et al., 2023a; Juneja et al., 2024; Lurie and Mustafaraj, 2018), or online encyclopedias (Elmimouni et al., 2022). At the same time, they address various types of end users. Piccolo et al. (2021) perform a human study with two groups of participants, academics/students and random social media users. Rijo and Waldzus (2023) looked into how voting patterns and political beliefs influence the way people evaluate information credibility. Jia et al. (2022) focused on liberal and conservative users. Other works (e.g., (Hughes et al., 2024)) focus on social media users in general. Regarding the scope of credibility signals, some works focus on signals provided directly by (social media) platforms (Wang et al., 2023a; Lurie and Mustafaraj, 2018), some investigate a single credibility signal (e.g., hyper-partisanship in (Jia et al., 2022)) or a whole spectrum of signals (e.g., 28 signals analysed in (Piccolo et al., 2021)). Jia et al. (2022) compared credibility signals produced by three various sources – an algorithm, a community, and a third-party fact-checker. Besides content- and context-based signals, Chang et al. (2021) considered also an additional group of design signals (consisting of four signals: interaction design, interface design, navigation design, and security settings). While the most of these studies focused on English content, the number of studies comparing the importance of credibility signals on non-English data is rather limited. The work by Gao et al. (2015) performs the human study on the importance of credibility signals for the Weibo Chinese misinformation dataset regarding health and safety information. All these humans studies resulted into valuable findings about which credibility signals are the most important and have the greatest effect on the end-users. Out of 28 credibility signals, Piccolo et al. (2021) found that context-based (presence of links, publisher, author) contribute most towards human judgement. Interestingly, different groups of participants exhibit different patterns towards content-based signals. For example, fallacy and partisanship, as well as clickbait title, were ranked high by academics and students only, while social media users did not find this information important. In the human study on e-Health information, Chang et al. (2021) found that content signals were most important, accounting for 57.3% of cases, followed by source-related or context-based signals (26.0%). Finally, they found that design-related signals, such as website layouts and overall design are least important and account for 16.7% of decisions. Lin et al. (2016) found that the authority and retweet patterns are most important for human participants when deciding on information credibility. Gao et al. (2015) showed that the impact of source credentials, such as the authenticity of the authors, on users’ credibility assessment was limited. On the other hand, content-based signals, such as objective claims, are considered more insightful for credibility assessment. 2.3. Existing surveys The existing surveys on automatic credibility assessment focus on specialised domains. Alrubaian et al. (2019) conducted a literature review on the credibility of social media information while considering 4 levels of credibility assessment (i.e., post, member, topic/event, hybrid level). The surveyed methods were categorised into automated-, human-, and hybrid-based approaches. Automated-based approaches are further subcategorised into machine learning approaches, graph-based semisupervised approaches, and weighted and IR algorithms. Their analysis spans until 2018, and therefore, the automated methods are limited to traditional machine learning (e.g., classifications algorithms like decision trees, logistic regression) and do not include more recent transformer or LLM-based approaches. Similarly, Qureshi et al. (2021) perform an analysis of the credibility detection methods in the social media and microblog domains. The authors adopt the level-based classification of the task proposed by Alrubaian et al. (2019). In addition, they propose a fine-grained framework based on the existing approaches that targets various aspects behind credibility assessment. Their theoretical framework consists of content-, media-, website- and interaction-based assessment of social media information. Within this framework, they identify two main directions of research, namely, post-level and user-level constructs. The post-level constructs are further divided into the detection of: (i) deceptive and trustful information, (ii) bias and objectivity detection, and (iii) hate speech and offensive language detection tasks. The user-based constructs, on the other hand, are based on: (i) user deception level, (ii) user competence, and (iii) user authority. Another group of the surveys focuses on particular categories of credibility signals. For example, Nakov et al. (2024) surveyed the literature on factual accuracy and biases in social media, Panchendrarajan and Zubiaga (2024) analysed claim detection for automated fact checking, or Wu et al. (2025) addressed machine-generated text detection. In contrast to these existing surveys, we focus on in-depth analysis of Natural Language Processing (NLP) research, with a specific focus on automatic credibility assessment and detection of individual textual credibility signals in the context of online social media content. This focus is motivated and determined by the recent emergence of large language models, that have significantly influenced the NLP field. The research works on credibility assessment are already adopting this new generation of models, nevertheless, their potential is still far from being explored completely. This survey thus stands on the intersection of credibility assessment (which is an application domain of utmost importance for protecting our society and democratic values) and artificial intelligence (which is a research domain rapidly advancing thanks to many recent discoveries). 2.4. Scope of the survey At first, this survey aims to systematically cover research that explicitly addresses the entire process of automatic credibility assessment (i.e., both steps of credibility signals detection as well as their aggregation into a single credibility label/score). However, restricting our review solely to such works would overlook a substantial body of research focused on the detection of individual categories of credibility signals. Although these studies may not explicitly frame their contributions within the context of credibility assessment, i.e., they often do not adopt the relevant credibility-related terminology or perform aggregation of multiple signals, they provide valuable insights and methods that are highly relevant to the broader credibility assessment framework. More specifically, we define the scope of this survey as follows: 1. Out of methods to credibility assessment, this survey specifically tackles automation-based approaches. 1. Regarding the levels of credibility assessment, the research works falling into the scope of this survey typically address the credibility assessment at the content level, but indirectly also at user/source, topic/event, media and hybrid levels. 1. Out of various taxonomies of credibility signals, the survey covers both content- as well as context-based signals, predicting high as well as low credibility that are text-based (while relation to other modalities is briefly documented in Section 8.7). 1. Regarding the categorization of credibility signals, the survey systematically covers all 9 categories of textual credibility signals from the unified categorization of textual credibility signals (as shown in Figure 2). Out of 9 categories of textual credibility signals, we put a specific focus and provide an in-depth analysis for three of them, namely: 1. Factuality, subjectivity and bias; 1. Persuasion techniques and logical fallacies; 1. Check-worthy and fact-checked claims. This selection is motivated by several factors. (i) The selected categories have attracted a considerable amount of interest in the NLP community, resulting in a substantial body of work that is suitable for systematic review. Moreover, the authors of this survey have direct practical experience in training and deploying detection models for these specific signals, which enables a more informed and application-oriented analysis and discussion. (ii) Although some of these categories have been partially addressed in the previous surveys, they have not been examined through the lens of credibility and credibility assessment, nor in terms of their integration with other signal detection approaches. Importantly, these categories offer strong potential for effective methodological integration. For example, subjectivity, bias, and persuasion techniques share underlying linguistic features, which can be exploited through transfer learning or multitask learning applied to shared datasets. (iii) The selected categories have been also identified in human-centered studies as particularly influential in shaping perceptions among social media users and domain experts (e.g., journalists and media professionals) (see, e.g., Piccolo et al. (2021) and Chang et al. (2021), and Section 2.2 for further discussion). Regarding the time span, the survey covers the research papers published before May 2024, when the systematic identification of research works was done. The first papers covered by the survey originate in 2004. 3. Methodology 3.1. Research paper collection process In order to collect the relevant papers, we adopted the conventional and standardized methodology for literature review called Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) (Takkouche and Norman, 2011) and specifically its updated version PRISMA 2020 (Page et al., 2021). In addition to being commonly used in literature surveys, PRISMA was also previously used to perform the analysis of the topics related to credibility signals, such as media bias (Rodrigo-Ginés et al., 2023). Figure 3 depicts the standardized PRISMA 2020 flow diagram (Haddaway et al., 2022) illustrating individual steps and the number of research papers processed. <details> <summary>x1.png Details</summary>  ### Visual Description ## Flowchart: Study Identification and Selection Process ### Overview The image is a flowchart illustrating the process of identifying and selecting studies for inclusion in a review or meta-analysis. It outlines the steps involved in identifying studies from databases and registers, as well as from other methods like citation searching, and the subsequent screening and inclusion/exclusion criteria. ### Components/Axes The flowchart is divided into two main columns: 1. **Left Column (Yellow Header):** "Identification of new studies via databases and registers" 2. **Right Column (Gray Header):** "Identification of new studies via other methods" The flowchart also has three vertical sections, indicated by colored bars on the left: 1. **Top (Blue):** "Identification" 2. **Middle (Blue):** "Screening" 3. **Bottom (Blue):** "Included" The flowchart uses rectangular boxes to represent stages in the process, with arrows indicating the flow of studies between stages. Each box contains a description of the stage and the number of studies at that stage (n = value). ### Detailed Analysis or ### Content Details **Left Column (Databases and Registers):** * **Identification:** * "Records identified from: Databases (n = 13,243):" * "ACM DL (n = 417)" * "ACL Anthology (n = 12,132)" * "IEEE Xplore (n = 311)" * "ScienceDirect (n = 383)" * "Records removed before screening: Duplicate records (n = 0)" * **Screening:** * "Records screened (n = 13,243)" * "Records excluded (n = 13,010)" * "Reports sought for retrieval (n = 233)" * "Reports not retrieved (n = 9)" * **Included:** * "Reports assessed for eligibility (n = 224)" * "Reports excluded: Reason: Non-NLP (n = 38), Reason: Non-relevant (n = 52)" * "Reports of new included studies (n = 175)" **Right Column (Other Methods):** * **Identification:** * "Records identified from: Citation searching (n = 52)" * **Screening:** * "Reports sought for retrieval (n = 52)" * "Reports not retrieved (n = 0)" * **Included:** * "Reports assessed for eligibility (n = 52)" * "Reports excluded: Reason: Non-NLP (n = 6), Reason: Non-relevant (n = 5)" * The arrow from "Reports of new included studies (n=175)" in the left column points to the right column, but does not connect to any specific box. ### Key Observations * The majority of records were identified through databases and registers (13,243) compared to citation searching (52). * A large number of records were excluded during the screening process (13,010 from databases and registers). * The primary reasons for excluding reports were "Non-NLP" and "Non-relevant". * The final number of included studies from databases and registers is 175. * The number of reports not retrieved is significantly different between the two methods (9 vs 0). ### Interpretation The flowchart provides a clear overview of the study selection process. The large number of records excluded during screening highlights the importance of well-defined inclusion and exclusion criteria. The reasons for exclusion ("Non-NLP" and "Non-relevant") suggest that the search strategy may have retrieved many studies outside the scope of the review. The difference in the number of reports not retrieved between the two methods could indicate differences in accessibility or availability of studies identified through databases versus citation searching. The arrow from the left column's final box to the right column is ambiguous and may indicate a combination of studies from both methods. </details> Figure 3. PRISMA 2020 flow diagram (Haddaway et al., 2022) depicting the standardized methodology applied to collect the relevant research papers, together with a number of research papers processed in each step. Table 1. Search queries used during the systematic literature search | Automatic credibility assessment | Generic credibility assessment approaches based on credibility signal detection and their aggregation | ”credibility signals” OR ”credibility indicators” OR ”confidence indicators” OR ”non-credibility index” OR ”noncredibility index” OR ”credibility features” | Title, abstract, keywords, the whole article body | | --- | --- | --- | --- | | Factuality, subjectivity and bias | Credibility signal category comprising factuality, subjectivity, various forms of biases, objectivity-reporting, partisanship, tone, emotional valence, framing | (“factuality” OR “text subjectivity” OR “text objectivity” OR ”subjective bias” OR ”sentiment bias” OR ”media bias” OR “news framing” OR “partisan”) AND (detection OR classification OR identification OR extraction) | Title, abstract, keywords | | Persuasion techniques and logical fallacies | Credibility signal category comprising persuasion techniques, logical fallacies, propaganda | (”propaganda” OR ”persuasion” OR ”logical fallacies”) AND (detection OR classification OR identification OR extraction) | Title, abstract, keywords | | Check-worthy and fact-checked claims | Credibility signal category comprising check-worthy claims, previously fact-checked claims | (”check-worthy” OR ”fact-checked”) AND (claim OR statement OR misinformation) AND (detection OR classification OR identification OR extraction) | Title, abstract, keywords | | Additional credibility signals | Machine-generated text, Text Quality, References and citations, Clickbaits and title representativeness, Originality and content reuse, Offensive language | Search is done on ad-hoc basis, no systematic keywords were used | Title, abstract, keywords | Identification – Search queries. To conduct a systematic literature search aimed at identifying papers with high potential relevance, we formulated custom search queries (see Table 1) targeting two main parts of this survey: (i) studies that explicitly address the credibility assessment process, including both the detection of credibility signals and their aggregation, and (ii) studies focused on the automatic detection of the three selected individual categories of credibility signals. Each query combined keywords and logical search operations (AND, OR) to achieve the highest possible coverage and reduce false positives at the same time. The keywords have been carefully selected to capture inconsistent and ambiguous terminology used in the existing works. The first version of queries was proposed following our past experience and research in the respective area, and iteratively refined to cover additional alternative phrases with the same semantics identified in the research papers. If a search query returned a paper that was more appropriately aligned with a different category of credibility signals, the paper was reassigned accordingly. Thanks to the use of mutually distinctive categories of credibility signals, no duplicates were present across the identified sets of papers. Additional categories of credibility signals were covered through ad hoc searches, conducted without predefined search queries. Identification – Databases and registers. The proposed queries were submitted to four large repositories of peer-reviewed scientific publications, namely: (i) ACL Anthology https://aclanthology.org/; (ii) ScienceDirect https://www.sciencedirect.com/; (iii) ACM DL https://dl.acm.org/; and (iv) IEEE Xplore https://ieeexplore.ieee.org/. Identification – Search scope. Papers explicitly addressing credibility assessment and credibility signals detection are quite scarce, therefore, for this type of papers we performed the search for the query terms in the whole body of the document. For individual categories of credibility signals, on the other hand, we searched for the given query term in the title, abstract and user-specified keywords The search in author-specified keywords is available only at the ScienceDirect digital library. only. For ACM Digital Library (n = 417), IEEE Xplore (n = 311) and ScienceDirect (n = 383) we considered all returned papers as identified. Since ACL Anthology uses Google Search as a search engine, the search returned an extensive number of results, many of them being irrelevant. By a manual check, we found out that the relevant results appeared in first 10 pages or less, therefore for each search query we considered as identified papers that appear on first 10 pages of results (still resulting in a large number of 12,132 identified papers). Identification – Extended search. Besides identification of the relevant research papers by the direct search, we applied also citation searching (i.e., search for additional potentially-relevant papers in references from/to the papers identified by the direct search). To some extent, we also checked additional papers published by the same author and at the same venue (e.g., in case of data challenges that are directly focused on the respective category of credibility signals). In this way, we identified additional papers (n = 52), that partially comes from the pre-print servers, especially arXiv. Being a pre-print (a paper that have not underwent the peer-review yet), we carefully checked the quality of such specific cases to make sure that only high-quality papers are included in our survey. Screening. After identifying all papers matching the search queries (n = 13,243), we performed an initial screening of titles and abstracts directly within the user interfaces of the respective digital libraries. Based on this step, we excluded non-relevant papers (n = 13,010). Papers were excluded at this stage if they: (i) used the search keywords in a context that falls outside the scope of this survey as defined in Section 2.4 (e.g., not related to online content credibility or NLP), or (ii) were not full, short, journal, or workshop papers (e.g., abstracts of invited talks or workshops). From the remaining pool of potentially relevant papers (n = 233), the full-text PDF could not be retrieved for only a negligible number (n = 9). For all papers with an accessible full text (n = 224), a final eligibility assessment was performed to exclude any remaining irrelevant papers not identified during the initial screening. Based on an in-depth analysis of the full papers, exclusions were made for the following two reasons: - Non-NLP (n = 38 for direct search, n = 6 for extended search): These papers explored credibility or credibility signals from theoretical, behavioral, human annotation, or other non-NLP perspectives (e.g., computer vision, social network analysis). - Non-relevant (n = 52 for direct search, n = 5 for extended search): These papers mentioned the search terms but in contexts unrelated to credibility signals (e.g., “confidence indicator” in construction risk assessment). Such papers were excluded from further analysis. Included. Ultimately, we identified a set of relevant NLP papers (n = 175) that addressed either credibility assessment or one of selected credibility signal categories, with a primary focus on the NLP domain (i.e., papers that proposed automatic detection or aggregation methods for credibility signals, introduced relevant datasets, or developed systems for automatic credibility assessment). Among the included works, the smallest group consists of papers that explicitly address credibility assessment, whereas the most populated category is check-worthy and fact-checked claims (see Figure 5). Due to the survey’s focus on NLP, the ACL Anthology emerges as the most prominent source for all three individual categories of credibility signals. In contrast, papers addressing credibility assessment are more evenly distributed across digital libraries (see Figure 5). In terms of publication trends, we observe an increase beginning in 2016, likely corresponding to heightened research attention following the U.S. presidential elections (see Figure 6). The continued growth in subsequent years – despite a temporary slowdown during the COVID-19 pandemic – demonstrates sustained and growing interest in the topic addressed by this survey. <details> <summary>images/stats-by-category-direct-extended-search.png Details</summary>  ### Visual Description ## Stacked Bar Chart: Search Methods by Category ### Overview The image is a stacked bar chart comparing the usage of "Direct search" and "Extended search" across four categories: "Automatic credibility assessment", "Factuality, subjectivity and bias", "Persuasion techniques and logical fallacies", and "Check-worthy and fact-checked claims". The y-axis represents a numerical value, but the units are not specified. The chart uses blue bars to represent "Direct search" and red hatched bars to represent "Extended search". ### Components/Axes * **X-axis:** Categorical axis with four categories: * Automatic credibility assessment * Factuality, subjectivity and bias * Persuasion techniques and logical fallacies * Check-worthy and fact-checked claims * **Y-axis:** Numerical axis ranging from 0 to 80, with increments of 10. * **Legend:** Located at the bottom of the chart. * Blue bar: Direct search * Red hatched bar: Extended search ### Detailed Analysis * **Automatic credibility assessment:** * Direct search: Approximately 7 * Extended search: Approximately 5 * Total: Approximately 12 * **Factuality, subjectivity and bias:** * Direct search: Approximately 45 * Extended search: Approximately 11 * Total: Approximately 56 * **Persuasion techniques and logical fallacies:** * Direct search: Approximately 35 * Extended search: Approximately 5 * Total: Approximately 40 * **Check-worthy and fact-checked claims:** * Direct search: Approximately 47 * Extended search: Approximately 20 * Total: Approximately 67 ### Key Observations * "Direct search" is used more frequently than "Extended search" in all categories. * "Check-worthy and fact-checked claims" has the highest total usage, followed by "Factuality, subjectivity and bias". * "Automatic credibility assessment" has the lowest total usage. * The difference between "Direct search" and "Extended search" is most pronounced in "Factuality, subjectivity and bias" and "Check-worthy and fact-checked claims". ### Interpretation The chart suggests that "Direct search" is the preferred method across all categories related to credibility assessment, fact-checking, and logical fallacies. The higher usage in "Factuality, subjectivity and bias" and "Check-worthy and fact-checked claims" may indicate that direct search is more effective or efficient for these types of tasks. The relatively low usage in "Automatic credibility assessment" could reflect the limitations or challenges associated with automated methods. The data implies that while extended search is used, it is not as prevalent as direct search in these contexts. </details> Figure 4. Number of included papers according the direct (by keywords) and extended (by citations) search. <details> <summary>images/stats-by-category-dl.png Details</summary>  ### Visual Description ## Bar Chart: Research Paper Distribution Across Categories and Sources ### Overview The image is a bar chart comparing the distribution of research papers across different categories related to credibility assessment and fact-checking. The chart shows the number of papers in each category, broken down by the source where the papers are published (ACM DL, ACL Anthology, IEEE Xplore, ScienceDirect, and Other). ### Components/Axes * **X-axis:** Categories of research papers: * Automatic credibility assessment * Factuality, subjectivity and bias * Persuasion techniques and logical fallacies * Check-worthy and fact-checked claims * **Y-axis:** Number of papers, ranging from 0 to 40, with increments of 5. * **Legend:** Located at the bottom of the chart. * Blue with diagonal lines from top-left to bottom-right: ACM DL * Red with horizontal lines: ACL Anthology * Yellow/Gold with diagonal lines from bottom-left to top-right: IEEE Xplore * Light Blue with a grid pattern: ScienceDirect * Green with vertical lines: Other ### Detailed Analysis Here's a breakdown of the number of papers in each category by source: * **Automatic credibility assessment:** * ACM DL: Approximately 4 * ACL Anthology: Approximately 3 * IEEE Xplore: Approximately 0 * ScienceDirect: Approximately 1 * Other: Approximately 4 * **Factuality, subjectivity and bias:** * ACM DL: Approximately 8 * ACL Anthology: Approximately 35 * IEEE Xplore: Approximately 7 * ScienceDirect: Approximately 5 * Other: Approximately 1 * **Persuasion techniques and logical fallacies:** * ACM DL: Approximately 5 * ACL Anthology: Approximately 18 * IEEE Xplore: Approximately 4 * ScienceDirect: Approximately 10 * Other: Approximately 0 * **Check-worthy and fact-checked claims:** * ACM DL: Approximately 15 * ACL Anthology: Approximately 27 * IEEE Xplore: Approximately 1 * ScienceDirect: Approximately 6 * Other: Approximately 17 ### Key Observations * ACL Anthology has the highest number of papers in the "Factuality, subjectivity and bias" category, significantly more than other sources. * ACL Anthology also has a high number of papers in "Check-worthy and fact-checked claims". * IEEE Xplore consistently has a low number of papers across all categories. * The "Other" category shows a notable number of papers in "Check-worthy and fact-checked claims". ### Interpretation The chart suggests that ACL Anthology is a primary source for research related to "Factuality, subjectivity and bias" and "Check-worthy and fact-checked claims". ACM DL also contributes significantly to "Check-worthy and fact-checked claims". IEEE Xplore seems to have a limited number of publications in these specific areas. The "Other" category's contribution to "Check-worthy and fact-checked claims" indicates that there are other relevant publication venues not explicitly listed. The data highlights the varying focus and contributions of different publication sources to the fields of credibility assessment and fact-checking. </details> Figure 5. Number of included papers according to the digital library grouped by each credibility signal category. Other refers to additional repositories, such as arXiv. <details> <summary>images/stats-by-year.png Details</summary>  ### Visual Description ## Line Chart: Trends in Credibility Assessment Research ### Overview The image is a line chart illustrating the trends in research related to credibility assessment from 2010 to 2024. It shows the number of publications or studies in different categories over time. The categories include "Automatic credibility assessment," "Factuality, subjectivity and bias," "Persuasion techniques and logical fallacies," "Check-worthy and fact-checked claims," and "Total." ### Components/Axes * **X-axis:** Years from 2010 to 2024. * **Y-axis:** Numerical scale from 0 to 40, with increments of 5. * **Legend:** Located at the bottom of the chart, mapping colors to categories: * Blue: Automatic credibility assessment * Red: Factuality, subjectivity and bias * Yellow: Persuasion techniques and logical fallacies * Green: Check-worthy and fact-checked claims * Orange: Total ### Detailed Analysis * **Automatic credibility assessment (Blue):** * Trend: Relatively flat and low throughout the period. * Approximate values: * 2010: 0 * 2015: 0 * 2019: 2 * 2024: 2 * **Factuality, subjectivity and bias (Red):** * Trend: Generally increasing, with some fluctuations. * Approximate values: * 2010: 3 * 2015: 3 * 2019: 7 * 2024: 4 * **Persuasion techniques and logical fallacies (Yellow):** * Trend: Increasing until 2020, then decreasing. * Approximate values: * 2010: 0 * 2015: 0 * 2019: 9 * 2024: 3 * **Check-worthy and fact-checked claims (Green):** * Trend: Increasing, peaking in 2023, then decreasing. * Approximate values: * 2010: 0 * 2015: 0 * 2019: 7 * 2024: 5 * **Total (Orange):** * Trend: Significant increase, peaking in 2023, then decreasing. * Approximate values: * 2010: 3 * 2015: 3 * 2019: 13 * 2023: 35 * 2024: 12 ### Key Observations * The "Total" publications (orange line) show a significant increase over the years, peaking in 2023. * "Check-worthy and fact-checked claims" (green line) and "Persuasion techniques and logical fallacies" (yellow line) also show notable increases, but with a decrease in 2024. * "Automatic credibility assessment" (blue line) remains relatively low and stable. * "Factuality, subjectivity and bias" (red line) shows a moderate increase. ### Interpretation The chart indicates a growing interest and research activity in the field of credibility assessment, particularly in areas related to fact-checking and persuasion techniques. The peak in 2023 suggests a heightened focus on these topics, possibly driven by specific events or trends. The relatively low activity in "Automatic credibility assessment" might indicate that this area is either less explored or faces significant challenges. The overall increase in research suggests a growing awareness of the importance of credibility assessment in various domains. </details> Figure 6. Evolution of included papers according to year they have been published in. Year 2024 includes papers published online before May 2024. 3.2. Paper categorization and in-depth analysis Included papers were subsequently thoroughly analysed by the authors of this survey. Firstly, for each paper the basic metadata have been collected: - paper type (one of the following options: a full paper, a short paper, a journal paper, a workshop paper, other), - venue (an abbreviation of a conference, a journal or a pre-print), - year, when the paper was published in, - main outcome (one of the following options: a method/model, a study, a dataset, a tool, a survey). Moreover, we annotated several additional types of information that have been identified as the most useful for the analysis and description of the current state of the art, as well as for recognition of remaining open problems and potential future work, namely: - dataset used (if authors used an existing dataset, a name of such dataset, otherwise information about how own dataset was obtained), - content/context signals (whether the paper tackles with content, context or both types of credibility signals, see Section 2.2 for more information), - list of credibility signals (a specific list of signals addressed in the paper), - automated detection (whether a paper attempts to perform also an automatic credibility assessment and/or automatic detection of credibility signals), - models (if paper attempts to perform automated detection, what kind of models have been used), - metrics (if paper attempts to perform automated detection, what kind of metrics have been used), - human agreement (if paper introduces an annotated dataset, what kind of annotator mutual agreement was achieved, measured by standard metrics, e.g., by Cohen’s Kappa). The full list of papers included in this survey, including all annotations is available as a supplementary material to this article in the ACM Digital Library A full list of papers is available as supplementary material to this article in the ACM Digital Library and at https://kinit.sk/public/acm-tist-credibility-assessment-survey.html. Following this analysis, we describe credibility assessment papers as well as each individual category of credibility signals from three perspectives (that are also utilized to structure the consequent categories): (i) datasets, (ii) methods and models, and (iii) tools and services. For each category we summarised the current state of the art and identified problems and challenges that are specifically present in such credibility signal category (such analysis is further elaborated in Section 9, with an orthogonal discussion across all included papers). 4. Automatic credibility assessment This section provides an overview of research efforts that address automatic credibility assessment, encompassing both steps of the process: the detection and subsequent aggregation of credibility signals. In some cases, however, the features used as input to the credibility assessment algorithms are relatively shallow, relying primarily on surface-level linguistic characteristics (e.g., the number of unique words), which only approximate underlying credibility signals rather than explicitly capturing them. This group of approaches explicitly uses the terminology relevant to credibility (such as credibility assessment, credibility signal, credibility score), in contrast to approaches described in the following sections that address detection of a particular category of credibility signals, but typically do not denote them as such. 4.1. Datasets Most of datasets used in the credibility assessment works are annotated at the content level (see Section 2.2 for the overview of levels of credibility assessment). Table 2 provides an overview of the selected datasets created or adapted for the purpose of credibility assessment task. Zhang et al. (2018) created a thoroughly annotated, however, only very small credibility-related dataset. Out of news articles, being the most shared on social media, 40 articles were selected (covering multiple topics: public health, climate science, diseases, vaccines). In total, 6 annotators were recruited to annotate the credibility signals. These indicators were adopted from the initial list of signals created by the Credibility Coalition (see Section 2.2). Out of them, 16 content and context indicators were selected to be annotated. Namely, content indicators included: title representativeness, clickbait title, quotes from outside experts, citation of organizations and studies, calibration of confidence, logical fallacies, tone, inference. Context indicators included: originality, fact-checked, representative citations, reputation of citations, number of ads, spammy ads, number of social calls, and finally placement of ads and social calls. Besides that, all articles were evaluated for an overall credibility by domain experts on a 5-point scale. In the following dataset analysis, two backward stepwise multiple regression models were utilized to measure the predictive value provided by both sets of indicators. For content-based signals, after model convergence, two variables remained: clickbait title and logical fallacies (slippery slope). This model was found to significantly predict credibility (F = 13.972, p ¡ 0.001). For context-based signals, 6 variables remained: fact-checked–reported false, fact-checked–reported mixed results, number of social calls, number of mailing list calls, and placement of ads and social calls. Together, they were also found to significantly predict credibility (F = 12.986, p ¡ 0.001). Despite a wide range of annotated credibility signals as well as overall credibility score, the practical value of the resulting dataset, which was created only as a proof of concept, is somehow limited by its size (N = 40) which is insufficient even in few-shot learning scenarios. In the follow-up study, Bhuiyan et al. (2020) gathered a dataset of over 4,000 credibility assessments taken from 2 crowd groups (journalism students and Upwork workers) and 2 expert groups (journalists and scientists). Similarly as in the previous case, a varied set of 50 news articles related to climate science were selected. News articles were annotated by both crowd and expert groups on a 5-point Likert scale. The in-depth analysis of the obtained annotations revealed differences in annotation not only between crowd and expert groups, but also within expert groups between journalists and scientists due to differing expert criteria that journalism versus science experts use. Following the observations, authors proposed directions how to better design crowdsourcing of content credibility. El Ballouli et al. (2017) collected 17 million tweets in a period of two weeks that were written in Arabic and contained at least one hashtag. After preprocessing and grouping tweets by their hashtags, a topic-independent subset of 9,000 tweets was selected. Each of these tweets was annotated by three annotators on a binary scale (credible, non-credible), while the majority vote was used to determine the final label. Due to a lack of sufficiently large credibility-annotated datasets, some authors (e.g., (Wu et al., 2020; Podgurski et al., 2022; Leite et al., 2024)) decided to use available false information (fake news) datasets, such as LIAR (Wang, 2017), Weibo (Wu et al., 2020), FakeNewsNet (Shu et al., 2020), FakeNewsAMT (Pérez-Rosas et al., 2018) or Celebrity dataset (Pérez-Rosas et al., 2018). In such a case, various false information labels are mapped to credibility labels by taking an assumption that a false content is considered to be non-credible. Qureshi and Malick (2023) also proceeded from the FakeNewsNet dataset, however, only to identify a potential set of tweets that were further manually annotated by 12 experts for credibility on a 5-point scale. The resulting dataset consists of 4,958 tweets, out of them some were discarded due to low annotators’ agreement. Another group of datasets used in the existing works relates to credibility annotated at a source (webpage) level. Microsoft Credibility (Schwarz and Morris, 2011) dataset consists of top 40 search results on 25 pre-defined queries on the topics of Health, Politics, Finance, Environmental Science, and Celebrity News, which were manually annotated for credibility on a 5-point scale. Another dataset, Content Credibility Corpus (C3) (Kakol et al., 2017), consists of 15,750 evaluations of 5,543 pages by more than 2,000 annotators recruited through Amazon Mechanical Turk. The textual credibility signals annotated in this dataset include readability, language quality, informativeness, completeness, and objectivity. Finally, at the topic/event level, Mitra and Gilbert (2021) introduced a large scale dataset containing 60M tweets collected during a period of more than three months and grouped into 1,049 real-world events, each annotated by 30 human annotators on a 5-point scale. Table 2. Selected datasets used in the works addressing automatic credibility assessment 4.2. Methods and models The methods and models used in the research works falling into this group typically employ a methodology consisting of three steps. Firstly, a set of credibility signals is selected. Secondly, various techniques are employed for their automatic detection. Finally, detected credibility signals are then used to predict an overall credibility which can be subsequently utilized for example as an input to information retrieval engine. In the following description, we provide an in-depth description of each of these three steps. Credibility signals selection. In the earliest works, credibility signals were rather approximated by various (mostly shallow) linguistic features that can be easily detected by the machine, such as a number of words or exclamation marks. While being potentially helpful for traditional machine learning algorithms, people would not usually use such features for credibility assessment due to the absence of their direct interpretability. Weerkamp and De Rijke (2008) adopted a subset of 11 such features for blog posts proposed in (Rubin and Liddy, 2006) while focusing on those that are text-based and can be easily detected automatically, such as capitalization, shouting, spelling or post length. Similarly, Kang et al. (2012) opted for 12 numeric and 7 binary content-based credibility indicators for Twitter, such as a positive sentiment factor or a number of mentions. Approximately half of these indicators were adopted from the previous work by Castillo et al. (2011). El Ballouli et al. (2017) selected 48 credibility signals, out of them 26 signals were content-based. Besides shallow linguistic features, as already included in the previous works (e.g., count of hashtags, count of unique words, count of exclamation/question marks) the authors utilized also more advanced sentiment extraction to assess positive sentiment, negative sentiment, and objectivity. A shift towards more complex credibility signals is also present in the work by Qureshi and Malick (2023). Authors initially complied a list of 33 potential credibility signals. By proceeding from the initial experiments, a final set of 11 content- and context-based signals were used. In the case of content-based signals, shallow features (such as a number of hashtags, length of text) were discarded in favor of more complex signals like, post hate, informativeness, or deception. For the purpose of website credibility evaluation, Esteves et al. (2018) selected 15 content-based indicators by proceeding from the taxonomy created by Olteanu et al. (2013), such as authority (authoritative keywords within the page HTML content), readability metrics, text category or sentiment. In contrast to works mentioned so far, the following works selected signals from the list created by the Credible Web Community Group (see Section 2.2). At first, Podgurski et al. (2022) proposed a modular credibility assessment system of web pages using the 23 most frequent content- and context-based signals. The employed signals range from naive numerical and linguistic features, such as a number of external links, linguistic statistics (average text and word count, overall word count, etc.), spelling and grammar, domain endings and author information, to more complex features, such as sentiment, readability and presence of a clickbait title. Similarly, Leite et al. (2024) employed 18 credibility signals, including advanced ones that have not been considered in the previous works, such as call to action, impoliteness, sensationalism or explicitly unverified claims. Credibility signal detection. Early shallow linguistic features were detected primarily by simple rule- and heuristic-based techniques (Weerkamp and De Rijke, 2008; Kang et al., 2012), such as a predefined lexicon of positive and negative words, counting/searching for specific characters/patterns (e.g., capital letters, emoticons). By proceeding from such superficial linguistic features to more advanced credibility signals, also their detection methods became more complex. For sentiment analysis various lexicon and rule-based sentiment analysis were employed, such as Vader for English (Esteves et al., 2018; Podgurski et al., 2022), or ArSenL for Arabic (El Ballouli et al., 2017). For text category classification, well-known approaches employed in other NLP tasks were adopted, such binary multinomial Naïve Bayes (NB) classifiers or Latent Semantic Analysis (LSA) (Esteves et al., 2018). Various NLP libraries were used for automatic detection of additional credibility signals, such as LanguageTool library for detection of grammar & spelling errors or TextBlob for subjectivity detection (Podgurski et al., 2022). Even further, Qureshi and Malick (2023) detected the selected credibility signals by means of the classification methods proposed in the previous research works. Surprisingly, adoption of LLMs for detection of credibility signals is only very rare despite a great potential of LLMs to target a wide range of various credibility signals categories. To this end, Leite et al. (2024) used 3 instruction-tuned LLMs (GPT-3.5-Turbo, Alpaca-LoRA-30B and OpenAssistant-LLaMa-30B) with a specific prompt designed for each credibility signals (e.g., “Does the article make use of sensationalist claims?”). Credibility label/score prediction. To estimate overall credibility from the credibility signals, Weerkamp and De Rijke (2008) opted for simple weighting schemata. The resulting credibility score was subsequently incorporated into a blog post retrieval model. Kang et al. (2012) proposed three computational models (based on a weighted combination, and on a probabilistic language-based approach) for assessing tweet’s credibility, using not only content-based but also social and hybrid strategies. Experiments on the test set consisting of 1,023 instances (while using 10-fold cross-validation) revealed that social model was able to outperform content-based and hybrid model. This can be explained by a short textual content of tweets, as well as very shallow linguistic and possibly noisy credibility signals, such as a presence of exclamation mark. Podgurski et al. (2022) combined individual signals’ subscores with signal weights through a linear combination function. Signal weights were determined by authors empirically by considering prior research results, signal measurement accuracy, and experimental calculations on test data. By advancing from simple weighting and heuristics, consequent works started to employ feature engineering with traditional machine learning approaches. El Ballouli et al. (2017) trained a random forest decision tree classifier on the top of detected credibility signals – each serving as a feature. To predict website credibility score, Esteves et al. (2018) used Gradient Boosting and AdaBoost classification algorithms on the top of Microsoft Credibility and Content Credibility Corpus (C3). For the purpose of feature selection, Qureshi and Malick (2023) utilized the Kolmogorov–Smirnov (KS) test to identify discriminating features. Subsequently, a wide range of machine learning algorithms (12 regression and 10 classification ones) were used to predict original 5-scale as well as simplified binary credibility label. Results revealed that context-bases signals, such as high user influence and medium-to-high spread count, are very indicative of credible tweets. Credible tweets are normally spread by topic experts (the topic of the tweet is in the top 3 topics for that user), typically do not contain deceptive information and are not posted by malicious accounts. In terms of content-based signals, credible tweets have high informativeness and tent to not contain hate speech or links to low-credible sources. Analogically to other ML/NLP areas, also credibility assessment underwent a shift towards deep learning methods. In this direction, Wu et al. (2020) introduced an ANSP model based on adversarial networks and multi-task learning to capture differential credibility features for information credibility evaluation. The input to the model consists of the concatenations of word embeddings (Word2Vec) and meta-data embeddings (various credibility signals provided by the datasets). Evaluation on English and Chinese datasets, LIAR and Weibo respectively, showed that the combination of all features performs best for both datasets, while content-based features were consistently outperformed by contextual ones. Among these, some signals are particularly powerful on their own, such as the speaker, state info, party affiliation, or credit history. Finally, Leite et al. (2024) used Prompted Weak Supervision (PWS) on the top of LLM-predicted credibility signals. This approach was compared with unsupervised and supervised baselines. In the supervised finetuning scenario, a RoBERTa-Base model was finetuned with the ground-truth labels. In the zero-shot scenario, an instruction-tuned LLM based on LLaMa2 was prompted without any signals or in-context examples. Experimental results on four datasets (FakeNewsNet - GossipCop, FakeNewsNet - PolitiFact, FakeNewsAMT, and Celebrity) showed that Prompted Weak Supervision outperformed the zero-shot baseline by $38.3\%$ , and achieved $86.7\%$ of the performance of the state-of-the-art supervised baseline. Furthermore, in cross-domain settings, where the domain of the train set differs from the domain of the test set (e.g., Politics and Gossip), Prompted Weak Supervision outperformed the supervised baseline by $63\%$ . Lastly, the authors study the association between credibility signals and veracity through (i) a statistical test, where $12$ out of the $19$ signals were shown to have an association with veracity, and (ii) an ablation study, in which signals were individually removed from the train set in order to inspect their contribution to the model’s performance. This showed that the method’s strength is in the combination of a wide range of signals rather than relying on a small set of signals that could be strong predictors of veracity. Through these analyses, the authors verified that some credibility signals are domain-specific. For example, signals such as Source Credibility and Misleading about Content improve performance mostly for the political domain, while others such as Expert Citation and Call to Action show benefits in entertainment news. In addition to the efficiency of this approach in terms of being independent of the costly annotation of long news articles with each signal, the main advantage of this method is generalisation capability. The authors found that their approach outperforms the fine-tuned methods on the out-of-domain test data. This result is particularly important in the constantly-evolving online environment, when new non-credible and false information emerge every day, making it impossible to have up-to-date annotated data. 4.3. Tools and services While several services dedicated to credibility assessment exist, many of them are on a border with false information detection. From 2019, the Credibility Coalition maintain a CredCatalog https://credibilitycoalition.org/credcatalog/ – a catalogue of initiatives that have a stated aim to improve information quality. Besides various organizations (like fact-checking or academic institutions), it provides overview of relevant tools. Besides that, there are commercial projects supporting end-users in evaluating the credibility of the online content directly in their browsers. NewsGuard https://www.newsguardtech.com/ provides News Reliability Ratings for news outlets based on nonpartisan journalistic criteria. Similarly, GroundNews https://ground.news/ rate news stories according to their bias distribution (left-/right-leaning bias) or factuality (in the meaning of source reliability and general factual accuracy). The labelling is, however, done rather indirectly through sources writing about such news stories, instead of analysing a content of individual news articles themselves. Another relevant tool is the Tanbih (Zhang et al., 2019) – a news aggregator that performs media-level and article-level analyses, aiming to help users better understand the content they consume. It organizes news articles into event-based clusters and generates media outlet profiles, which include indicators such as the general factuality (in the meaning of factual accuracy) of reporting, the degree of propagandistic content, hyper-partisanship, leading political ideology, overall framing, and the stance of the outlet toward various claims and topics. Finally, an automatic analysis of several credibility signals (persuasion techniques, subjectivity, or machine-generated text among others) is available as a part of the Assistant tool in the well-known Verification plugin https://www.veraai.eu/category/verification-plugin developed and further enhanced as a part of EU-funded projects (InVID, WeVerify and vera.ai). The Assistant tool allows users to provide a URL or a local file as input, from which the text is extracted and various text analysis are performed, credibility signals being a recent addition to them. 4.4. Discussion Limited research focus with absence of LLM-based solutions. As already shown in Figure 5, the number of works explicitly addressing automatic credibility assessment is considerably lower in comparison with other individual signal categories. This is in contrast to false information detection, which attracted a plethora of research (and also practitioners’ and public) attention. In parallel, our literature survey pointed out a lack of approaches utilizing LLMs. At the same time, LLMs provide a great opportunity to automatize detection of multiple credibility signals even in a zero-shot settings, as the work by Leite et al. (2024) clearly demonstrated. Even further improvement in the terms of accuracy can be achieved by employing in-context learning, or instruction-tuning LLMs. While such LLM-based approaches would inquire higher computational costs, existing Parameter-efficient Fine Tuning (PEFT) techniques (Xu et al., 2023) may be employed if necessary. Lack of multilingual datasets with annotated credibility signals. Similarly to a limited research focus, also the situation with credibility-annotated datasets is falling behind false information research. There are very few datasets with manual (human) annotations at the level of individual credibility signals as well as overall credibility (Zhang et al., 2018; Bhuiyan et al., 2020). Unfortunately, these datasets are very small, which prevents the use of language models even in few-shot settings. Despite the declared original intentions to extend them in future, to the best of our knowledge, no follow-up datasets have been created so far. The remaining datasets are larger (containing hundreds or thousands of instances), nevertheless, annotated only at the level of overall credibility (see Table 2). Absence of datasets providing annotations of multiple categories of credibility signals for the same set of instances, prevents multi-task training of models. Multi-task approaches hold considerable potential, particularly because many credibility signals inherently share underlying similarities that can enhance model performance, as already demonstrated by the results of Prompted Weak Supervision (PWS) in (Leite et al., 2024). Robustness of credibility signals. For reliable credibility assessment, it is necessary that credibility signals and their relation to overall credibility remains the same in time, or at least is not influenced by significant domain and data drifts. To shed more light on the robustness of credibility signals, Ji et al. (2023b) evaluated how the signals and their prediction capability change over time. By utilizing the posts from the Weibo platform, two topic-specific datasets were created, one for climate change and second one for genetically modified organisms (GMO). The authors analysed how credibility signals evolve over time by splitting these datasets year-wise. They found that certain signals remain more stable over time, however, this stability is topic-dependent. For example, post sentiment performed better for the climate change dataset in most of the years compared with other features, while the post topic was the best predictor for the GMO dataset. In general, results showed that content-based features remained effective across time. On the other hand, contextual features, such as activeness and gender, were not correlated with veracity in the climate change dataset, while popularity showed no significant correlation with GMO misinformation in any given year from 2010 to 2020. In summary, content-based signals (especially sentiment and topic) seem to be more robust and less prone to the temporal data drift compared to context-based signals. 5. Factuality, subjectivity and bias Within false information research, the term factuality has been used to refer to a variety of related problems, with no unified ontology to define its levels. For example, Nakov et al. (2024) equate factuality with factual accuracy (i.e., veracity) and propose a four-level ontology encompassing the claim, article, user, and source factuality levels. In this interpretation, factuality is seen not as as a credibility signal, but as the final verdict regarding truthfulness of information. In the context of large language models (LLMs), factuality is typically defined as a model’s ability to generate outputs grounded in established real-world facts (Wang et al., 2023b). This aligns with the Cambridge English Dictionary’s definition of factual as “based on or containing facts” https://dictionary.cambridge.org/dictionary/english/factual. From an NLP perspective, however, factuality is often approached differently. Rather than relying solely on external fact-checking, NLP literature tends to treat factuality as a contextual signal – reflecting the degree of certainty a speaker conveys about the occurrence of events (Li et al., 2020b; Cao et al., 2021; Qian et al., 2019; Li et al., 2024c; Rovera et al., 2025). As identified through the systematic literature review conducted as part of this work, factuality in this view functions as a credibility signal rather than an overall veracity assessment, expressing how confidently an event is presented as having occurred or not occurred. This more fine-grained perspective follows earlier work by Saurí and Pustejovsky (2009), which defines factuality as the extent to which events are portrayed as corresponding to real-world situations, hypothetical scenarios, or uncertain interpretations. Under this definition, the task of factuality detection can be used as the first step towards identifying check-worthy claims (discussed in Section 7), as only the events that the speaker is confident about typically form claims. In the rest of this section, we use the terms factuality and event factuality interchangeably. Where the meaning is not obvious from the context or could lead to ambiguity, we distinguish whether event factuality, factual accuracy or a different notion of factuality is meant there. The literature on event factuality traditionally distinguishes between individual event factuality detection (EFD) (Li et al., 2020b) and document-level event factuality identification (DEFI) (Cao et al., 2021) tasks. The main distinction between these two sub-tasks is in the scope of information concerned – while the EFD task considers individual event mentions, DEFI aims to aggregate several mentions of a certain event within a document. Subjectivity represents an indicator of the overall subjectivity (or objectivity) of information. The task of identifying subjective information is modelled as either a binary or a more fine-grained problem that distinguishes between various degrees of subjectivity. Over the past years, there have been many shared tasks that focus on specific types of subjectivity. For example, Piskorski et al. (2023) discriminate between objective reporting, opinionated news and satire. Derczynski et al. (2017) and Gorrell et al. (2019) organised the challenges for rumour detection. Irony (Derczynski et al., 2017) and sarcasm (Gorrell et al., 2019) is another common type of non-factual information that received close attention during the last years, with shared tasks and challenges dedicated to this problem. Finally, bias represents a similar but more complex phenomenon of imbalance in terms of opinions or facts. As highlighted by (Nakov et al., 2024), there is no one single concept of bias among scholars. However, in many cases bias is seen as a systematic favouring of a certain ideology when covering information (Waldman and Devitt, 1998). This can be expressed in deliberately withholding certain part of information that contradict the favourable point of view (Smith et al., 2001) or vice versa, specifically searching for the facts that cover information from a certain political or ideological viewpoint (Hassell et al., 2020). In some cases, even if the choice of information sources is unbiased, the presentation of information in those sources may be performed in a biased manner that highlights the importance of only certain facts. Some instances of biased presentation include framing (Piskorski et al., 2023; Entman, 2007), opinionated reporting style (Piskorski et al., 2023; Soroka, 2012) and even visual clues (Barrett and Barrington, 2005) used to distort the perception of information. 5.1. Datasets Table 3. Selected datasets used for event factuality detection. Table 4. Selected datasets used for subjectivity detection. Table 5. Selected datasets used for bias detection. Tables 3, 4, and 5 represent a summary of the papers that introduce datasets annotated for detecting event factuality, subjectivity, and bias respectively. As can be seen, the prevalent majority of the datasets are only available in English. Besides English, there are datasets in Urdu (Mukund and Srihari, 2010), Arabic (Mourad and Darwish, 2013) and German (Atalla et al., 2011; Aksenov et al., 2021). Furthermore, multilingual subjectivity detection was a part of shared tasks at CheckThat! Labs of CLEF 2023 (Barrón-Cedeño et al., 2023) and CLEF 2024 (Barrón-Cedeño et al., 2024), covering Arabic, German, English, Italian, and Turkish languages. The ontology proposed by Qian et al. (2019) is the most widely accepted classification for detecting event factuality, with subsequent extension of the dataset with document-level event factuality annotations (Qian et al., 2022). More recently, Li et al. (2024c) built the first large-scale annotated dataset based on this classification, by using LLM predictions with human judgments as a final step. MPQA opinion corpus (Wiebe et al., 2005) is a particularly widespread benchmarking dataset for subjectivity detection, appearing in the majority of studies covered by the systematic review (Lin et al., 2011; Wiegand and Klakow, 2011; Wang and Liu, 2011; Biyani et al., 2014). The dataset is manually annotated with three frames, objective, expressive subjective and direct subjective. The dataset contains span-level annotations along with the annotation of the source (author, specific person, etc.) who expresses the subjective frame and the intensity of subjectivity. Direct subjective expressions are typically more explicit than expressive subjective. Originally created in English for detecting a phrase-level subjectivity, it is widely used in multilingual tasks by utilising parallel translations into other languages, such as Arabic, French, German, Romanian and Spanish (Banea et al., 2010; Mogadala and Varma, 2012). In addition, some of the datasets adopted the ontology of MPQA to create comparable corpora in other languages (Atalla et al., 2011). In addition to the datasets for bias detection provided in Table 5, Wessel et al. (2023) introduced Media Bias Identification Benchmark (MBIB) collection, which is the most comprehensive set of the benchmark corpora for media bias detection consisting of 22 datasets. The types of biases covered include hate speech, lexical, contextual, linguistic, gender, cognitive, racial and political biases. 3 out of 22 datasets, however, represent a more high-level annotation of information into fake and trustworthy rather than biases. 5.2. Methods and models The methods discussed in this section can be categorized into three groups – event factuality, subjectivity and bias/framing detection techniques. The methods in the first group can be further divided into two subcategories: (i) those targeting the factuality of individual event mentions and (ii) those assessing the document-level factuality of events. Event factuality detection (EFD). The methods falling into this category aim to identify the author’s level of certainty regarding the possibility of the individual mentioned event (Li et al., 2020b; Qian et al., 2015; Cao et al., 2022; Yi et al., 2022; Lee et al., 2015). The most common architectures for event factuality detection are LSTM and bi-LSTM models trained on BERT representations (Li et al., 2020b; Cao et al., 2022; Yi et al., 2022). A few studies also employ traditional machine learning approaches, such as SVM with LOSSO regression (Lee et al., 2015) and a combination of rule-based and maximum entropy methods (Qian et al., 2015). All the reviewed methods for event factuality detection are trained on either English (Qian et al., 2015; Lee et al., 2015) or Chinese (Li et al., 2020b; Yi et al., 2022), or a combination of both languages (Cao et al., 2022). Document-level event factuality identification (DEFI). Sentence-level event factuality often results in conflicts within a document, as different mentions of the same event may reflect varying degrees of factuality. Therefore, the methods in this sub-group aim to conclude the overall event factuality in a document based on the various sentence-level factuality values within that document. Cao et al. (2021) propose an Uncertain Local-to-Global Network (ULGN) that makes use of two important characteristics of event factuality, local uncertainty and global structure. Similarly, Qian et al. (2019) address the challenge of multiple event factuality values within a document by employing an LSTM model trained with both intra- and inter-sequence attention mechanisms to assess document-level event factuality. The model incorporates two types of input features, syntactic and semantic. Syntactic features are based on dependency paths from negative or speculative references to the event, while semantic features are derived from the sentences containing the event. Another approach to estimating document-level factuality is the Sentence-to-Document Inference Network (SDIN) proposed by Zhang et al. (2023). This architecture features a multilayer interaction network that aggregates individual event mentions into a global prediction. The last step employs gated aggregation that uses a sigmoid function to generate a mask vector that captures the most critical semantic and factual features of the event mentions. The training process applies a multi-task learning approach, where individual and document-level event factuality prediction tasks share the same pretrained model and interaction network. As an input, the model receives all the event mentions encoded into BERT representations at the final hidden state of [CLS] token. This approach makes it possible to significantly outperform the models by Cao et al. (2021) and Zhang et al. (2023) described above on Chinese and English DLEF corpora (Qian et al., 2019). Qian et al. (2022) propose an approach called Document-level Event Factuality identification via Machine Reading Comprehension Frameworks with Transfer Learning (DEFI-MRC-TL). The authors use BERT as a backbone model to train on a number of large-scale MRC corpora and fine-tuned on the DEFI task. As a target dataset, Qian et al. construct their own MRC-style DEFI corpus called DLEFM by annotating both events and document-level event factuality. They achieve significantly higher results on their dataset than the models by Cao et al. (Cao et al., 2021) and Zhang et al. (Zhang et al., 2023). Subjectivity. Subjectivity detection is a well-established NLP task that emerged long before the advent of deep learning and transformer models. As a result, the vast majority of the reviewed methods rely on traditional machine learning models, such as Multinomial Naïve Bayes, Convolution Kernels, Support Vector Machines, Logistic Regression, Latent Dirichlet Allocation and Decision Trees over lexical, syntactic and semantic features (Biyani et al., 2014; Lin et al., 2011; Cortis and Davis, 2022; Wiegand and Klakow, 2011; Wang and Liu, 2011; Banea et al., 2010; Atalla et al., 2011; Mukund and Srihari, 2010; Mourad and Darwish, 2013; Mogadala and Varma, 2012). Rule- and lexicon-based approaches are also common for this task (Ansar and Goswami, 2021; Wiebe et al., 2004; Hammer et al., 2014; Jeronimo et al., 2020; Maks and Vossen, 2012). The only transformer-based method identified through the systematic review is that by Savinova and Del Prado (2023). The authors test the efficiency of RoBERTa model in predicting subjectivity. They found a very high correlation with the human annotators and a significantly better performance than the existing rule-based and regression methods. Some studies formulate the task of subjectivity detection as a part of the sentiment analysis task, where the “neutral” sentiment corresponds to the non-subjective class, and both “positive” and “negative” sentiments indicate subjective information (Chaturvedi et al., 2018). Barbosa and Feng (2010) explore traditional ML approached to perform the tweet sentiment prediction as a two-step approach. The authors first distinguish between subjective and non-subjective tweets, and then further perform “positive” and “negative” tweet prediction for subjective tweets. Finally, Lu et al. (2024) address the task of sarcasm detection, which is often seen as a specialised type of subjective information. The authors propose a novel multimodal Fact-Sentiment Incongruity Combination Network (FSICN) approach that integrates the factual similarity and the sentiment information into the sarcasm detection task. The FSICN method incorporates a dynamic connection component that identifies the most relevant image-text pairs to detect fact incongruities between them. Bias. The methods used for the task of bias detection can be categorized based on the type of bias concerned. At a high level, the task can be approached as a binary classification of biased vs non-biased information. The methods falling into this group typically involve traditional machine learning models and lexicon-based methods (Spinde et al., 2021d; Aggarwal et al., 2020; Hamborg et al., 2020; Lazaridou et al., 2017; Yan et al., 2019). Over the last years, transformer models and LLMs became state-of-the-art approaches towards this task (Spinde et al., 2021b; Lin et al., 2024b; Wessel and Horych, 2024; Liu et al., 2019a; Fan et al., 2019; Maab et al., 2023; Krieger et al., 2022; Benson and Cruickshank, 2024). More fine-grained bias analysis methods are predominantly focused on detecting the political leaning bias. Chen et al. (2020) explore various levels of textual granularity (word, sentence, paragraph and discourse level) for political bias and unfairness detection. Their main method consists of a recursive neural network and uses GloVe embeddings (Pennington et al., 2014) as an input. According to their analysis on the bias locations, last paragraph typically contains the most biased text segment, while all biased articles start with a neutral tone. Baly et al. (2019) model the task of political leaning bias detection jointly with the task of general trustworthiness detection. The authors use multi-task ordinal regression framework as the main model and consider 7 scales of political leaning. They found that each of the two task benefits from the joint training. Sakketou et al. (2022) look into the task of detecting user bias. They identify potential spreaders of biased information by applying Graph Attention Networks (GAT) over User2Vec embeddings. The latter is obtained through applying the SentenceBERT (SBERT) (Reimers and Gurevych, 2019) over the user’s history of posts. The majority of recent approaches for political bias detection are based on transfer learning using transformer models (Aksenov et al., 2021; Baly et al., 2020a, b, 2018; Ness et al., 2023; Tran, 2020). Another group of fine-grained bias detection methods concerns the idea of framing, where frames are seen as perspectives used to discuss the same topic. While some of the frames can represent political leaning, they can also be seen as discussion sub-topics. Liu et al. (2019a) analyse the dataset of news headlines on the topic of gun violence in the United States. Within this topic, they predict 9 discussion frames, such as mental health, politics, ethnicity, economical consequences, legal regulations, gun rights, public space safety and society. The authors found that BERT model significantly outperforms deep learning approaches, such as RNNs, LSTMs, Bi-LSTMs and Bi-GRU, for each of 9 frames. Finally, Wessel et al. (2023) introduce the Media Bias Identification Benchmark (MBIB) task of diverse bias detection, by unifying the existing datasets annotated with gender, political, cognitive, racial, textual and linguistic biases. They benchmark their dataset on ConvBERT, Bart, RoBERTa-Twitter, ELECTRA, and GPT-2 models. The authors found that individual bias detection tasks benefit from different models. For example, ConvBERT is better at predicting political, textual and cognitive bias, while ELECTRA model shows better average performance at racial and gender bias detection based on macro-average F1-scores. 5.3. Tools and services The in-depth analysis of the research papers covered by this survey revealed several publicly available tools and services used by the NLP methods aimed at bias and factuality analysis. Media Bias/Fact Check (MBFC) website https://mediabiasfactcheck.com/ is a widely used service for media factuality (combining factual accuracy with other factors) and bias detection. It provides systematic human-centered annotations, spanning over 8 years and covering over 2,000 news websites. The annotators affiliated with International Fact-Checking Network (IFCN) assess factuality based on 4 factors: failed fact checks, sourcing, transparency, and one-sidedness/omission, each rated on a scale of 0 to 10. The final factuality score (also on a scale of 0 to 10) is a weighted average of these factors. This score is further mapped into a 6-level scale, indicating a “very low”, “low”, “mixed”, “mostly factual”, “high” and “very high” rankings. To detect bias, the annotators perform an aggregation of 4 factors: biased wording, factual sourcing, story choice leaning (e.g., pro-liberal or pro-conservative), and political affiliation. Each of these categories is rated by experts on a scale of 0 to 10, from least to most biased. The final score is then calculated as a sum of the four respective scores and is mapped to the categories indicating a range of categories between extreme and moderate left and right bias and unbiased/centered reporting. The original goal of the service was to educate the public on media factuality and bias and on deceptive news practices. In addition, the API makes the tool highly useful for NLP classification tasks (Baly et al., 2018, 2020a; Barrón-Cedeño et al., 2023; Aires et al., 2020b; Sales et al., 2021; Kangur et al., 2024). AllSides Media Bias Ratings https://www.allsides.com/ is a service similar to MBFC in terms of covering the political leaning bias based on a scale from right to left. The source coverage is also comparable to MBFC, with over 2,400 websited analysed. However, unlike MBFC, the judgment is not limited to a closed group of experts, but also includes ordinary people across the political spectrum trained to perform the analysis. This provides useful insights into how people with certain political leaning biases rate media news without knowing the source. Additionally, there is an option of a community feedback to agree or disagree with the provided rating. The labelled data from AllSides has been used in a number of classification tasks for political bias detection (Baly et al., 2020a; Spinde et al., 2021a; Li and Goldwasser, 2021; Spinde et al., 2021a, d, c). Ad Fonted Media https://adfontesmedia.com/ provides an interactive media bias chart where media sources are annotated based on two different dimensions, reliability and political bias. The bias dimension consists of most extreme left/right, hyperpartisan left/right, strong left/right, skews left/right and middle or balanced bias. The reliability dimension consists of numerical scores that are mapped to 8 categories: fabricated news, misleading information, selective or incomplete story/propaganda, opinionated of highly varied reliability news, simple fact reporting, and high effort original fact reporting. This resource is also commonly used by NLP approaches to analyse media bias with various degrees of granularity (Chen et al., 2020; Jaradat et al., 2024). Finally, The University of Sheffield as a part of their GATE Cloud infrastructure (Cunningham et al., 2002) provides public tools for multilingual news genre and framing detection based on its submission (Wu et al., 2023) to SemEval 2023 Shared Task 3 (Piskorski et al., 2023). The models used in these tools achieved competitive results in the shared task, placing in top 3 for most of these languages. The genre detection tool https://cloud.gate.ac.uk/shopfront/displayItem/news-genre-classifier tackles the task of classifying the news into opinionated, objective and satire. The framing detection tool https://cloud.gate.ac.uk/shopfront/displayItem/news-framing-classifier allows the analysis of the 9 principal frames used to present the information. 5.4. Discussion As noted from the survey results, the tasks of factuality, subjectivity and bias detection cover various subtasks and task formulations. Among the subtasks, while subjectivity detection distinguishes between subjective and non-subjective content, certain formulations also perceive it as a sub-task of sentiment analysis. In turn, under certain formulations, sentiment detection tasks can be seen as a specific case of bias detection. As an example, distinction between neutral and positive/negative sentiment is a type of subjectivity detection task. In turn, positive and negative sentiment towards a certain topic can be seen as a biased representation of information. Lack of unified definitions and approaches. One of the main challenges of this category of credibility signals is a high diversity in how researchers perceive and address factuality, subjectivity and bias, resulting in many definitions and ontologies. This problem strengthened by the inherent subjective nature of the subjectivity/bias is present already in dataset annotation causing also a low inter-annotator agreement (Spinde et al., 2023). Additionally, the problem of subjectivity detection is often seen as a sentiment detection (Mourad and Darwish, 2013) or a hate speech detection task (Lin et al., 2022). The problem of bias detection is sometimes seen as a specific case of fake news detection, where the biased presentation of facts is automatically seen as false information (Lin et al., 2022). The problem of factuality detection is sometimes seen as the degree of certainty of a speaker regarding the occurrence of events, e.g., (Li et al., 2020b; Cao et al., 2021; Rovera et al., 2025) and sometimes as factual accuracy (veracity) (Nakov et al., 2024) or a mixture of veracity and other credibility signals (as is the case of the MBFC service). Finally, factuality, subjectivity and bias is a cumulative notion that is highly nuanced depending on how balanced the opinionated information is and how it is quoted and presented in text (Piskorski et al., 2023). Data scarcity. The biggest challenge in multilingual subjectivity and bias detection is data scarcity. Many English studies are US-centric and only few datasets are available in other languages. For this reason, the multilingual datasets are not well-representative for benchmarking models in global analyses and multicultural settings, since bias detection requires cultural context and background knowledge about a country’s political spectrum. For instance, the concepts of right- and left-leaning media can be different across European or Middle East media. Focus on binary classification. Despite the explanatory advantage of a fine-grained approach preferred by media professionals, the majority of the models are trained to perform the binary classification of texts into biased and non-biased. This can be, at least partially, a result of the scarcity of datasets available for a fine-grained classification (as illustrated in Table 5). Adoption of LLMs. The use of large language models for detecting factuality, subjectivity, and bias is still in its early stages, largely due to two key limitations: the implicit biases embedded within LLMs themselves, and their tendency to hallucinate or generate factually incorrect information. Recently, the task of subjectivity detection has been the focus of two large-scale studies examining it from the perspective of large language models. Suwaileh et al. (2025) conducted a comprehensive benchmark of both mono- and multilingual models for subjectivity detection in Arabic sentences. Their findings show that LLMs outperform smaller multilingual fine-tuned models even in zero-shot settings. Performance further improves when LLMs are given few-shot examples. Similarly, Shokri et al. (2024) evaluated transformer-based and large language models on subjectivity detection across three diverse datasets and domains. While fine-tuned models generally outperformed zero-shot LLMs, the latter still achieved performance close to that of the smaller fine-tuned models. The authors also explored various prompting strategies and chain-of-thought techniques, observing that in-context learning is not consistently robust, as its effectiveness is highly sensitive to the specific examples used in the prompt. The task of bias detection has primarily been approached through the lens of framing analysis using LLMs. Pastorino et al. (2024) conducted the first benchmarking study of LLMs on framing detection, evaluating their performance in zero-shot, few-shot, and explainable prompting settings. The few-shot setting was further analysed using both in-domain and cross-domain examples. The authors found that prompting LLMs to generate explanations alongside predictions led to more robust and reliable results. While some LLMs performed particularly well in few-shot settings, the models often perceive emotional style as framing, which poses a significant challenge for reliably applying LLMs to this task. An additional challenge in applying LLMs to political bias detection is that LLMs themselves often exhibit inherent biases, which can shift in response to subtle changes in phrasing or context (Lunardi et al., 2024; Bang et al., 2024). Strategies such as prompt engineering and fine-tuning for debiasing (Lin et al., 2024a), as well as increasing model transparency regarding bias-related behaviour (Mohanty, 2025), may enhance the reliability of LLMs for future applications in this area. Mujahid et al. (2025) explored the application of LLMs to assess political bias and factuality, where factuality is interpreted as factual accuracy (i.e., veracity) rather than as a contextual credibility signal. The authors propose two prompt-based approaches: a handcrafted method and a systematic profiling method. In the handcrafted approach, LLMs are presented with a series of questions designed to elicit judgments about a domain’s stance on specific political figures or topics, as well as assessments of the domain’s overall trustworthiness. The systematic approach involves using LLMs to profile editorial political bias across 16 policy areas. In both cases, the models were also prompted to provide justifications for their judgments. The outputs generated by the LLMs were then used as features to train traditional machine learning models and transformer-based models for predicting degrees of factuality (i.e., factual accuracy) and political bias. The study also compared these prompt-based methods to a zero-shot baseline, in which the LLM receives only the domain name, or a domain name plus a summary of five articles from the domain. Results show that the supervised prompt-based approaches significantly outperform the zero-shot baselines in both bias and factuality (i.e., factual accuracy) prediction tasks. As mentioned above, factuality is also one of ubiquitous problems when evaluating the quality of LLMs. Research in this direction is related to how grounded LLMs’ replies are to real-world facts rather than applying LLMs to estimate factuality of information. Augenstein et al. (2024) summarised major challenges of large language models in this directions, which include unreliable citations, hallucinations when grounding information, domain-specific unreliability, incoherent responses, factual inaccuracies and difficulties performing deductive and inductive inferences. Additionally, factual accuracy of LLMs is affected by the knowledge gaps, which, depending on the information cut-off dates, can result in factually incorrect replies. These challenges become particularly dangerous when accompanied with confident tone and highly persuasive first-person replies. To address these challenges, the research community must consider a set of techniques, such as alignment of LLMs with certain values and safeguards, retrieval-based generation (RAG) to mitigate ungrounded replies, timely knowledge update and mitigation of hallucination. Finally, better evaluation metrics addressing the factuality (i.e., factual accuracy) of LLMs’ outputs need to be introduced to ensure the quality of the generated replies. Certain benchmarks have already been introduced to perform the factuality assessment of LLMs and facilitate reliable and accurate replies (Iqbal et al., 2024; Wang et al., 2024c). 6. Persuasion techniques and logical fallacies Over the past decade, propagandistic efforts have been widely used on social media platforms to shape public opinion and drive engagement on a massive scale. To address this issue, several computational methods aimed at automatic detection of such a content have been proposed (Da San Martino et al., 2021). Within the scope of propagandistic content, persuasion techniques and logical fallacies aim to deliberately influence others’ opinions using rhetorical and psychological mechanisms (Miller, 1939). Recently, the task of automatic detection of persuasion techniques has gained increased attention from the NLP research community, with several resources introduced over the last years. 6.1. Datasets The majority of existing datasets are designed to support two related tasks: (i) classifying the input text as either containing persuasion techniques or not (binary classification), and (ii) extracting all specific persuasion techniques present in the input text (multi-label classification). While the binary task provides a simpler framework, it lacks the necessary granularity for detailed analysis of propagandistic content. The multi-label task is a more complex extension, requiring the model to identify individual persuasion techniques and thus enabling a more nuanced evaluation. Importantly, these tasks are implemented at different levels of granularity across datasets – some annotate propaganda at the token-level (fine-grained), others at the sentence-level (coarse-grained), and some at the fragment-level (mid-grained), where continuous spans of text within a sentence are labelled. These distinctions significantly affect how models are trained and evaluated. For this reason, and due to its relevance for fine-grained analysis, we focus our analysis on the multi-label task. A summary of existing datasets and their key attributes, including annotation granularity, is presented in Table 6. Table 6. Selected datasets used for detection of persuasion techniques. The NLP4IF-2019 shared task (Da San Martino et al., 2019a) introduced a dataset for fine-grained propaganda detection consisting of news articles from $36$ propagandist and $12$ non-propagandist news outlets, and annotated with $18$ different propaganda techniques. Two subtasks were featured: sentence-level classification (binary task) and fragment-level classification (multi-label span identification task). Similarly, the SemEval-2020 shared task introduced the PTC-SemEval20 dataset (Da San Martino et al., 2020a), which considered the same techniques used in the NLP4IF-2019 task, however, certain techniques were merged or removed due to their low frequency: Red Herring and Straw man were combined with Whataboutism, and Bandwagon was merged with Reductio ad Hitlerum, while Obfuscation, Intentional vagueness, Confusion was removed entirely. In SemEval-2021 (Dimitrov et al., 2021), a dataset of multimodal persuasion techniques was introduced, containing Facebook posts with images and texts representing memes shared by users from 26 public groups. The visual modality provided additional context that was not present in the text modality alone. For instance, techniques such as Smears, Doubt, and Appeal to Fear/Prejudice appeared more frequently when considering the image along with the text. Other datasets were introduced for non-English languages. Baisa et al. (2019) introduced a dataset for detection of persuasion techniques in the Czech language with around $7,000$ news articles. The techniques considered in their dataset differed substantially from others, with the presence of domain-specific techniques such as ‘Russia’ (indicating that Russia was a topic discussed in the document). WANLP-2022 (Alam et al., 2022b), ArAIEval-2023 (Hasanain et al., 2023), and Almotairy et al. (2024) introduced datasets for persuasion detection in Arabic texts. The three Arabic datasets share several characteristics such as number of instances (between $2,000$ and $6,000$ ), content type (mainly tweets, with Hasanain et al. (2023) additionally containing news articles), and label scheme (at least $10$ techniques are shared between the three datasets). Macagno (2022) introduced the first multilingual dataset for detection of persuasion techniques, containing tweets in English, Italian, and Portuguese. However, the dataset is small ( $<3,000$ instances in total) for the purpose of training deep learning models, and similarly to (Baisa et al., 2019), the set of persuasion techniques considered differs substantially from other datasets. SemEval-2023 (Piskorski et al., 2023) introduced a large-scale multilingual dataset covering $9$ languages (English, French, German, Georgian, Greek, Italian, Polish, Russian, and Spanish), with almost $50,000$ news articles labelled with $23$ different persuasion techniques, therefore being the largest dataset currently available in terms of number of instances, languages, and persuasion techniques. Three languages were considered “surprise languages” (Georgian, Greek, and Spanish), for which training sets were not available during the competition, thus encouraging systems to deal with out-of-domain data. Furthermore, the set of $23$ persuasion techniques were grouped into $6$ coarse-grained categories. For example, the techniques of Loaded Language, Obfuscation, Intentional Vagueness, Confusion, Exaggeration or Minimisation, and Repetition, were grouped into the umbrella of Manipulative Wording. 6.2. Methods and models The majority of methods and models developed for the task of automatic detection of persuasion techniques were introduced in the shared tasks discussed in the previous section. The joint effort of multiple different attempts at producing the best-scoring system allows to identify which methodological decisions are key to producing accurate models to detect persuasion. Table 7 summarises the highest-scoring Shared task systems that did not publish a description of their approach are not considered in our analysis. systems across the shared tasks aimed at automatic detection of persuasion techniques. Table 7. Top-scoring systems for automatic detection of persuasion techniques across different datasets. Transformer-based models were employed in all top-scoring systems for automatic detection of persuasion techniques due to their ability to capture nuanced contextual information (Vaswani et al., 2017). In the NLP4AI-2019 shared task (Da San Martino et al., 2019a), $5$ out of the $6$ submissions included transformer-based models, while more recently in SemEval-2023 (Piskorski et al., 2023), all $16$ submissions were comprised of transformer-based models. In earlier shared tasks, BERT was initially preferred over other architectures such as RoBERTa, ALBERT, and DeBERTa, which were adopted more frequently later, specially RoBERTa. In SemEval-2019 BERT was used in all transformer-based submissions. In SemEval-2020 and SemEval-2021, BERT was used in a total of $25$ systems, while RoBERTa is used in $14$ systems. Nevertheless, systems using RoBERTa achieved the top $2$ submissions in both shared tasks (Morio et al., 2020; Jurkiewicz et al., 2020; Tian et al., 2021; Gupta et al., 2021a). For tasks in Arabic (WANLP-2022 and ArAIEval-2022), monolingual models (AraBERT and variations) outperformed multilingual models (e.g., XLM-RoBERTa and mBERT) (Hasanain et al., 2023). In SemEval-2023 where $9$ languages were available, multilingual models (e.g., mBERT and XLM-RoBERTa) outperformed monolingual models trained with each language separately (Wu et al., 2023). Unsurprisingly, the larger variations of the models generally outperformed the smaller versions. For reference, RoBERTa large has almost triple the size of RoBERTa base ( $355$ and $125$ million parameters, respectively). Also, model ensembles combining different pretrained models were widely and effectively employed, although at the cost of requiring more computational resources (Morio et al., 2020; Tian et al., 2021; Gupta et al., 2021a; Hossain et al., 2021; Chernyavskiy et al., 2020). In addition to the contextual embeddings generated by transformer-based models, some works have experimented with supplementary input features such as part-of-speech (POS) tags (Morio et al., 2020; Khosla et al., 2020; Roele, 2021), named entity recognition encodings (Morio et al., 2020; Khosla et al., 2020), word-level n-grams (Khosla et al., 2020; Kim and Bethard, 2020; Ghadery et al., 2021), character-level n-grams (Tian et al., 2021; Chernyavskiy et al., 2020), and sentiment scores (Khosla et al., 2020; Roele, 2021). However, apart from the contextual embeddings, it is not clear if other input representation methods play a significant role in improving performance. In fact, apart from Morio et al. (2020) who used named entity recognition encodings, all other 1st and 2nd placing systems in shared tasks have not used other supplementary features apart from the contextual embeddings. Arguably the most relevant methods employed in top-performing systems are aimed towards dealing with data skewness. Most datasets for this task suffer from class imbalance, meaning the majority of persuasion techniques are underrepresented, while a small set of persuasion techniques comprise the majority of the dataset. To deal with this issue, several different methods are employed, such as oversampling underrepresented techniques (Yoosuf and Yang, 2019; Ek and Ghanimifard, 2019; Pauli et al., 2023), scaling the contribution of the techniques to the loss function according to their proportion (i.e., class weighting) (Yoosuf and Yang, 2019; Morio et al., 2020; Abdel-Salam, 2023; Razuvayevskaya et al., 2024), and using supplementary data obtained with (i) data augmentation (Gupta and Sharma, 2021), (ii) semi-supervised methods (Jurkiewicz et al., 2020), or (iii) similar datasets (Tian et al., 2021; Abujaber et al., 2021; Hussein et al., 2022; Mittal and Nakov, 2022). 6.3. Tools and services In terms of production grade tools to detect persuasion techniques in texts, to the best of our knowledge, two services are currently available. The Propaganda Persuasion Techniques Analyzer (PRTA) (Da San Martino et al., 2020b) is a discontinued tool that is no longer available online. It was designed to detect instances of propaganda in texts by highlighting the spans where specific techniques were used. It provided users with the ability to compare texts based on their use of propaganda techniques, offering detailed statistics on the prevalence of these techniques, both overall and over time. PRTA used a BERT model trained on the SemEval-2019 dataset, fine-tuned for both fragment-level and sentence-level classification tasks. To gather data, PRTA crawled a growing list of over $250$ RSS feeds, Twitter accounts, and websites, extracting text via the Newspaper3k library and performing deduplication using a hash function. The system then identified sentences containing propaganda, organised the articles into topics (such as COVID-19 or Brexit), and allowed users to compare the use of propaganda techniques across various media sources. The tool also allowed for filtering based on time intervals, keywords, or political orientation of the media. Although the user interface of the PRTA tool is no longer available, the underlying API for propaganda detection (along with other credibility signals) remains accessible https://apihub.tanbih.org/propaganda/. The GATE Cloud infrastructure (Cunningham et al., 2002) operated by the University of Sheffield provides a service to detect $23$ different persuasion techniques across multiple languages using a multilingual BERT (mBERT) model https://cloud.gate.ac.uk/shopfront/displayItem/persuasion-classifier. The model was fine-tuned using data from SemEval-2023 in English, French, German, Italian, Polish, and Russian. Class weighting was applied during fine-tuning to account for the label skewness issue. The tool is capable of processing up to $1,200$ documents per day free of charge through its API, with an average processing rate of 2 documents per second. Researchers can request higher quotas if needed for larger-scale analysis. 6.4. Discussion Most commonly adopted persuasion techniques. Table 6 shows that $7$ out of the $10$ datasets present a similar set of persuasion techniques (Da San Martino et al., 2019b, 2020a; Dimitrov et al., 2021; Alam et al., 2022b; Hasanain et al., 2023; Piskorski et al., 2023; Almotairy et al., 2024), while (Baisa et al., 2019; Lawson et al., 2020; Macagno, 2022) largely diverge from them. Considering these seven similar datasets, we observe a set of $12$ persuasion techniques that are consistent among them: Appeal to authority, Appeal to fear/prejudice, Causal oversimplification, Doubt, Exaggeration or minimization, Flag-waving, Loaded language, Name calling or labelling, Slogans, and Whataboutism appear in all seven datasets, while Repetition appears in six datasets (except (Almotairy et al., 2024)), and Reductio ad Hitlerum appears in five datasets (except (Hasanain et al., 2023; Piskorski et al., 2023)). Combining existing datasets with overlapping persuasion techniques could be a promising research direction to develop more robust models and benchmarking resources. For example, Tian et al. (2021) achieved 1st place in SemEval-2021 Task 6 by leveraging the dataset from (Da San Martino et al., 2019b) as additional training data. Label skewness. A common characteristic inherent to this label scheme is the significant skewness in distribution of the persuasion techniques. For example, in SemEval-2023 (Piskorski et al., 2023) (the largest multilingual dataset, with $50,000$ instances), a small set of $6$ techniques represent $71.8\%$ of the entire dataset - Loaded Language ( $18.5\%$ ), Name Calling-Labelling ( $23.7\%$ ), Casting Doubt ( $12.5\%$ ), Questioning the Reputation ( $7.6\%$ ), Appeal to Fear-Prejudice ( $4.8\%$ ), and Exageration/Minimisation ( $4.7\%$ ) - while the remaining $17$ techniques account for only $28.2\%$ of the dataset. The same trend can be seen in (Da San Martino et al., 2019b), with $6$ techniques representing $68.2\%$ of the dataset - Loaded language ( $34\%$ ), Name calling, labelling ( $17.3\%$ ), Repetition ( $10.2\%$ ), Exaggeration, minimization ( $7.6\%$ ), Doubt ( $7.5\%$ ), and Appeal to fear/prejudice ( $4.9\%$ ) - while the remaining $12$ techniques account for $31.2\%$ of the dataset. This sparse label distribution poses several challenges for training and evaluation. In particular, the overrepresentation of a few dominant techniques may lead models to bias their predictions toward these frequent categories, while underrepresented techniques suffer from insufficient training examples. This data imbalance increases the risk of overfitting to the few available samples of rare classes, potentially resulting in poor generalization. Evaluation. Shared tasks on persuasion techniques detection typically employ the F1-Micro measure as the official evaluation metric (Dimitrov et al., 2021; Da San Martino et al., 2020a; Alam et al., 2022b; Hasanain et al., 2023; Piskorski et al., 2023), however, the F1-Micro does not account for skewed class distributions, as opposed to F1-Macro Generally F1-Macro is also reported as a secondary metric, but for the purpose of the competition, the F1-Micro determines which system is best.. This evaluation strategy encourages models that excel at predicting a few densely distributed persuasion techniques. For example, in ArAIEval-2023 (Hasanain et al., 2023), the baseline model (majority vote classifier that always predicts the most common persuasion technique), achieves an F1-Micro of $0.3599$ , and an F1-Macro of $0.0279$ , representing a difference of $92\%$ between the two scores. Similarly, the best submission by Lamsiyah et al. (2023) achieves F1-Micro and F1-Macro scores of $0.5666$ and $0.2156$ , a difference of $62\%$ . Similar figures are seen in other shared tasks such as SemEval-2023 (Piskorski et al., 2023), with the average F1-Micro score for first place submissions across all languages resulting in $0.454$ , and the F1-Macro in $0.211$ , a difference of $53\%$ . These figures highlight how current state-of-the-art models trained for this task still lack the ability to predict underrepresented persuasion techniques effectively. Multilinguality. Despite efforts to create multilingual datasets such as SemEval-2023 (Piskorski et al., 2023) and Macagno (2022), most datasets focus on English or Arabic. For languages without annotated datasets, machine-translation and zero-shot classification approaches may be viable alternatives (if some level of noise introduced by machine translation is acceptable). Nevertheless, joint multilingual training often result in more accurate models. For instance, Razuvayevskaya et al. (2024) experimented training monolingual models with translated data from SemEval-2023, which improved performance only for English, but not for other languages. Therefore, multilingual datasets with wider variety of languages are still required to produce state-of-the-art models and evaluation benchmarks. Adoption of LLMs. Recent research has increasingly explored the capabilities of large language models in generating persuasive and deceptive content, particularly in the context of disinformation and propaganda (Simchon et al., 2024; Breum et al., 2024; Rogiers et al., 2024; Xu et al., 2024; Pauli et al., 2024). LLMs can manipulate existing factual content into misleading or false narratives at scale, often outperforming human-written disinformation in believability and fluency (Matz et al., 2024; Chen and Shu, 2023) and can even personalize the content to specific target groups (Zugecova et al., 2025). However, while the generation side has been widely studied, significantly less attention has been paid to using LLMs for detecting or countering persuasive techniques. Bridging this gap by harnessing LLMs for persuasion detection presents a promising but underexplored research direction. 7. Check-worthy and fact-checked claims Given the impossibility to verify the veracity of every single piece of information posted online, two credibility signals and corresponding tasks are particularly crucial in supporting fact-checkers and other actors involved in the fight against disinformation and misinformation (Guo et al., 2022): check-worthiness detection and the retrieval of previously fact-checked claims. The former is aimed at identifying which claims are check-worthy because of their relevance and interest to the general public (Hassan et al., 2015). The second, which ideally takes place after the first one, is meant to ease claim verification by checking whether the given claim was already verified before by searching in existing repositories of fact-checked claims (Shaar et al., 2020), both monolingual and multilingual. While these tasks are usually studied in separation as a part of the fact-checking pipelines, their outputs can be and also are used (usually in combination) as credibility signals to indicate whether the check-worthy (central) claims contained in a piece of content have been fact-checked and if so, what was the given veracity value or values. In fact, they are recognized as such both in the credibility signals list created by the Credible Web Community Group (signals ‘article has a central claim’ and ‘fact-check status of a claim’; see (Multiple authors and Hawke, 2019)) as well as in the related works (Zhang et al., 2018). The check-worthiness of a claim can be considered a content-based signal, while fact-check status of a claim is a context-based one, since it requires additional external sources to be detected. On the other hand, fact-checking itself is not considered a credibility signal per se – its aim is to directly ascertain the veracity value of a piece of information, while the former are only signals that the information might be check-worthy (and thus more attention is needed before the users make credibility judgment) or that it contains a claim that have already been fact-checked (but it does not by itself analyse the stance towards that claim, only its presence). It is also worth noting that these signals differ from factuality signal as it is understood in the surveyed works as discussed in Section 5). 7.1. Datasets Check-worthy claim detection is a popular task within the NLP community thanks to the series of CheckThat! shared tasks organised at CLEF, which led to the release of related datasets. Indeed, check-worthy claim detection is the only subtask that has been proposed at all seven CheckThat! editions (Atanasova et al., 2018, 2019; Barrón-Cedeño et al., 2020; Nakov et al., 2021, 2022a; Alam et al., 2023; Barrón-Cedeño et al., 2024). Through those editions, datasets for training check-worthy claim detection models have been constantly extended to cover additional languages, starting from English and Arabic at CheckThat! 2018 (Atanasova et al., 2018) to Arabic, Dutch, English and Spanish at CheckThat! 2024 (Barrón-Cedeño et al., 2024) and even multimodal and multi-genre data in the 2023 edition (Alam et al., 2023). Beside CheckThat!, additional datasets have been created and made available for research, as shown in Table 8. The sources covered by such datasets are typically social media, news and political debates, i.e. three relevant areas where the presence of disinformation may have detrimental effects on a large audience. Particular relevance was given to COVID-related content during the pandemic (Hadj Ameur and Aliane, 2021; Salek Faramarzi et al., 2023), when the continuous flow of misleading information could negatively affect public health. Although English is the most represented language, datasets for check-worthiness detection have also been developed in several other languages. This can be attributed to the fact that public relevance is both time- and geographically bounded, with claims typically referring to events specific to individual countries or regions. Table 8. Selected datasets used for check-worthiness detection. Note that the datasets used in the different CLEF CheckThat! editions often overlap. | Dataset | Language(s) | # Instances | Content type | Classes | | --- | --- | --- | --- | --- | | TR-Claim19 (Kartal and Kutlu, 2020) | Turkish | 2,287 | Tweets | Check-worthiness 26 rationale categories | | CW-USPD-2016 (Gencheva et al., 2017) | English | 5,415 | Political debates | Check-worthiness | | Dhar and Das (Dhar and Das, 2021) | Bengali, Hindi | 2,402 | Political news, Twitter | Check-worthiness | | MM-Claims (Cheema et al., 2022) | English | 3,400 | tweets (image + text) | Claim detection, check-worthiness, visual relevance | | Sheikhi et al. (Sheikhi et al., 2023) | Norwegian | 4,885 | News | Check-worthiness | | AraCOVID19-MFH (Hadj Ameur and Aliane, 2021) | Arabic | 10,828 | COVID-related tweets | Check-worthiness, factual, hateful | | Faramarzi et al. (Salek Faramarzi et al., 2023) | English | 7,017 | COVID-related tweets | Check-worthiness, claim extraction | | ClaimBuster dataset (Arslan et al., 2020) | English | 22,281 | Presidential debates | Check-worthiness, factual | | CLEF-2018 CheckThat! Lab Task 1 (Atanasova et al., 2018) | English, Arabic | 17,300 | Political debates | Check-worthiness | | CLEF-2019 CheckThat! Lab Task 1 (Atanasova et al., 2019) | English | 24,000 | Debates, speeches, press conferences | Check-worthiness | | CLEF-2020 CheckThat! Lab Task 1 (Barrón-Cedeño et al., 2020) | English, Arabic | 962 (EN) 7,500 (A), | Tweets, political debates, speeches | Check-worthiness | | CLEF-2021 CheckThat! Lab Task 1 (Nakov et al., 2021) | Arabic, Bulgarian, English, Spanish, Turkish | 18,014 (tweets), 50,123 (sentences) | Debates, speeches, tweets | Check-worthiness | | CLEF-2022 CheckThat! Lab Task 1 (Nakov et al., 2022a) | Arabic, Bulgarian, Dutch, English, Spanish, Turkish | 30,363 | Tweets | Check-worthiness, verifiable, harmful | | CLEF-2023 CheckThat! Lab Task 1 (Alam et al., 2023) | Arabic, English, Spanish | 70,806 | Tweets, political debates, speeches | Multimodal and multigenre check-worthiness | | CLEF-2024 CheckThat! Lab Task 1 (Barrón-Cedeño et al., 2024) | Arabic, Dutch, English, Spanish | 64,700 | Tweets, political debates, speeches | Check-worthiness | Concerning the retrieval of previously fact-checked claims, there are several datasets containing verified (fact-checked) claims collected from professional fact-checking organizations, such as X-Fact (Gupta and Srikumar, 2021), MultiFC (Augenstein et al., 2019) or ClaimsKG (Tchechmedjiev et al., 2019). Being usually collected for the task of automatic fact-checking, these are not directly usable for the task of retrieval of previously fact-checked claims by themselves, as they lack the input claims (e.g., social media posts) and the pairs of the input and the verified claims. The first datasets designed specifically for the task appeared in 2020 in the works by Shaar et al. (Shaar et al., 2020) and Vo and Lee (Vo and Lee, 2020) who independently prepared datasets based on both Snopes and PolitiFact. The former became the basis for a series of CheckThat! Lab shared tasks (Task 2) organised at CLEF in 2020 (Barrón-Cedeño et al., 2020), 2021 (Shaar et al., 2021) and most recently 2022 (Nakov et al., 2022b). The tasks gradually expanded the original dataset in size and also by adding an additional language (Arabic). The political debates part of the dataset was additionally expanded in (Shaar et al., 2022b). The datasets for retrieval of previously fact-checked claims are usually collected by using one of the following approaches: (i) looking into the fact-checking articles for links to the original content making the claim that is being verified, e.g., (Shaar et al., 2020; Barrón-Cedeño et al., 2020; Shaar et al., 2021; Nakov et al., 2022b; Pikuliak et al., 2023; Singh et al., 2024) or (ii) searching for (social media) content (such as discussion threads) that contains URL of or semantic links to the fact-checking articles, e.g. (Vo and Lee, 2020; Hardalov et al., 2022; Nielsen and McConville, 2022). The former typically achieves high precision at the cost of lower number of pairs and possible missing connections (i.e., many false negatives), while the latter can maximise recall at the cost of introducing noise unless manual checking is applied. Typical example of a high noise are CrowdChecked (Hardalov et al., 2022) or MuMiN (Nielsen and McConville, 2022) which both contain a large number of social media posts which distinguishes them from other datasets. While many available datasets focus on English only, there are several newer ones supporting other languages (e.g., Arabic (Shaar et al., 2021; Nakov et al., 2022b) or Spanish (Martín et al., 2022)) or even a range of languages (Kazemi et al., 2021; Pikuliak et al., 2023; Nielsen and McConville, 2022; Singh et al., 2024). The most notable among these are MultiClaim (Pikuliak et al., 2023) and MMTweets (Singh et al., 2024) due to the number of included languages, high precision of identified pairs and their amount as well as due to the fact that they both introduced a task of crosslingual retrieval, in which input claims are in different language than that of the verified claims. A full list of selected relevant datasets is reported in Table 9. Besides these, it is also worth mentioning two datasets focusing on COVID-related claims, namely CoAID (Cui and Lee, 2020) and MM-COVID (Li et al., 2020a); however, these focus on a different task of fake news/disinformation detection. A dataset of COVID-related tweets and fact-checks is also presented in (Jiang et al., 2023; Haouari et al., 2021). In this case, although the dataset was introduced for a different task, it could also be useful for previously fact-checked claim retrieval. Table 9. Selected datasets used for retrieval of previously fact-checked claims. 7.2. Methods and models To propose methods for check-worthy claim detection, existing works first define how check-worthiness is operationalized. In a recent survey, Panchendrarajan and Zubiaga (2024) identify two main aspects making a claim check-worthy: its verifiability and its priority. The first item refers to the possibility of determining the veracity of a claim, which can be likely supported by evidence. A claim is verifiable when it contains a factual statement that can be checked, which means that personal opinions or events presented as uncertain are excluded. Second, given that verifying all statements about the world is impossible, it is important to prioritize claims which are considered timely, interesting to the general public and whose verification might have a broader impact (Das et al., 2023). In this latter aspect it differs from a related, but a distinct task of claim detection which solely aims to identify what constitutes a claim in a text (either using a binary classification or by identification of spans) (Mittal et al., 2023; Wuehrl et al., 2023; Gupta et al., 2021b). Given this need for prioritization, check-worthiness detection has been cast as a ranking problem since the first editions of the CheckThat! Lab within the CLEF Evaluation initiative https://www.clef-initiative.eu/, which made the task well-known within the NLP community, although some related works had already been presented before (Hassan et al., 2015). In particular, given a political debate, the first CheckThat! task was aimed at predicting which claims should be prioritized for fact checking. This is reflected in the evaluation approach proposed for the CheckThat! series, which has become the de facto standard in check-worthiness detection: systems should output the list of input claims ranked by check-worthiness, which is usually evaluated using Mean Average Precision (MAP), reciprocal rank and $P@k$ for $k$ $∈$ $\{1,3,5,10,20,30\}$ . More recently, check-worthiness has been, however, evaluated using F1 as a binary classification task rather than ranking (Alam et al., 2023; Barrón-Cedeño et al., 2024). Regarding methods, in the CheckThat! editions up to 2023 state-of-the-art results for check-worthy claim detection were reached by methods that rely on fine-tuned transformer-based methods such as BERT, RoBERTa, DistilBERT, (Williams et al., 2021; Frick et al., 2023; Sawinski et al., 2023) and language-specific variants, (Alam et al., 2023), often combined with data manipulation or model ensembling strategies. For example, the best results on the English portion of the CheckThat 2022 dataset were achieved by a RoBERTa model that leveraged a back-translation-driven data augmentation process (Savchev, 2022). The 2024 edition, instead, has seen an increased interest in using also generative LLMs for the task such as LLama 2 and 3, GPT3.4 and 5, Mixtral and Mistral (Barrón-Cedeño et al., 2024). For instance, the best performing system on the English language dataset relied on fine-tuned Llama2 7b on the original training data, using prompts generated by ChatGPT (Li et al., 2024b). Nevertheless, despite the notable progress in data and methods for check-worthy claim detection, there is currently a lack of an extended coverage across languages and topics. Concerning previously fact-checked claims, the task is also formulated as a retrieval one, although the name of the task may vary across the literature; fact-checking URL recommendation (Vo and Lee, 2018), detection of previously fact-checked claims (Shaar et al., 2020), verified claim retrieval (Barrón-Cedeño et al., 2020), searching for fact-checked information (Vo and Lee, 2020), claim matching (Kazemi et al., 2021) or retrieval of previously fact-checked claims (Pikuliak et al., 2023) have all been previously used to denote it. Being a retrieval task, the existing methods apply one or a series of (re-)rankers and mean average precision (MAP), mean reciprocal rank (MRR), $P@k$ or a $Success@k$ ( $Hit@k$ ) are used as evaluation metrics. In case of a series of rankers, the works tend to use a baseline ranker that is easy to compute and has a good recall and then one or more subsequent rerankers that work over a progressively smaller subset of results retrieved by a previous ranker in the pipeline, see, e.g., (Shaar et al., 2020; Hardalov et al., 2022). The rerankers’ task is to improve precision by moving the relevant results to the top; since they are working with a smaller set of results, they can be more computationally demanding. Alternatively, the rankers could be used in combination as an ensemble to improve the precision at the cost of higher computational demands, but this is not observed in the surveyed works. In most of the works, BM25 (Robertson and Zaragoza, 2009) or similar information retrieval algorithms are used as a baseline. Various neural text embedding models are used as either sole rankers (e.g., in (Pikuliak et al., 2023)), rerankers (e.g., in (Shaar et al., 2020)) or as ensembles (Martín et al., 2022), especially sentence transformers (Reimers and Gurevych, 2019), which use Siamese networks to pre-train text representations, usually on various sentence similarity datasets. The approaches also use several other techniques to improve the retrieval performance, such as text embedding models fine-tuning (Pikuliak et al., 2023; Kazemi et al., 2021), distance supervision to work with noisy data (Hardalov et al., 2022), key sentences extraction (Sheng et al., 2021), extended context of the input and verified claims (especially for political debates (Shaar et al., 2022a)), extraction of text from images (Vo and Lee, 2020; Pikuliak et al., 2023), using multimodal representation combining text and images (Vo and Lee, 2020), or query (input claim) modification or rewriting to be more easily matched with the fact-checked claims (Bhatnagar et al., 2022; Kazemi et al., 2023; Sundriyal et al., 2023). The approaches may further differ by the use of loss, selection of negative examples and other (hyper-)parameters when fine-tuning the neural models. All surveyed approaches (including solutions submitted to the CheckThat! Lab challenge (Barrón-Cedeño et al., 2020; Shaar et al., 2021; Nakov et al., 2022b)) rely on smaller languages models, such as BERT, XLMRoBERTa, etc. One exception is the work of Sundriyal et al. (2023), where the authors employ LLMs to normalize the claims for the purpose of query rewriting for the retrieval task; however, they do not perform experiments on the retrieval task, thus focusing only on the first step (claim detection) in the pipeline. 7.3. Tools and services Given that automatizing check-worthiness detection can greatly support and speed up fact-checking activities, a number of systems has been already developed for the task, some of which are based on insights and databases actually used by fact-checking organisations. ClaimBuster (Hassan et al., 2017) was the first end-to-end system for computer-assisted fact-checking trained on a human-labeled dataset of check-worthy factual claims from the U.S. general election debate transcripts. The first component of the pipeline is a detector of check-worthy factual claims which, given a sentence, first labels it as being ‘non-factual’, ‘factual and unimportant’ or ‘factual and check-worthy’. In case of the latter, a ranking score is assigned based on SVM decision function. Patwari et al. (2017) present Tathya, a tool focusing only on check-worthiness detection, which compared to ClaimBuster can yield a significant performance improvement, particularly on recall. It is based on a multi-classifier system using features such as topics, entity history and PoS tuples. ClaimRank (Jaradat et al., 2018) performs check-worthy claim detection and supports English and Arabic texts. Its strength is that it was trained on actual annotations from nine reputable fact-checking organizations, therefore mimicking their real strategy for claim selection. The ranking is based on a number of lexical, structural and semantic features, used to train a neural network with two hidden layers as proposed by Gencheva et al. (2017). Another system, focusing specifically on tweets in Arabic, is Tahaqqaq (Sheikh Ali et al., 2023), which includes the possibility to identify check-worthy claims, estimate the user credibility in terms of spreading fake news, and find authoritative accounts. dEFEND (Cui et al., 2019) is another end-to-end system that, given a link to a post or a news, detects check-worthy sentences by assigning them a score, with the goal to distinguish between check-worthy factual claims from subjective ones. The system displays also the propagation network of the text as well as an analysis of the news comments and the textual evidence supporting the classifier decision. Another similar end-to-end platform, providing evidence snippets to credibility classification and check-worthiness decisions, is BRENDA (Botnevik et al., 2020), which provides also the possibility to collect users’ feedback about wrong predictions. The tool is released as Google Chrome extension. Among the few systems dealing with languages other than English, FactRank (Berendt et al., 2021) was the first system able to process check-worthy texts in Dutch. The classification algorithm was developed iteratively, combining expert fact-checker input, a codebook to support reliable human labelling, and active-learning. Check-worthiness classification performance is comparable to results obtained on English with ClaimBuster. Concerning retrieval of previously fact-checked claims, it is supported by some of the end-to-end verification systems mentioned above, namely ClaimBuster (Hassan et al., 2017), Tahaqqaq (Sheikh Ali et al., 2023) or BRENDA (Botnevik et al., 2020). Besides these, Google Fact-Check Explorer https://toolbox.google.com/factcheck/explorer is often used to perform the task since it indexes a large corpus of fact-checks. Other specialized tools include Fact-Check Finder https://fact-check-finder.kinit.sk built on models developed in (Pikuliak et al., 2023). 7.4. Discussion Multilinguality. We observe in both check-worthiness detection as well as in retrieval of previously fact-checked claims stronger shift towards multilinguality. This is, on one hand, reflected in newer datasets containing also languages other than English (either multilingual ones or focused on a specific language), on the other by the more prevalent use of multilingual models. Since the amount of data in other languages is often limited, approaches for transfer learning (Kazemi et al., 2021; Lucas et al., 2022), low-resource fine-tuning (Cekinel et al., 2024) or adapter fusion (Schlicht et al., 2023) are explored. However, using translation to English in combination with an English language model can still sometimes outperform a multilingual approach, as was observed, e.g., in the previously fact-checked claim retrieval task (Pikuliak et al., 2023). This is likely to change in the future with the employment and/or development of larger and better balanced multilingual models. Multimodality. Although most available datasets are mostly textual, if they contain links to the original content where the claim was made, it is sometime possible to get to other modalities, such as images, videos or audio, which can be contained in a piece of content (e.g., a social media post). These can be important, because in many cases it is there where the actual claim is being made or the claim requires multiple modalities to be properly interpreted. Multimodal content can also be perceived as more credible by users, has a higher engagement and is increasingly easier to produce (Akhtar et al., 2023). Thus, multimodal approaches continue to grow in both prevalence and importance. At the moment, most approaches transcribe the modality to text by either using OCR or image description approaches (Mansour et al., 2022; Pikuliak et al., 2023), however, there are already some approaches that process the other modalities directly, be it images (Vo and Lee, 2020) or speech (Ivanov et al., 2024). The future advances will likely lie in advancements of the latter category of approaches. Availability of datasets. Although there are available resources for both check-worthiness detection and retrieval of previously fact-checked claims, both tasks have their own (sometimes overlapping) sets of challenges. In case of check-worthiness detection, it is relatively easy to collect check-worthy claims – these are all claims verified by fact-checkers. However, collecting non-check-worthy ones is much more challenging. In case of retrieval of previously fact-checked claims, the challenge lies in collecting input claims (e.g., in the form of social media posts) and in identification of pairs between the input and the fact-checked claims. As discussed above, existing methods either lead to too strict matching with many unidentified (false negative) pairs or to too much noise in the data. Another issue for both tasks is that many datasets were previously built using Twitter. If only the IDs of the tweets have been published, it is now very expensive for researchers to use them due to the X’s current API limitations and pricing, thus making their use impractical or completely unfeasible. Combination of check-worthy claim detection and retrieval of previously fact-checked claims. As can be seen from the surveyed works, most of them approach the tasks in separation. This is reasonable from the scientific perspective, but more end-to-end (combined) approaches capable of first detecting a check-worthy claim and then retrieving previously fact-checked claims are needed for practitioners to use. Adoption of LLMs. Although we observe some approaches using LLMs in check-worthiness detection (Barrón-Cedeño et al., 2024), their potential for the retrieval of fact-checked claims begins only now to be more systematically explored. Most recently, LLMs have been used and evaluated as text embedding models as well as rerankers of retrieved fact-checked claims in multilingual and crosslingual settings in (Ramponi et al., 2025), outperforming the fine-tuned smaller models. They have been also employed in zero- and few-shot settings as binary classifiers filtering out irrelevant retrieved results using a range of prompting strategies (Vykopal et al., 2025b; Pisarevskaya and Zubiaga, 2025); these recent works showed that while useful also in this setup, no single prompting strategy proved as the best overall and the performance of the current LLMs is lower for low-resource languages compared to high-resource ones. The use of LLMs (as text embedders, filters or rerankers) have been further explored in system papers submitted to the recent SemEval-2025 Task 7 as summarised in (Peng et al., 2025). Finally, the benefit of LLMs for both check-worthiness detection or retrieval of previously fact-checked claims lies in input claim normalization (Sundriyal et al., 2023), using retrieval augmented generation (Huang et al., 2022), or in providing summaries of the check-worthy content or of the retrieved fact-checks (Vykopal et al., 2025a). 8. Additional credibility signals 8.1. Text quality Text quality is a broad category of credibility signals measuring text’s linguistic accuracy, such as readability, grammatical correctness, or spelling mistakes. It is strongly related to perceived credibility, since high-quality, more professional, content is often seen as more trustworthy. Research on statin-related websites found that more readable and accurate information significantly improves users’ perceptions of credibility (Ling et al., 2024). A similar study by Kiili et al. (2007) highlighted that professionalism in text, such as proper grammar and clear structure, plays a crucial role in how credibility is judged. The similar relation to credibility can be observed for a low-quality content. Harris (1997) showed that poor grammar or frequent spelling errors can be a signal of lack of credibility, prompting readers to question the reliability of the information presented. Greškovičová et al. (2022) investigated how various editorial elements such as superlatives, clickbaits, boldface and poor grammar affect the quality and credibility of online messages. Thus, the quality of the text directly enhances trust in it. Various NLP techniques have been developed to rate text quality and, by extension, its credibility. Mosquera and Moreda (2021) evaluated text by extracting features like contractions, slang, misspellings, emoticons and readability (using the Readability Index). They also measured information content through entropy and emotional tone using emotional distance and, finally, applied the Expectation-Maximization (EM) algorithm to cluster texts by informality levels. The tool obtained a F1 score of 60.6%. In Pitler and Nenkova (2008), the proposed model combines lexical, syntactic, and discourse features to predict human judgments of text readability. It evaluates vocabulary difficulty using unigram language models, syntactic complexity by measuring parse tree height, and discourse relations through annotated markers. Additionally, entity coherence is examined by analysing semantic continuity between sentences. When all features are used to feed a linear regression, the accuracy on readability results in 88.88%. The model proposed by Mesgar and Strube (2018) uses neural networks to evaluate the quality of the text. It captures semantic connections between adjacent sentences by representing sentences as vectors and identifying the most similar states between them. The model uses Recurrent Neural Networks to account for word context within sentences and a Convolutional Neural Network to identify patterns of coherence across the text. This allows the analysis of sentence-to-sentence transitions to predict the readability with a 97.77% accuracy. For the purpose of text quality classification, datasets of various content types originating in a different sources have been used so far. To name a few, Mosquera and Moreda (2021) used a dataset of 50 Yahoo! Answers posts, rated by 6 people in 4 informality level. Pitler and Nenkova (2008) utilized 30 Wall Street Journal articles from the Penn Treebank, rated by college students for readability; and Mesgar and Strube (2018) used a dataset of 105 texts human labeled from the British National Corpus and Wikipedia. 8.2. References and citations When speaking of credibility, elements such as references, citations and partially also quotes can influence the confidence of the user in the content and naturally represent important credibility signals (Aires et al., 2020a; Patricia Aires et al., 2019; Zhang et al., 2018; Baier et al., 2022). It should be noted that the role and influence of references or citations is different to the influence of the quotes although they can occasionally have a similar role (Zhang et al., 2018; Baier et al., 2022). References and citations are traditionally analysed in the context of scientific articles, where they give support by acknowledging prior work and providing a knowledge basis, and thus enhancing their reliability and supporting the validity of their findings (Zhang et al., 2018; Baier et al., 2022). Commonly, explicit references make the sources more transparent, incrementing credibility perception. Quotes from outside experts further enhance the credibility of scientific articles by offering validation and expert perspectives (Aires et al., 2020a; Zhang et al., 2018). Analogously, the role of references and citations can be extrapolated to other domains and online content, such as journalism and newspapers, social media content and marketing (Zhang et al., 2018; Baier et al., 2022). In Baier et al. (2022), it is discussed how the explicitness of references and the depth of assurance provided by citations influence the perceived credibility of the information with experimental results showing that news articles with explicit citations are perceived as more credible by readers. Similarly, Hamborg et al. (2017) highlights the importance of citations in presenting diverse perspectives. Besides overall credibility, references, citations and quotes also can be used in network analysis to detect biases (Aires et al., 2020a). Patricia Aires et al. (2019) used references and citations in link analysis and bias detection, while quotes are discussed in regards to the effectiveness for bias detection. The datasets used in these studies are obtained from multiple domains, including journalism, social media, and scientific content. In Baier et al. (2022), the dataset comprises articles with varying levels of citation explicitness to experimentally assess reader perceptions of credibility. Hamborg et al. (2017) aggregated news articles from diverse sources and examined their references and citation patterns. Similarly, works (Aires et al., 2020a; Patricia Aires et al., 2019) derived the datasets from media outlets, focusing on link analysis and citation patterns to detect bias and assess credibility. These datasets reflect a broad range of content types, enhancing the generalizability of the findings across different informational contexts. 8.3. Clickbaits and title representativeness The term clickbait refers to content designed to raise curiosity and attract users to click on links, often by using sensationalized or misleading headlines. The goal is typically to increase web traffic and advertising revenue (Zhang and Clough, 2020). Clickbait headlines often blur the lines between fact and fiction, contributing to the spread of fake news online (Spezzano et al., 2021). To detect clickbait, various NLP methods can be employed, focusing on analysing lexical and semantic features of headlines. For instance, frequent use of sensationalist language, unresolved pronouns, and forward-referencing structures can indicate clickbait (Zhang and Clough, 2020). Traditional machine learning models, such as Support Vector Machines (SVM) and Naïve Bayes classifiers, are often used to identify these features by assigning probabilities to words and phrases, which are then used to classify headlines as clickbait or non-clickbait (Chen et al., 2015). In this direction, Zhang and Clough (2020) focused on Chinese social media and found significant regional variations in clickbait prevalence. Their studies highlight the effectiveness of SVM and Naïve Bayes classifiers, achieving an F1-measure of 0.834. Combining these approaches with other models like Long Short-Term Memory (LSTM) networks can further enhance classification accuracy, with the SVM model achieving 98.53% accuracy (Winarto et al., 2023). Title representativeness, which measures how accurately a title reflects the content of the article, is another crucial aspect. Spezzano et al. (2021) examined the credibility of news when provided with different combinations of title, image, and source bias. Their study found that combining these elements yielded the best accuracy for automated detectors 0.83, underscoring the importance of integrating multiple meta-data elements for improved accuracy in detecting misleading content. The datasets used in these studies vary in their scope and origin. For example, one of them focused on Chinese social media platforms like WeChat, where they analysed regional clickbait patterns using a dataset of social media posts (Zhang and Clough, 2020); whereas the study by Winarto et al. (2023) leveraged YouTube titles in their study, creating a large, labeled dataset of video headlines to train their machine learning models. Additionally, Spezzano et al. (2021) collected news articles with varying degrees of title representativeness, image content, and source bias, analysing how these elements impacted credibility perceptions. Together, these datasets provide a diverse foundation for training and evaluating models aimed at detecting clickbait across different platforms and content types. 8.4. Originality and content reuse Originality and content reuse have a direct impact on the credibility of information. When the content is original, it reflects the author’s unique insights, analysis, or research, which strengthens its authenticity and trustworthiness. However, when the content is reused, whether through plagiarism, replication without attribution, or even subtle forms of paraphrasing, it can undermine the perceived credibility of the information. Plagiarism is defined as taking intellectual property from another and passing it off as one’s own without citation (Bin-Habtoor and Zaher, 2012). In the context of NLP, plagiarism detection has become more difficult due to the amount of texts available both in traditional print publications and now online. The challenge lies in detecting not only direct copying but also other forms of plagiarism, such as paraphrasing, translations or using hired writers to produce content. The exploration of originality and credibility signals in text reuse involves a variety of approaches combining NLP techniques with advanced machine learning methods. One prominent strategy is leveraging sentence segmentation, tokenization, and syntactic parsing to dissect the structure of text (Chong et al., 2010). The proposed framework integrates these NLP techniques with trigram similarity measures and dependency relations matching, showing a detection accuracy of 70.53% for plagiarized short excerpts of text. This proves how traditional linguistic tools can be enhanced by statistical models to detect subtle patterns in content reuse. Similarly, trigram similarity alongside language model probability and longest common subsequence methods to deal with plagiarism detection can be adopted (Pal et al., 2023). Use of a Naïve Bayes classifier resulted in the same accuracy rate of 70.53%, underscoring the importance of pairing conventional text analysis techniques with machine learning algorithms to capture nuanced instances of content duplication. Other works advanced further by incorporating deeper layers of analysis through methods like Latent Semantic Analysis (LSA) and Latent Dirichlet Allocation (LDA) (Elngar et al., 2021). The proposed solution employs lexical, semantic, and syntactic analysis to detect similarities, achieving an accuracy of 89%. By focusing on semantic relationships between words and phrases, it makes it particularly effective at discovering more sophisticated forms of content reuse. Beyond general plagiarism detection, more specialized approaches concentrate themselves on verifying the authenticity of authorship. Enriquez et al. (2023) developed an Authorship Verification model aimed at identifying hired plagiarism, where someone other than the credited author produces the work. By analysing stylistic features, such as writing patterns and comparing them with the claimed author’s profile, their approach achieved an accuracy of 85%. This sheds light on the possibilities of stylistic analysis as a signal of originality, distinguishing between genuine and outsourced authorship. The datasets used in these studies on plagiarism detection can be found from a variety of sources and text types, reflecting the different challenges posed by plagiarism. For example, Bin-Habtoor and Zaher (2012) focus on general plagiarism detection systems, analysing publicly available datasets from academic publications. Chong et al. (2010) utilize a dataset of academic articles to explore plagiarism through techniques like trigram similarity and dependency relations, applying them to short passages to measure detection accuracy. Enriquez et al. (2023) use a dataset of student writing to train their Authorship Verification model, particularly aimed at detecting plagiarism in commissioned work. Elngar et al. (2021) apply their algorithm to a dataset comprising academic and non-academic texts, with an emphasis on lexical, semantic, and syntactic analysis. Lastly, Pal et al. (2023) use a dataset of online content, including articles and essays, to experiment with automatic plagiarism detection using advanced NLP techniques, combining feature extraction with machine learning models. These datasets represent a range of content, allowing for the robust evaluation of plagiarism detection methods across different textual formats. 8.5. Offensive language The use of offensive language (and its related phenomena, such as hate speech, abusive or toxic language; see, e.g., (Papcunová et al., 2023) for definitions), particularly in online discourse, often correlates with the low credibility of the content, containing false information, prejudices, biases, and what in general is considered as toxic language. For example, Botella-Gil et al. (2024) show that there is a direct association between the credibility of media sources and the presence of hate in online comments, while Bourgeade et al. (2023) show how fake stories are used to spread hate against immigrants by analysing racial hoaxes on Twitter. Also some of the most widely used datasets for check-worthy claim detection already include information about the presence of hateful (Hadj Ameur and Aliane, 2021) or harmful (Nakov et al., 2022a) content, showing the importance of adopting a multi-faceted view on the problem of misleading information. Indeed, the use of abusive or toxic language, has also been included in the W3C list of credibility signals (Multiple authors and Hawke, 2019). Offensive language detection has been extensively investigated within the NLP community for at least a decade, starting from English (Waseem and Hovy, 2016) and then comprising more and more languages (for example the Hate Speech Datasets repository https://hatespeechdata.com/ that lists more than one hundred datasets in 25 languages). The creation of resources, which led in turn to the development of several approaches to offensive language detection, has been fostered by the shared tasks on hate or offensive language detection organised throughout the years at SemEval and other data challenges (Fersini et al., 2018; Bosco et al., 2018; Wiegand et al., 2018; Zampieri et al., 2019, 2020; Pavlopoulos et al., 2021). Among the different approaches implemented for offensive language detection, transformer-based architectures based on RoBERTa (Liu et al., 2019b) and its multilingual variants have proven to be very effective for the task when fine-tuned on offensive data, with F1 scores $>$ 0.90 (Zampieri et al., 2020). Nevertheless, the problem of online content moderation is far from being solved given the amount of toxic content still circulating on social media, which suggests that research on the topic still needs to address several understudied aspects of the phenomenon. Current works have identified some research directions worth studying, such as the problem of human label variation when creating datasets for offensive language detection (Leonardelli et al., 2021), the role of annotators’ biases (Sap et al., 2022), the presence of spurious correlations in existing datasets (Wiegand et al., 2019; Ramponi and Tonelli, 2022) and the lack of robustness when classifying data from different online platforms and domains (Salminen et al., 2020). Concerning generative LLMs, recent experiments showed that classifying offensive language using zero- and few-shot learning LLMs yields considerably lower results in comparison to smaller models fine-tuned on the entire training set (Edwards and Camacho-Collados, 2024). Generative LLMs have been alternatively used to create synthetic datasets for hate speech detection, with the goal to address problems such as data decay and privacy concerns related to social media (Casula et al., 2024). Similarly, LLMs have also been employed to generate functional tests for a fine-grained evaluation of hate speech detection (Jin et al., 2024). However, the fact that major generative LLMs have been adjusted through so-called alignment to avoid generating hateful content may represent an obstacle for future research in this direction. 8.6. Machine-generated text The task of machine-generated text detection (also called synthetic or neural text detection or authorship attribution (Uchendu et al., 2023)) has already been researched since about 2018 when the arrival of GPT-1 and transformer architecture have enabled generation of reasonably coherent texts (Uchendu et al., 2023). However, it has been only with the arrival of generative LLMs that the task gained upon practical importance due to increased fidelity and quality of the machine-generated texts in English (Uchendu et al., 2021), but also in many other languages (Macko et al., 2023; Wang et al., 2024b; Macko et al., 2025) and due to low costs of such generation. Since then, there has been recognised potential of the machine-generated texts to be misused for influence operations (Goldstein et al., 2023), disinformation (Buchanan et al., 2021; Vykopal et al., 2024; Lucas et al., 2023; Zugecova et al., 2025), spam or unethical authorship (Crothers et al., 2023). Although there have been voices tempering down fears of massive misuse of LLMs for disinformation generation (Simon et al., 2023), actual field studies are missing and the rapid advancement of the new models makes the misuse of LLMs to generate and/or amplify disinformation increasingly easier. Nevertheless, when considering using machine-generated text detection as a credibility signal, it has to be noted that there are also many benign and legitimate uses (e.g., machine translation, use of LLMs to improve stylistics or grammar, etc.), so the context of such a use needs to be taken into account as well. Due to increased interest in the topic, there have been lately several new datasets released extending the task to a range of languages and domains (Macko et al., 2023; Wang et al., 2024b; Macko et al., 2025), often supported by specialized data challenges and tasks, such as SemEval-2024 Task 8 (Wang et al., 2024a). As to the detection methods and models, these range from statistical, e.g., Binoculars (Hans et al., 2024), Fast-DetectGPT (Bao et al., 2024), etc. to fine-tuned language models, such as Longformer (Beltagy et al., 2020; Li et al., 2024a), RoBERTa (Solaiman et al., 2019) or MDeBERTa (Macko et al., 2023), but also increasingly including fine-tuned LLMs or their combinations (Spiegel and Macko, 2024). Despite the undeniable progress in the task, there are several remaining challenges, such as robustness of detectors to the out of distribution data and adversarial attacks (such as authorship obfuscation (Macko et al., 2024)), their interpretability and explainability or detection of not purely generated texts (Jawahar et al., 2020; Uchendu et al., 2023; Wu et al., 2024). 8.7. Non-NLP research on credibility signals While the goal of this survey is to give an overview of the credibility assessment and credibility signals from the perspective of Natural Language Processing (NLP), additional credibility signals that can be extracted from ancillary elements are not of lesser relevance. These non-NLP credibility signals can range from the multimedia attached or linked to the textual information (Zade et al., 2023; Hansen et al., 2020; Hamborg et al., 2017), to other contextual cues, such as the impact of an author (Briscoe et al., 2013; Jahanbakhsh et al., 2022; Zhao et al., 2016), or even the acceptance and feedback provided by the engaged public (İŞ and TUNCER, 2018; Kondo and Abe, 2023; Lin et al., 2016). For example, it can be expected that a fake video would unlikely be a part of credible content; likewise, a biased author with an untrustworthy historic record could lead to the same outcome. Several taxonomies can be used to categorize non-NLP credibility signals, for example, a distinction can be made on the signals coming from different content modalities, such as complementary videos or images, or contextual entities, like author or source information (Ye and Skiena, 2019; Patricia Aires et al., 2019; Aires et al., 2020a). A general classification of non-NLP credibility signals is summarised in Table 10 with corresponding signals’ examples and references. It is noticeable that the credibility signals are from a large variety of sources and formats, and some of them have been presenting good research results recently (Ciobanu et al., 2014; Ye and Skiena, 2019; Riegelsberger et al., 2005). Table 10. Overview of complementary non-NLP signals | Image-based signals | Image quality (resolution), Relevance of images to the rest of the content, Source attribution, Image authenticity (fake or generated), Presence of alternative text metadata | (Zhang et al., 2018; O’Grady, 2006; Spezzano et al., 2021) | | --- | --- | --- | | Video-based signals | Video quality (high definition, video production), Relevance of video to the rest of the content, Source attribution, Video authenticity, Presence of transcriptions and subtitles | (Zhang et al., 2018; O’Grady, 2006) | | Audio-based signals | Audio quality, Relevance of audio to the rest of the content, Source attribution, Audio authenticity, Presence of transcriptions | (Ciobanu et al., 2014) | | Interactive and embedded content signals | Quality and usability of interactive elements (interactive frameworks, maps, utility elements or forms), Trustworthiness of embedded elements (e.g. YouTube, newspapers, software providers) | (Chua et al., 2023) | | Source signals | Domain authority (reputation of the website, domain authority scores), Publisher reputation (historical credibility of publisher/organization), Website technology and design (quality, usability, loading speed, SEO implementation, consistency in design, branding/marketing), Traffic statistics (website analytics) | (Spezzano et al., 2021; Ye and Skiena, 2019) | | Ownership and transparency | Author information (availability, detail depth, verifiability of author), Editorial policies (regulations and moderation/revision processes of medium), Funding and sponsorship disclosure | (O’Grady, 2006) | | External references | External references or reviews from renowned sites, External fact-checking from trustworthy sites | (Kondo and Abe, 2023; Lu et al., 2012; Jahanbakhsh et al., 2022; Ye and Skiena, 2019) | | User interaction | Comments and discussion or readers’ profiles, User reviews and engagement, Social media activity (e.g. virality, influencers) | (Kondo and Abe, 2023; Lu et al., 2012; Solanki et al., 2021; Zade et al., 2023; Lin et al., 2016; Ji et al., 2023b; Zhao et al., 2016) | | Security of website | HTTPS protocol, Legal policies and service terms, Compliance with standards (GDPR or local regulations), Cybersecurity practices | (O’Grady, 2006) | | Content presentation | Consistency in design and structure, Visual and audio aesthetics | (Spezzano et al., 2021) | Studies have shown that combining textual analysis with multimedia analysis leads to significant improvements in credibility assessment as well as misinformation detection (Choudhary et al., 2024; Alam et al., 2022a; Hangloo and Arora, 2022; Briscoe et al., 2013; Jahanbakhsh et al., 2022). This combination is typically achieved through ensemble models or architectures that unify the outputs of various sub-models focused on different media types. A common method involves using transformer-based models such as BERT or GPT for text analysis alongside convolutional neural networks (CNNs) for image or video analysis. These models work in tandem, the text models identifying linguistic signals of credibility while the CNNs assess visual features like image authenticity, quality, and relevance. Additional layers of analysis might focus on audio quality, transcription accuracy, or even the relationship between the text and accompanying media. For example, a multimodal model might analyse a tweet’s text for sentiment and coherence, while also assessing the attached image’s metadata and authenticity. If the image shows signs of manipulation or does not align with the textual claim, the credibility for that piece of content would be lowered. Another approach is the usage of propagation graphs (Solanki et al., 2021; Chua et al., 2023; Zade et al., 2023), where the social activity of the sharing and propagation of a piece of news in a social network is the subject of study. Finally, some works combine both the user and community engagement together with the actual content (including using NLP techniques to process the text) to detect the trustworthiness of the piece of news in particular (İŞ and TUNCER, 2018; Lin et al., 2016; Zhao et al., 2016). At the same time, studies in multimodal credibility assessment consistently show that while non-textual cues (e.g., image authenticity, video manipulation detection) significantly enhance the detection process, they are most effective when paired with deep text analysis. For instance, a recent work combines user engagement patterns with textual signals to identify low credibility content (Qureshi and Malick, 2023). Another promising approach is the integration of social media metadata (such as virality metrics and user behaviour) alongside textual content to detect the spread of such low-credible content (Zade et al., 2023; Spezzano et al., 2021). 9. Challenges and open problems Following the analysed research works as well as our own experience in the area of credibility assessment and credibility signal detection, we follow up with an overarching and orthogonal (to credibility assessment and categories of credibility signals described so far) discussion of challenges and open problems. 9.1. Research fragmentation and need for integration The overarching conclusion of this literature survey is the high degree of fragmentation across the field of credibility assessment research. This issue became apparent throughout the entire survey process – beginning with the analysis of existing taxonomies/lists of credibility signals and our unified categorization (Section 2), continuing with the review of research on automatic credibility assessment (Section 4), and extending to approaches for detecting individual credibility signals (Sections 5 – 8). This fragmentation has several important consequences. First, there is a lack of standardized taxonomies for credibility signals. The most prominent attempt so far – an extensive list of approximately 200 crowdsourced credibility signals proposed by the W3C Credible Web Community Group (Multiple authors and Hawke, 2019) – remains incomplete and not fully categorized, limiting its practical utility for guiding research and tool development. Second, there is a significant disconnection within the NLP research community itself. Research on automatic credibility assessment is often conducted in isolation from work on the detection of individual credibility signals. As a result, even highly advanced methods for detecting certain signals (e.g., persuasion techniques) are rarely integrated into broader credibility assessment solutions. Consequently, little is known about how such signals may contribute to predicting overall content credibility in practice. Third, researchers working on individual credibility signals frequently appear unaware of related efforts in neighbouring categories, despite these signals often being identified under the shared conceptual umbrella of credibility. Stronger interconnection between these lines of research could unlock mutual benefits – for instance, the creation of shared, curated datasets annotated for both individual signals and overall credibility (also building on the first efforts in this direction by (Zhang et al., 2018; Bhuiyan et al., 2020), as discussed in Section 4.4). This would not only increase annotation quality and sample size but also enable the study of correlations between different signals and support the development of multitask learning models – techniques currently underutilized due to a lack of such integrated resources. Interestingly, although research efforts remain isolated in their own silos built around the single NLP task, some end-user-oriented tools already incorporate multiple credibility signals in practice. Notable examples include the Tanbih system https://tanbih.org/ (Zhang et al., 2019) and the Verification Plugin https://www.veraai.eu/category/verification-plugin. To illustrate the potential benefits of detecting and aggregating multiple credibility signals (representing individual categories systematically covered by this survey) into a unified credibility label/score, we provide two illustrative examples: one depicting a low-credibility social media post (Figure 7) and another representing a high-credibility example (Figure 8). These examples clearly showcase an untapped potential of synergy between detection of diverse advanced credibility signals, their explanations, as well as aggregation. To achieve such desired state, we advocate for significantly stronger integration of current research efforts. <details> <summary>x2.png Details</summary>  ### Visual Description ## Credibility Analysis: Flu Shot Claim ### Overview The image presents a credibility analysis of a statement against flu shots. The analysis breaks down the statement across several categories, assessing its factuality, subjectivity, bias, use of persuasion techniques, logical fallacies, fact-checked status, text quality, offensiveness, likelihood of being machine-generated, and use of clickbait. The overall credibility is rated as low (23%). ### Components/Axes The analysis is structured as a table with two columns: * **Left Column:** Categories of analysis (Event Factuality, Subjectivity, Bias, Persuasion Techniques, Logical Fallacies, Fact-checked status, Text Quality, Offensive Language, Machine-generated Text, Clickbait Title, Overall Credibility). * **Right Column:** Detailed assessment for each category. ### Detailed Analysis Here's a breakdown of each category and its assessment: * **Event Factuality:** High event factuality: Author is very sure that the events happened (three people got flu after being vaccinated, the shots are spreading the virus). * **Subjectivity:** Highly subjective: Relies on personal anecdotes ("I personally know...") and emotionally loaded language ("Big Pharma", "Don't be a sheep"). * **Bias:** Strong anti-vaccine bias: Positions vaccination as harmful and implies malicious intent by pharmaceutical companies. * **Persuasion Techniques:** Highly present: Text is using persuasion techniques like Fear appeal, Bandwagoning appeal or Appeal to nature. * **Logical Fallacies:** Medium present: Text is using logical fallacies like Ad hominem or Straw man. * **Fact-checked status:** Previously fact-checked: Contains claim that has been previously fact-checked as false (https://factcheck.afp.com/doc.afp.com.36JN78V). * **Text Quality:** Medium-low quality: Text is using informal style, all-caps emphasis, emojis, hashtags, and exaggeration. * **Offensive Language:** Mildly offensive: "Don't be a sheep" is a derogatory phrase implying stupidity or blind obedience. * **Machine-generated Text:** Likely machine generated: Text appears to use the selection of words corresponding to OpenAI models. * **Clickbait Title:** Uses clickbait elements: Urgent command ("Think Twice") and controversial implication (flu shots are harmful). * **Overall Credibility:** Low (23%) The original statement being analyzed is: > Think Twice Before Getting That FLU SHOT! > > I personally know three people who got the flu right after being vaccinated—coincidence? I don't think so. Big Pharma just wants your money and doesn't care if you get sick. > > They keep pushing these so-called "safe" vaccines every year, but how come flu cases *always* go up right after the campaigns? I am convinced that the shots are spreading the virus. > > Don't be a sheep. Protect yourself naturally—boost your immune system with vitamins, not chemicals! > > \#WakeUp \#FluShotScam \#NaturalImmunity ### Key Observations * The analysis consistently points to flaws in the statement's reasoning, evidence, and presentation. * The statement relies heavily on personal anecdotes and emotionally charged language. * The claim has been previously fact-checked and found to be false. * The overall credibility score is very low, indicating a high degree of unreliability. ### Interpretation The analysis suggests that the statement against flu shots is highly unreliable and should be treated with extreme skepticism. The low credibility score is justified by the presence of strong bias, logical fallacies, poor text quality, and the fact that the claim has been previously debunked. The use of clickbait tactics further undermines the statement's credibility. The analysis effectively demonstrates how to critically evaluate claims and identify potential misinformation. </details> Figure 7. An illustrative example of diverse credibility signals determining the social media post as being of a low credibility. Each signal is associated with predicted values and a short explanation. The overall credibility label and score reflects the credibility predicted by individual signals. <details> <summary>x3.png Details</summary>  ### Visual Description ## Credibility Assessment: Flu Shot Information ### Overview The image presents a credibility assessment of a statement regarding the benefits of flu shots. It includes the statement itself, along with evaluations of its event factuality, subjectivity, bias, persuasion techniques, logical fallacies, fact-checked status, text quality, offensive language, machine-generated text likelihood, clickbait title, and overall credibility. ### Components/Axes The image is divided into two main sections: 1. **Left Section:** Contains the statement being assessed, titled "Why Getting a Flu Shot Matters," along with supporting information and hashtags. 2. **Right Section:** Presents the credibility assessment criteria and their corresponding evaluations. The credibility assessment criteria are: * Event Factuality * Subjectivity * Bias * Persuasion Techniques * Logical Fallacies * Fact-checked status * Text Quality * Offensive Language * Machine-generated Text * Clickbait Title * Overall Credibility ### Detailed Analysis or Content Details **Left Section:** * **Title:** "Why Getting a Flu Shot Matters" with a green checkmark icon. * **Statement:** "Flu vaccines do not cause the flu. In fact, they are designed to help your immune system recognize and fight the virus before you get exposed." * **Supporting Information:** "While it's possible to catch the flu after vaccination (especially if you were already exposed), the shot significantly reduces your chances of severe illness and hospitalization. According to the CDC, flu vaccination prevented an estimated 7.5 million illnesses and 6,300 deaths in the U.S. during the 2019-2020 flu season. [https://www.cdc.gov/flu/vaccines-work](https://www.cdc.gov/flu/vaccines-work) If you have concerns, talk to your doctor—not the internet." * **Hashtags:** "#PublicHealth #VaccinesWork #FluPrevention" **Right Section:** * **Event Factuality:** "Medium-high event factuality: Author is quite sure that the event happened (flu vaccination prevented illnesses and deaths)." * **Subjectivity:** "Objective tone: Neutral and informative, avoids emotional or personal wording." * **Bias:** "Slight institutional bias: Text contains bias toward health promotion, but aligned with scientific consensus and transparently sourced." * **Persuasion Techniques:** "Medium present: Text contains Evidence-based persuasion, Call to action, but uses facts and statistics, not fear or emotion." * **Logical Fallacies:** "None present: Reasoning is logically sound; does not overclaim or generalize." * **Fact-checked status:** "No existing fact-checks: No existing fact-checked claim was detected." * **Text Quality:** "High quality: Clear grammar, complete sentences, link to source, and no excessive punctuation or slang." * **Offensive Language:** "None: Language is inclusive and respectful." * **Machine-generated Text:** "Likely human written: Text appears to use selection of words that correspond to human-written texts." * **Clickbait Title:** "No clickbait: Straightforward and non-sensationalized title." * **Overall Credibility:** "High (95%)" - Indicated by a green bar. ### Key Observations * The statement is assessed as having medium-high event factuality and is written in an objective tone. * There is a slight institutional bias towards health promotion, but it is aligned with scientific consensus. * The text uses evidence-based persuasion techniques. * No logical fallacies are present. * No existing fact-checks were detected. * The text quality is high, with clear grammar and complete sentences. * The language is inclusive and respectful. * The text is likely human-written. * The title is not clickbait. * The overall credibility is rated as high (95%). ### Interpretation The credibility assessment suggests that the statement regarding the benefits of flu shots is generally reliable and trustworthy. While there is a slight bias towards health promotion, it is supported by scientific consensus. The high overall credibility score indicates that the information presented is likely accurate and can be considered a credible source. The assessment considers multiple factors, providing a comprehensive evaluation of the statement's reliability. </details> Figure 8. An illustrative example of diverse credibility signals determining the social media post as highly credible. Each signal is associated with predicted values and a short explanation. The overall credibility label and score reflects the credibility predicted by individual signals. 9.2. Adoption and potential of generative LLMs Generative Large Language Models (LLMs) have demonstrated substantial improvements in complex tasks that require reasoning abilities (Qiao et al., 2023). Brown et al. (2020) showed that pretrained LLMs are capable of few-shot learning, meaning they can learn to perform new tasks with only a few training examples. Similarly, Petroni et al. (2019) highlighted the strong ability of LLMs to recall relational knowledge acquired during pretraining to perform various tasks without further annotated labels or human supervision (i.e., zero-shot learning). Additionally, several key advancements, such as Retrieval-Augmented Generation (RAG) (Lewis et al., 2020), Reinforcement Learning with Human Feedback (RLHF) (Ouyang et al., 2022), and robust prompting techniques (Wei et al., 2022; Qiao et al., 2023), have further enhanced the capabilities of LLMs. In this context, recent generative LLMs operate as dialogue systems, where the model is prompted by the user with instructions to perform specific tasks. As we have discussed in the sections on individual credibility signals (see Sections 5, 6, 7, but also 8), the uptake of LLMs differs across the signals. In some cases, first such uses appeared only in late 2024 and beginning of 2025, such as in the case of previously fact-checked claims (Section 7). Nevertheless, their prevalence gradually increases. However, approaches using LLMs for credibility assessment (see Section 4) are still rare with one such notable exception being the work of Leite et al. (2024). Consequently, there are still several unexplored or underexplored opportunities how such models could address challenges related to the automatic detection of credibility signals. One of the key advantages of using prompting with LLMs is the flexibility in adapting a single foundational model to handle multiple subtasks associated with credibility assessment. With a carefully designed framework, LLMs can be guided to focus on different aspects of content analysis, such as detecting persuasion techniques, evaluating the veracity of claims, identifying potential bias, and recognizing patterns of misinformation. This flexibility reduces the need to develop and fine-tune separate models for each task, allowing practitioners to use the same model across various credibility-related tasks. In fact, a promising research direction is to explore multi-task learning, as in verifying if the capacity of performing certain credibility-assessment tasks can aid in other related ones (e.g., persuasion and bias). Moreover, the capability of learning with zero/few examples is particularly valuable for tasks where domain-specific data is scarce or constantly changing, as is the case with credibility assessment. An enormous amount of human effort is required to curate high-quality annotated datasets for the different subtasks involved in assessing credibility. Specially since labelling most of credibility signals (such as biased content) often demand the expertise of domain specialists such as fact-checkers, journalists and social scientists. Adding to this challenge, credibility indicators can be highly context-dependent, varying across cultural and temporal dimensions. In this context, LLMs offer a more scalable approach by drawing on vast amounts of unsupervised pretraining data, and by adjusting to specific end-tasks through careful prompting strategies, which require far less human effort than manual data labelling. As an example, in Leite et al. (2024), a generative LLM was employed to predict $19$ different credibility signals present in textual content without using any training data (i.e., in a zero-shot setting). Finally, the generative capabilities of large language models can be leveraged to produce more explainable and interpretable Here, the concept of interpretability differs from explainability, which is often used in the field of machine learning to refer to specific methods to analyse how intermediate states of the model lead to certain outcomes (Burkart and Huber, 2021). outputs, which is crucial for subject-matter experts that may leverage the model’s predictions. Instead of providing only binary or scalar outputs (e.g., true/false, misinformation/non-misinformation, credible/non-credible) as in usual classification tasks, generative LLMs can produce detailed explanations or summaries that can highlight the reasoning behind their predictions. This transparency allows human experts to critically assess the model’s outputs, cross-check them with external information, and ensure that (i) incorrect predictions (in this context, known as model hallucinations (Ji et al., 2023a)) are properly mitigated, and (ii) any credibility assessment aligns with the context of the content being investigated. This property of interpretable outputs can significantly increase trust in model-driven decisions and reduce the likelihood of over-reliance on machine predictions, ensuring that humans remain in the loop for final judgments. We acknowledge that LLM adoption (despite providing a lot of potential) is also accompanied with several challenges. Fine-tuning as well as deployment of LLMs require a considerably higher computing power which directly translates to higher costs. Moreover, learning techniques commonly used in limited labelled data scenarios (prompting, in-context learning, fine-tuning, meta-learning, or few-shot learning) are known to be sensitive to various randomness factors (Pecher et al., 2024d), what can result in undesired performance instability. Beside randomness factors, also systematic choices, such as the format of the prompt or how many (in-context) samples are used, have significant effect on the overall performance and the stability of these approaches (Pecher et al., 2024c; Voronov et al., 2024). Nevertheless, ongoing research in the area of LLMs is already providing suitable solutions to such problems, such as Parameter-efficient Fine Tuning (PEFT) techniques (Xu et al., 2023), which have been demonstrated to perform well also on credibility signal detection tasks (Razuvayevskaya et al., 2024); or instability mitigation techniques, such as ensembling, noise regularisation and model interpolation (Pecher et al., 2024a). At the same time, we would like to stress that employing LLMs does not necessarily provide benefit in all cases. When a sufficient amount of labelled samples is available (100-1000 depending on the specific task and model), fine-tuned smaller language models can outperform larger general ones (Pecher et al., 2024b). Finally, as highlighted in Section 5, implicit biases present in various LLMs can make them unreliable in when assessing external biases or other credibility signals (Lunardi et al., 2024; Bang et al., 2024). Efficient debiasing strategies, evaluation metrics for bias detection and alignment strategies must be implemented before such models can be applied to real-world tasks (Lin et al., 2024a; Mohanty, 2025). Factual correctness of LLMs is also vulnerable to slight contextual shifts and hallucinations (Augenstein et al., 2024), and improvements in source attribution, domain-specific adaptation and reasoning capabilities are cornerstone tasks in this direction that need to be addressed. 9.3. Dataset availability and multilinguality Dataset availability heavily differs across various categories of credibility signals. Firstly, we observed a significant lack of large-enough datasets providing overall (expertly-determined) annotation of credibility that can be used for training and evaluating solutions on credibility assessment task (Section 4.1). Furthermore, datasets containing annotations for overall credibility and (at least some) credibility signals at the same time are even more scarce. As a result, many researchers opted to use (more available) fake news datasets as a replacement. At this place, we would like to highlight again that fake news annotations cannot reliably replace credibility annotations (non-credible content is not necessarily only false content and vice versa, since credibility is rather a parallel and complementary dimension to veracity; see Section 2). On the other hand, the situation with dataset availability for individual categories of credibility signals is much better. This is especially thanks to data challenges (particularly SemEval tasks and CLEF CheckThat! Labs as evidenced in Sections 5, 6, or 7) in which either new datasets were introduced or existing datasets were extended (with more data, new languages, or additional types of annotations). Unfortunately, in some cases (e.g., persuasion techniques dataset by Piskorski et al. (2023) introduced in Section 6), the datasets from data challenges are not published completely and a hidden/testing set is not shared with researchers not even when the competition is over (and thus only the training/validation sets remain available for further research). Especially for international news or the investigation of global claims, journalists and fact-checkers need to verify the credibility of information by cross-checking sources in different languages. Furthermore, emerging disinformation in one country could be spread to other countries, especially when its topic is global (e.g., pandemic, wars, international relations). Therefore, it is important to implement credibility analysis tools that support multiple languages. However, due to the scarcity of multilingual datasets, trained models can exhibit biases towards some languages and cultures and hence can underperform on content in low-resourced languages (as is for example the case for fact-checked claims retrieval as evidenced in (Vykopal et al., 2025b); see Section 7). In Table 11, we present a summarised overview of language-specific datasets available for different categories of credibility signals. As in many other areas of NLP, English remains the dominant language, with the majority of available resources focused on English-language content. Nevertheless, notable datasets also exist for other widely spoken languages, particularly Arabic and Spanish. In contrast, there is a clear lack of annotated resources for many low- and mid-resource languages, such as Czech or Tamil. Among the covered categories, fact-checked claims demonstrate the broadest language coverage. This can be attributed to the global presence of fact-checking organizations, which produce multilingual artifacts that serve as the foundation for constructing such diverse and multilingual datasets. On the other hand, categories such as factuality and bias show limited availability of resources beyond English, highlighting a significant gap in the development of multilingual tools for credibility signals detection. Table 11. Summary of language-specific resources per individual categories of credibility signals. In a case of (Pikuliak et al., 2023; Nielsen and McConville, 2022) datasets – which are highly multilingual, comprising 42 and 39 languages respectively – we directly report those languages that have an overlap with other datasets, while the number of remaining languages is provided in the Additional languages row. For a full list, please, refer to the original papers. | Language | Credibility Signals | | | | | | | --- | --- | --- | --- | --- | --- | --- | | Subjectivity | Factuality | Bias | Persuasion techniques | Check- worthiness | Fact-checked claims | | | English | (Piskorski et al., 2023; Spinde et al., 2023; Biyani et al., 2014) (Spinde et al., 2021b; Banea et al., 2010; Pang and Lee, 2004) (Wiebe et al., 2005) | (Qian et al., 2022, 2019; Li et al., 2024c) | (Piskorski et al., 2023; Spinde et al., 2021d, b) (Liu et al., 2019a; Fan et al., 2019; Chen et al., 2020) | (Da San Martino et al., 2019a; Piskorski et al., 2023; Lawson et al., 2020) (Da San Martino et al., 2020a; Dimitrov et al., 2021; Macagno, 2022) | (Gencheva et al., 2017; Cheema et al., 2022; Salek Faramarzi et al., 2023) (Arslan et al., 2020; Atanasova et al., 2018, 2019) (Barrón-Cedeño et al., 2020; Nakov et al., 2021, 2022a) (Alam et al., 2023; Barrón-Cedeño et al., 2024) | (Shaar et al., 2020; Nakov et al., 2022b) (Vo and Lee, 2020; Kazemi et al., 2021) (Hardalov et al., 2022; Singh et al., 2024; Nielsen and McConville, 2022; Pikuliak et al., 2023) | | Chinese | – | (Qian et al., 2022, 2019) | – | – | – | (Nielsen and McConville, 2022; Pikuliak et al., 2023) | | German | (Atalla et al., 2011; Piskorski et al., 2023; Banea et al., 2010) | – | (Piskorski et al., 2023; Aksenov et al., 2021) | (Piskorski et al., 2023) | – | (Nielsen and McConville, 2022; Pikuliak et al., 2023) | | Urdu | (Wiebe and Riloff, 2005) | – | – | – | – | (Nielsen and McConville, 2022; Pikuliak et al., 2023) | | Arabic | (Mourad and Darwish, 2013; Banea et al., 2010) | – | – | (Alam et al., 2022b; Hasanain et al., 2023; Almotairy et al., 2024) | (Hadj Ameur and Aliane, 2021; Atanasova et al., 2018; Barrón-Cedeño et al., 2020) (Nakov et al., 2021, 2022a; Alam et al., 2023) (Barrón-Cedeño et al., 2024) | (Nakov et al., 2022b; Nielsen and McConville, 2022; Pikuliak et al., 2023) | | French | (Piskorski et al., 2023; Banea et al., 2010) | – | (Piskorski et al., 2023) | (Piskorski et al., 2023) | – | (Nielsen and McConville, 2022; Pikuliak et al., 2023) | | Polish | (Piskorski et al., 2023) | – | (Piskorski et al., 2023) | (Piskorski et al., 2023) | – | (Nielsen and McConville, 2022; Pikuliak et al., 2023) | | Italian | (Piskorski et al., 2023) | – | (Piskorski et al., 2023) | (Piskorski et al., 2023; Macagno, 2022) | – | (Nielsen and McConville, 2022; Pikuliak et al., 2023) | | Russian | (Piskorski et al., 2023) | – | (Piskorski et al., 2023) | (Piskorski et al., 2023) | – | (Nielsen and McConville, 2022; Pikuliak et al., 2023) | | Spanish | (Banea et al., 2010) | – | – | – | (Nakov et al., 2021, 2022a; Alam et al., 2023) (Barrón-Cedeño et al., 2024) | (Martín et al., 2022; Singh et al., 2024; Nielsen and McConville, 2022; Pikuliak et al., 2023) | | Romanian | (Banea et al., 2010) | – | – | – | – | (Pikuliak et al., 2023) | | Portuguese | (Jeronimo et al., 2020) | – | – | (Macagno, 2022) | – | (Singh et al., 2024; Nielsen and McConville, 2022; Pikuliak et al., 2023) | | Dutch | (Maks and Vossen, 2012) | – | – | – | (Nakov et al., 2022a; Barrón-Cedeño et al., 2024) | (Nielsen and McConville, 2022; Pikuliak et al., 2023) | | Czech | – | – | – | (Baisa et al., 2019) | – | (Nielsen and McConville, 2022; Pikuliak et al., 2023) | | Turkish | – | – | – | – | (Kartal and Kutlu, 2020; Nakov et al., 2021, 2022a) | (Nielsen and McConville, 2022; Pikuliak et al., 2023) | | Hindi | – | – | – | – | (Dhar and Das, 2021) | (Kazemi et al., 2021; Singh et al., 2024; Nielsen and McConville, 2022; Pikuliak et al., 2023) | | Bengali | – | – | – | – | (Dhar and Das, 2021) | (Kazemi et al., 2021; Nielsen and McConville, 2022; Pikuliak et al., 2023) | | Norwegian | – | – | – | – | (Sheikhi et al., 2023) | (Nielsen and McConville, 2022; Pikuliak et al., 2023) | | Bulgarian | – | – | – | – | (Nakov et al., 2021, 2022a) | (Pikuliak et al., 2023) | | Tamil | – | – | – | – | – | (Kazemi et al., 2021; Nielsen and McConville, 2022; Pikuliak et al., 2023) | | Malayalam | – | – | – | – | – | (Kazemi et al., 2021; Nielsen and McConville, 2022; Pikuliak et al., 2023) | | Additional languages | – | – | – | – | – | + 22 (Nielsen and McConville, 2022), + 18 (Pikuliak et al., 2023) | To overcome the scarcity of multilingual datasets, global collaborations could be initiated for creating multilingual resources. In this direction, we can already observe a positive trend within data challenges. Many of them introduced multilingual datasets, commonly considering some languages as surprise ones (i.e., languages that are present in a test set, but missing in a train/validation set). Such approach motivates participants to develop multilingual solutions that are capable to make a prediction in a zero-shot setting (considering a language a predicted content is written in). Besides datasets availability and multilinguality, a quality of annotations remain another challenge. Annotation of overall credibility as well as individual credibility signals is many times highly subjective (as also shown in (Bhuiyan et al., 2020)), especially in cases such as persuasion techniques where presence of a signal and borders between various signals can be blurred. The situation is getting even more challenging in multilingual settings, where typically native speakers are needed to annotate data. Firstly, acquiring human experts fluent in several languages is challenging itself. Secondly, organizing and consolidating annotation process (including post-annotation verification) is a complex task. Considering also our own experience working with such datasets, the provided labels cannot be easily verified and many times we identified (inevitable) incorrect labels. Last but not least, credibility assessment and automatic detection of credibility signals naturally happen in very dynamically evolving (online) environment. New topics and global events constantly emerge, causing significant data and concept drifts. In some cases, such drifts can cause that the existing datasets may become obsolete and non-representative. Secondly, the list of credibility signals itself evolves. We can take a machine-generated text as an illustrative example. This kind of signal become highly relevant only recently with generative LLMs becoming easily available for a large end user base (and unfortunately also bad actors). Such new/redefined signals thus naturally result into a demand for new datasets. Finally, the dynamics of this area also lies in its adversarial character. Bad actors (e.g., the ones who are spreading propaganda or disinformation) will always try to get their content undetected as low-credible one by employing various obfuscation techniques. This must be taken into consideration when introducing new datasets. By continuing with an illustrative example of machine-generated text, there are already datasets (e.g., (Macko et al., 2024)) providing besides machine-generated texts also their alternative versions after applying several authorship obfuscation techniques, which allows to train and evaluate more robust detection models. 9.4. Ethical and legal issues Credibility assessment of online (primarily social media) content is from its nature an area that must address several ethical and legal issues. First, such ethical and legal issues are especially prominent when the researched outcomes get deployed and used in practice by end users. The challenge for tool makers is to be fully transparent about the limitations of their tools, to provide guidelines to avoid misleading their users (e.g., by false positives and hallucinations) and to support human control (in line with the human-in-the-loop approach), for more details see Section 9.5. Second, similarly to other related research areas (e.g., false information detection), credibility assessment methods may be potentially misused by bad actors in order to create content that appears to be more credible. Also by following open-research spirit, publishing credibility assessment systems can theoretically result into misusing such system in the adversarial manner to tune disinformation/propaganda generation systems and allowing them to stay undetected. This kind of potential threat is, however, an analogical issue to the security domain and the principle of security by obscurity. Nothing prevents bad actors to develop their own credibility assessments systems and apply then in adversarial training scenario. Moreover, positive outcomes of credibility assessment research are more tangible, with many practical tools (as also showed across this survey) already put into the hands of media professionals or general public. Besides potential misuse, additional ethical considerations must be addressed thoroughly during the research activities. First of all, training various classification/detection systems is inherently a subject of potential biases. Such biases can come directly from the datasets used for the training purposes (in terms of data selection, data pre-/post-processing, or data annotation itself; see Section 9.3) or can be introduced during training the models, especially when fine-tuning pre-trained LLMs that have incorporated biases by themselves (including biases between high- and low-resource languages; see Section 9.2). Secondly, the authors should always clearly formulate intended use and failure cases of the trained models/deployed tools (e.g., by means of model cards). In this way, we may prevent media professionals to over-rely on the predicted (potentially incorrect) values. Fortunately, the two-step approach to credibility assessment (i.e., detect more granular credibility signals and then aggregate them) makes the whole process more transparent. Last but not least, explainability and interpretability of the models’ predictions plays a crucial role in this area, since credibility assessment must be credible itself, otherwise it would not provide expected level of trust to its end users. Besides ethical issues, the research in this area must address also several legal issues. Many of them are shared with other research works on social media. Firstly, as we have also witnessed recently, social media platforms can change their data processing policies and restrict access to their data, which may delay progress in research and development of credibility tools. Additionally, social media data must be anonymized to protect user privacy before being used as training data or for the model inference. Media organizations also impose limitations due to copyright laws, with some not permitting their content to be used in AI tool development. LLMs, especially closed LLMs such as ChatGPT and GPT-4, lack transparency regarding pre-training data and LLMs can memorize content in their pretraining data (Karamolegkou et al., 2023; Mueller et al., 2024). Therefore, anonymization and removal of copyrighted content is crucial to credibility tools, even when they serve as foundational or backbone models. Paradigms such as unlearning (Chen and Yang, 2023) or LLM editing could be potential research directions to tackle these issues. 9.5. Practical deployment Deploying credibility assessment and credibility signal detection models in real-world settings presents significant challenges. For media professionals, editorial guidelines in most newsrooms and fact-checking organizations recommend treating AI-based credibility assessment or fact-checking tools as exploratory aids rather than definitive sources of evidence. Fact-checkers are generally advised to use AI-generated outputs as complementary cues – only after accumulating sufficient independent evidence pointing to the falsity of the content in question. A similar critical challenge concerns the general public, who often lacks the expertise to critically assess AI-detected credibility signals (Lu et al., 2022). This poses serious risks – if such indicators are incorrect or poorly communicated, users may be misled, potentially reinforcing belief in or further disseminating false information. These practical concerns stem from multiple underlying (not only purely technological) challenges. First, explainability, interpretability, and transparent communication of model confidence and limitations are essential components of any credibility assessment system intended for real-world use for both types of end users – media professionals as well as the general public. Many classifiers (e.g., on subjectivity or persuasion techniques) are providing clues that the editor can proofread, maintaining editorial control on the analysis. At the same time, the epistemic shift of the AI-generated content is that humans are now struggling to determine if the content is synthetic or not, and therefore tend to rely more on automated tools, on which exercising editorial control is much more complicated. How can an editor trust an AI-based detector that has an unexplained and non-negligible known rate of false positives? How can she take the reputation risk of writing that the content is synthetic, especially in the context of political life, elections, and public debate, if it proves to be authentic later on? To this end, end users must be able to understand the strengths and weaknesses of models, and model providers must disclose the constraints and limits of the tools they provide (including information on the provenance of the datasets and the training of the models). The previous research showed that appropriate visual explanations foster end users’ trust in AI-predicted classification labels (Yang et al., 2020). In this direction, Przybyła and Soto (2021) proposed a set of interactive visualizations designed to explain the rationale behind automatic credibility assessments, with the aim of increasing users’ confidence in the underlying methods. In a user study involving 14 participants, the authors found that the stylometric classifier was perceived as more interpretable than the neural classifier, although the latter achieved higher predictive performance. Participants were also significantly more accurate and more confident in their credibility judgments after interacting with the visual explanations. These findings underscore that providing meaningful explanations and interpretability – beyond a simple black-box credibility label or score – remains an open challenge. Moreover, the study highlights that addressing this issue is non-trivial, particularly given that state-of-the-art models such as deep neural networks often suffer from limited inherent explainability, despite their high performance. Another critical challenge lies in the communication gap between technical practitioners and end users, particularly media professionals and the general public. These groups often operate using distinct terminologies and conceptual frameworks, which can hinder the interpretation of AI-generated outputs or the understanding of AI system limitations. Furthermore, many existing datasets and benchmarks have been developed primarily by computational linguists or computer scientists, without active involvement of end users. This can lead to divergent perspectives regarding what constitutes credible content or when a credibility signal is present. As our survey shows, certain credibility signal detection tasks, such as bias detection (Section 5) and check-worthiness assessment (Section 7), are inherently subjective, which may further exacerbate these discrepancies. Consequently, model predictions may align poorly with end users’ expectations or only succeed on simple unambiguous examples. Second, detection models making use of credibility signals should minimize false positive rates as much as possible for integration in media organisations and journalism toolkits. However, based on both literature and our own practical experience, we observe that model performance often deteriorates when transitioning from offline evaluations to real-world applications. This discrepancy largely stems from the out-of-distribution (OOD) nature of deployment data, which may differ from training and testing datasets in various aspects, such as topic, format or language. These shifts highlight the need for more robust and generalizable models. However, achieving such robustness is complicated by the already-discussed limited availability of diverse datasets (Section 9.3) and the dynamic nature of online environments. Furthermore, deployed methods and tools for automatic credibility assessment are inherently exposed to potential adversarial threats. Malicious actors who intentionally disseminate non-credible information may attempt to evade detection by slightly altering the content – creating adversarial examples that exploit vulnerabilities in detection models and trigger incorrect classifications (Przybyła et al., 2024). To be effective in real-world applications, automatic classifiers must therefore be robust to such adversarial manipulations. Recent studies have demonstrated that LLMs, when combined with diversity incentives techniques, can be effectively leveraged for data augmentation (Cegin et al., 2024). By generating diverse paraphrases and content variations, such approaches not only improve model generalization but also increase resilience against potential adversarial attacks – including those generated using similar LLM-based strategies. From a technical deployment perspective, resource constraints present additional barriers. While from the research perspective, open-source models are desirable for their transparency and reproducibility, hosting and running multiple models can be financially burdensome for media organizations or academic institutions. Researchers and practitioners often face a trade-off between model size and deployment feasibility: larger language models generally yield higher classification performance but require costly GPU infrastructure to support real-time inference. This necessitates compromises between computational cost and acceptable performance degradation. Techniques such as model distillation, which transfer knowledge from large models into smaller more efficient versions, may alleviate some of these challenges, but often at the cost of reduced performance. 9.6. Multimodal approaches While text is still one of the most common methods to spread information, in practice, it is commonly combined with other media types, like images, videos and audios. The demand for multimodal approaches also grows with a continuous shift of social media platforms towards short multimedia formats, such as short videos, reels or slideshows. While Natural Language Processing (NLP) allows to detect linguistic patterns that may signal low credibility content (as shown throughout this survey), it can overlook cues that are often found in such non-textual elements. For example, low-credibility content can back-up its misleading text with an altered or fully generated image or video to enhance its perceived credibility. As previously discussed in Section 8.7, there are promising results on multimodal approaches that involve processing multiple data streams simultaneously, offering a richer understanding of the content and its credibility. The main potential of these methods lies in their ability to overcome limitations inherent in text-only analysis. To this end, they combine traditional NLP methods with analysis of visual, auditive, contextual or interactive content. Even in such multimodal systems, textual analysis still remains an important component as text provides rich semantic information, often containing the explicit claims or arguments being made. Without appropriate text analysis, it would be difficult to discern the specific intent behind multimedia content. In the future, there is potential to enhance multimodal models by integrating additional modalities, such as biometric signals (e.g., eye-tracking data (Hansen et al., 2020)), interactive content quality, or real-time user engagement metrics. Research on multimodal credibility assessment is therefore still ongoing and provides several promising directions for future research. 10. Conclusions With the rapid development of generative AI, the potential misuse of LLMs for hybrid operations has become one of the most frequently cited risks (Goldstein et al., 2023; Buchanan et al., 2021). Recent research has highlighted the particularly high capacity of LLMs to generate (personalized) disinformation across both global and local narratives (Vykopal et al., 2024; Zugecova et al., 2025). In this context, the large-scale capability to automatically assess the credibility of online social media content becomes even more crucial as it was in the pre-LLM era. Potential negative impact on society and democratic values was recognized by the research community and previously led to emergence of many false information detection approaches. Unfortunately, automatic credibility assessment and the detection of credibility signals have not yet received the same level of attention. At this point, we emphasize that credibility assessment, while closely related to false information detection, provides complementary insights. In particular, false information that appears credible can have far more detrimental effects than content that is easily recognized as non-credible. Thanks to inherent two-step nature of credibility assessment (to detect credibility signals and then to aggregate them into a credibility label/score), credibility assessment encompasses a high level of explainability and wide opportunities for application in research as well as in practice. Detected high-credible content can be promoted by recommender systems or search engines, or highlighted in a user interface of social media platforms, while low-credible content can be accompanied with a low-credibility warning. With a breakdown of predicted credibility score to individual credibility indicators, end users (either media professionals or the general public) are able to explore the evidence leading to the credibility assessment or manually evaluate individual detected credibility signals and make a final assessment by themselves. Such level of explainability and pro-active human involvement in the decision process is vital and unfortunately lacks in many current false information detection approaches that commonly result into a single predicted value (commonly a binary one) with challenging or even impossible explanation caused by a black-box nature of the employed techniques (e.g., deep learning approaches). Despite these considerable advantages of credibility assessment, the current state-of-the-art research works suffer from multiple drawbacks. Most crucially, credibility assessment field can be characterized as highly fragmented. On one side, there are credibility assessment approaches that automatically detect credibility signals and aggregate them to make a final prediction about the content credibility. The utilized credibility signals are, however, mostly shallow linguistic ones (such as a number of hashtags), their automatic detection relies mostly on outdated and simple methods (like rule- or heuristic-based techniques), and also prediction utilizes mostly basic weighting schemata. There are only very few approaches that are in line with the current state of the art (deep learning, LLMs), such as (Leite et al., 2024); or building upon the previous research results to detect more complex credibility signals, such as (Qureshi and Malick, 2023). On the other side, there are automatic approaches detecting various categories of credibility signals, like event factuality, biases, persuasion techniques, or previously fact-checked claims. The prevalence of state-of-the-art techniques (including LLMs and various fine-tuning approaches, including PEFTs) is much higher in these works. However, such approaches remain isolated from credibility assessment, many times even not mentioning that their prediction can be considered as one of more advanced and more reliable credibility signals. To address this undesired gap between research works, lack of interconnection of research results, as well as hindering application of the outcomes in practice, we conducted this systematic survey on automatic credibility assessment and detection of credibility signals from the NLP perspective. By collecting and describing 175 research papers, we not only systematically summarised the current state of the research in (currently fragmented) research areas, but also identified shared challenges and potential for future research. Our thorough analyses and discussions aim to point to an interesting avenues for future research – particularly, we would like to highlight three most promising research streams: - Adoption of advanced multilingual and multimodal LLMs. The emerging multilingual and multimodal LLMs offer substantial – yet still largely untapped – potential for advancing credibility assessment. Their capabilities can significantly improve both detection performance and language coverage. Moreover, these models open new avenues for deploying credibility assessment tools to support both media professionals and the general public. Such tools can serve both purposes – to detect non-credible content as well as to identify credible one that is worth further reading and sharing in social media environment. - Development of multilingual and multi-category benchmark datasets. The current lack of standardized, multilingual benchmark datasets hampers direct comparison between methods and slows progress in the field. We advocate for future work to focus on unifying existing – but often fragmented – efforts into a shared benchmark that would include diverse credibility signals annotated over the same content. Such a resource would enable novel research opportunities – such as multi-task learning or novel credibility assessment algorithms – and at the same time, would potentially reduce current redundant efforts in dataset creation. Especially existing highly multilingual datasets used for previously fact-checked claim retrieval could serve as a foundation for building such a benchmark. - Addressing practical deployment challenges. Finally, bridging the gap between academic research and real-world deployment requires greater attention to practical considerations – ranging from ethical and legal implications to the robustness of models against out-of-distribution and even adversarial inputs. Additionally, issues of computational efficiency and explainability must be addressed to ensure credibility assessment systems are both effective, trustful and reliable. Despite these challenges, existing tools already demonstrate the positive real-world impact such systems can have, both in supporting media professionals and enhancing the information ecosystem for everyday users. Acknowledgements. This work was partially supported by the European Union under the Horizon Europe projects: vera.ai (GA No. 101070093), AI-CODE (GA No. 101135437), and by AI4TRUST (GA No. 101070190); by the UK’s innovation agency (Innovate UK) grant 10039055; by EU NextGenerationEU through the Recovery and Resilience Plan for Slovakia under the project No. 09I03-03-V03-00020. References - (1) - A.b. et al. (2023) Athira A.b., S. D. Madhu Kumar, and Anu Mary Chacko. 2023. A Systematic Survey on Explainable AI Applied to Fake News Detection. Engineering Applications of Artificial Intelligence 122 (June 2023), 106087. https://doi.org/10.1016/j.engappai.2023.106087 - Abdel-Salam (2023) Reem Abdel-Salam. 2023. rematchka at ArAIEval Shared Task: Prefix-Tuning & Prompt-tuning for Improved Detection of Propaganda and Disinformation in Arabic Social Media Content. In Proceedings of ArabicNLP 2023, Hassan Sawaf, Samhaa El-Beltagy, Wajdi Zaghouani, Walid Magdy, Ahmed Abdelali, Nadi Tomeh, Ibrahim Abu Farha, Nizar Habash, Salam Khalifa, Amr Keleg, Hatem Haddad, Imed Zitouni, Khalil Mrini, and Rawan Almatham (Eds.). Association for Computational Linguistics, Singapore (Hybrid), 536–542. https://doi.org/10.18653/v1/2023.arabicnlp-1.52 - Abujaber et al. (2021) Dia Abujaber, Ahmed Qarqaz, and Malak A. Abdullah. 2021. LeCun at SemEval-2021 Task 6: Detecting Persuasion Techniques in Text Using Ensembled Pretrained Transformers and Data Augmentation. In Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), Alexis Palmer, Nathan Schneider, Natalie Schluter, Guy Emerson, Aurelie Herbelot, and Xiaodan Zhu (Eds.). Association for Computational Linguistics, Online, 1068–1074. https://doi.org/10.18653/v1/2021.semeval-1.148 - Aggarwal et al. (2020) Swati Aggarwal, Tushar Sinha, Yash Kukreti, and Siddarth Shikhar. 2020. Media bias detection and bias short term impact assessment. Array 6 (2020), 100025. https://doi.org/10.1016/j.array.2020.100025 - Aires et al. (2020a) Victoria Aires, Altigran da Silva, Fabiola Nakamura, and Eduardo Nakamura. 2020a. An Evaluation of Structural Characteristics of Networks to Identify Media Bias in News Portals. In Proceedings of the Brazilian Symposium on Multimedia and the Web (São Luís, Brazil) (WebMedia ’20). Association for Computing Machinery, New York, NY, USA, 225–232. https://doi.org/10.1145/3428658.3431077 - Aires et al. (2020b) Victoria Patricia Aires, Juliana Freire, Fabiola G Nakamura, Altigran Soares da Silva, and Eduardo F Nakamura. 2020b. An information theory approach to detect media bias in news websites. - Akhtar et al. (2023) Mubashara Akhtar, Michael Schlichtkrull, Zhijiang Guo, Oana Cocarascu, Elena Simperl, and Andreas Vlachos. 2023. Multimodal Automated Fact-Checking: A Survey. In Findings of the Association for Computational Linguistics: EMNLP 2023, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, Singapore, 5430–5448. https://doi.org/10.18653/v1/2023.findings-emnlp.361 - Aksenov et al. (2021) Dmitrii Aksenov, Peter Bourgonje, Karolina Zaczynska, Malte Ostendorff, Julian Moreno-Schneider, and Georg Rehm. 2021. Fine-grained Classification of Political Bias in German News: A Data Set and Initial Experiments. In Proceedings of the 5th Workshop on Online Abuse and Harms (WOAH 2021), Aida Mostafazadeh Davani, Douwe Kiela, Mathias Lambert, Bertie Vidgen, Vinodkumar Prabhakaran, and Zeerak Waseem (Eds.). Association for Computational Linguistics, Online, 121–131. https://doi.org/10.18653/v1/2021.woah-1.13 - Al-Eidan et al. (2009) Rasha M. BinSultan Al-Eidan, Hend S. Al-Khalifa, and AbdulMalik S. Al-Salman. 2009. Towards the measurement of Arabic Weblogs credibility automatically. In Proceedings of the 11th International Conference on Information Integration and Web-Based Applications & Services (Kuala Lumpur, Malaysia) (iiWAS ’09). Association for Computing Machinery, New York, NY, USA, 618–622. https://doi.org/10.1145/1806338.1806455 - Alam et al. (2023) Firoj Alam, Alberto Barrón-Cedeño, Gullal S. Cheema, Gautam Kishore Shahi, Sherzod Hakimov, Maram Hasanain, Chengkai Li, Rubén Míguez, Hamdy Mubarak, Wajdi Zaghouani, and Preslav Nakov. 2023. Overview of the CLEF-2023 CheckThat! Lab Task 1 on Check-Worthiness in Multimodal and Multigenre Content. In Working Notes of the Conference and Labs of the Evaluation Forum (CLEF 2023), Thessaloniki, Greece, September 18th to 21st, 2023 (CEUR Workshop Proceedings, Vol. 3497), Mohammad Aliannejadi, Guglielmo Faggioli, Nicola Ferro, and Michalis Vlachos (Eds.). CEUR-WS.org, 219–235. https://ceur-ws.org/Vol-3497/paper-019.pdf - Alam et al. (2022a) Firoj Alam, Stefano Cresci, Tanmoy Chakraborty, Fabrizio Silvestri, Dimiter Dimitrov, Giovanni Da San Martino, Shaden Shaar, Hamed Firooz, and Preslav Nakov. 2022a. A Survey on Multimodal Disinformation Detection. (2022). arXiv:2103.12541 [cs.MM] https://arxiv.org/abs/2103.12541 - Alam et al. (2022b) Firoj Alam, Hamdy Mubarak, Wajdi Zaghouani, Giovanni Da San Martino, and Preslav Nakov. 2022b. Overview of the WANLP 2022 Shared Task on Propaganda Detection in Arabic. In Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP), Houda Bouamor, Hend Al-Khalifa, Kareem Darwish, Owen Rambow, Fethi Bougares, Ahmed Abdelali, Nadi Tomeh, Salam Khalifa, and Wajdi Zaghouani (Eds.). Association for Computational Linguistics, Abu Dhabi, United Arab Emirates (Hybrid), 108–118. https://doi.org/10.18653/v1/2022.wanlp-1.11 - Almotairy et al. (2024) Bodor Moheel Almotairy, Manal Abdullah, and Dimah Hussein Alahmadi. 2024. Dataset for detecting and characterizing Arab computation propaganda on X. Data in Brief 53 (2024), 110089. https://doi.org/10.1016/j.dib.2024.110089 - Alrubaian et al. (2019) Majed Alrubaian, Muhammad Al-Qurishi, Atif Alamri, Mabrook Al-Rakhami, Mohammad Mehedi Hassan, and Giancarlo Fortino. 2019. Credibility in Online Social Networks: A Survey. IEEE Access 7 (2019), 2828–2855. https://doi.org/10.1109/ACCESS.2018.2886314 - Ansar and Goswami (2021) Wazib Ansar and Saptarsi Goswami. 2021. Combating the menace: A survey on characterization and detection of fake news from a data science perspective. International Journal of Information Management Data Insights 1, 2 (2021), 100052. https://doi.org/10.1016/j.jjimei.2021.100052 - Arslan et al. (2020) Fatma Arslan, Naeemul Hassan, Chengkai Li, and Mark Tremayne. 2020. A Benchmark Dataset of Check-Worthy Factual Claims. In Proceedings of the Fourteenth International AAAI Conference on Web and Social Media, ICWSM 2020, Held Virtually, Original Venue: Atlanta, Georgia, USA, June 8-11, 2020, Munmun De Choudhury, Rumi Chunara, Aron Culotta, and Brooke Foucault Welles (Eds.). AAAI Press, 821–829. https://ojs.aaai.org/index.php/ICWSM/article/view/7346 - Atalla et al. (2011) Malik Atalla, Christian Scheel, Ernesto William De Luca, and Sahin Albayrak. 2011. Investigating the applicability of current machine-learning based subjectivity detection algorithms on german texts. In Proceedings of Workshop on Robust Unsupervised and Semisupervised Methods in Natural Language Processing. 17–24. - Atanasova et al. (2018) Pepa Atanasova, Lluís Màrquez, Alberto Barrón-Cedeño, Tamer Elsayed, Reem Suwaileh, Wajdi Zaghouani, Spas Kyuchukov, Giovanni Da San Martino, and Preslav Nakov. 2018. Overview of the CLEF-2018 CheckThat! Lab on Automatic Identification and Verification of Political Claims. Task 1: Check-Worthiness. In Working Notes of CLEF 2018 - Conference and Labs of the Evaluation Forum, Avignon, France, September 10-14, 2018 (CEUR Workshop Proceedings, Vol. 2125), Linda Cappellato, Nicola Ferro, Jian-Yun Nie, and Laure Soulier (Eds.). CEUR-WS.org. https://ceur-ws.org/Vol-2125/invited_paper_13.pdf - Atanasova et al. (2019) Pepa Atanasova, Preslav Nakov, Georgi Karadzhov, Mitra Mohtarami, and Giovanni Da San Martino. 2019. Overview of the CLEF-2019 CheckThat! Lab: Automatic Identification and Verification of Claims. Task 1: Check-Worthiness. In Working Notes of CLEF 2019 - Conference and Labs of the Evaluation Forum, Lugano, Switzerland, September 9-12, 2019 (CEUR Workshop Proceedings, Vol. 2380), Linda Cappellato, Nicola Ferro, David E. Losada, and Henning Müller (Eds.). CEUR-WS.org. https://ceur-ws.org/Vol-2380/paper_269.pdf - Augenstein et al. (2024) Isabelle Augenstein, Timothy Baldwin, Meeyoung Cha, Tanmoy Chakraborty, Giovanni Luca Ciampaglia, David Corney, Renee DiResta, Emilio Ferrara, Scott Hale, Alon Halevy, et al. 2024. Factuality challenges in the era of large language models and opportunities for fact-checking. Nature Machine Intelligence 6, 8 (2024), 852–863. - Augenstein et al. (2019) Isabelle Augenstein, Christina Lioma, Dongsheng Wang, Lucas Chaves Lima, Casper Hansen, Christian Hansen, and Jakob Grue Simonsen. 2019. MultiFC: A Real-World Multi-Domain Dataset for Evidence-Based Fact Checking of Claims. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (Eds.). Association for Computational Linguistics, Hong Kong, China, 4685–4697. https://doi.org/10.18653/v1/D19-1475 - Baier et al. (2022) Christian Baier, Max Göttsche, Axel Hellmann, et al. 2022. Too Good To Be True: Influencing Credibility Perceptions with Signaling Reference Explicitness and Assurance Depth. In Journal of Business Ethics, Vol. 178. 695–714. https://doi.org/10.1007/s10551-020-04719-7 - Baisa et al. (2019) Vít Baisa, Ondřej Herman, and Ales Horak. 2019. Benchmark Dataset for Propaganda Detection in Czech Newspaper Texts. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019), Ruslan Mitkov and Galia Angelova (Eds.). INCOMA Ltd., Varna, Bulgaria, 77–83. https://doi.org/10.26615/978-954-452-056-4_010 - Baly et al. (2020a) Ramy Baly, Giovanni Da San Martino, James Glass, and Preslav Nakov. 2020a. We Can Detect Your Bias: Predicting the Political Ideology of News Articles. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (Eds.). Association for Computational Linguistics, Online, 4982–4991. https://doi.org/10.18653/v1/2020.emnlp-main.404 - Baly et al. (2018) Ramy Baly, Georgi Karadzhov, Dimitar Alexandrov, James Glass, and Preslav Nakov. 2018. Predicting Factuality of Reporting and Bias of News Media Sources. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun’ichi Tsujii (Eds.). Association for Computational Linguistics, Brussels, Belgium, 3528–3539. https://doi.org/10.18653/v1/D18-1389 - Baly et al. (2020b) Ramy Baly, Georgi Karadzhov, Jisun An, Haewoon Kwak, Yoan Dinkov, Ahmed Ali, James Glass, and Preslav Nakov. 2020b. What Was Written vs. Who Read It: News Media Profiling Using Text Analysis and Social Media Context. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (Eds.). Association for Computational Linguistics, Online, 3364–3374. https://doi.org/10.18653/v1/2020.acl-main.308 - Baly et al. (2019) Ramy Baly, Georgi Karadzhov, Abdelrhman Saleh, James Glass, and Preslav Nakov. 2019. Multi-Task Ordinal Regression for Jointly Predicting the Trustworthiness and the Leading Political Ideology of News Media. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Jill Burstein, Christy Doran, and Thamar Solorio (Eds.). Association for Computational Linguistics, Minneapolis, Minnesota, 2109–2116. https://doi.org/10.18653/v1/N19-1216 - Banea et al. (2010) Carmen Banea, Rada Mihalcea, and Janyce Wiebe. 2010. Multilingual Subjectivity: Are More Languages Better?. In Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010), Chu-Ren Huang and Dan Jurafsky (Eds.). Coling 2010 Organizing Committee, Beijing, China, 28–36. https://aclanthology.org/C10-1004 - Bang et al. (2024) Yejin Bang, Delong Chen, Nayeon Lee, and Pascale Fung. 2024. Measuring political bias in large language models: What is said and how it is said. arXiv preprint arXiv:2403.18932 (2024). - Bao et al. (2024) Guangsheng Bao, Yanbin Zhao, Zhiyang Teng, Linyi Yang, and Yue Zhang. 2024. Fast-DetectGPT: Efficient Zero-Shot Detection of Machine-Generated Text via Conditional Probability Curvature. arXiv:2310.05130 [cs.CL] https://arxiv.org/abs/2310.05130 - Barbosa and Feng (2010) Luciano Barbosa and Junlan Feng. 2010. Robust sentiment detection on twitter from biased and noisy data. In Coling 2010: Posters. 36–44. - Barrett and Barrington (2005) Andrew W. Barrett and Lowell W. Barrington. 2005. Bias in Newspaper Photograph Selection. Political Research Quarterly (formerly WPQ) 58 (2005), 609 – 618. https://api.semanticscholar.org/CorpusID:154387178 - Barrón-Cedeño et al. (2023) Alberto Barrón-Cedeño, Firoj Alam, Tommaso Caselli, Giovanni Da San Martino, Tamer Elsayed, Andrea Galassi, Fatima Haouari, Federico Ruggeri, Julia Maria Struß, Rabindra Nath Nandi, Gullal S. Cheema, Dilshod Azizov, and Preslav Nakov. 2023. The CLEF-2023 CheckThat! Lab: Checkworthiness, Subjectivity, Political Bias, Factuality, and Authority. In Advances in Information Retrieval: 45th European Conference on Information Retrieval, ECIR 2023, Dublin, Ireland, April 2–6, 2023, Proceedings, Part III (Dublin, Ireland). Springer-Verlag, Berlin, Heidelberg, 506–517. https://doi.org/10.1007/978-3-031-28241-6_59 - Barrón-Cedeño et al. (2024) Alberto Barrón-Cedeño, Firoj Alam, Tanmoy Chakraborty, Tamer Elsayed, Preslav Nakov, Piotr Przybyła, Julia Maria Struß, Fatima Haouari, Maram Hasanain, Federico Ruggeri, Xingyi Song, and Reem Suwaileh. 2024. The CLEF-2024 CheckThat! Lab: Check-Worthiness, Subjectivity, Persuasion, Roles, Authorities, and Adversarial Robustness. In Advances in Information Retrieval: 46th European Conference on Information Retrieval, ECIR 2024, Glasgow, UK, March 24–28, 2024, Proceedings, Part V (Glasgow, United Kingdom). Springer-Verlag, Berlin, Heidelberg, 449–458. https://doi.org/10.1007/978-3-031-56069-9_62 - Barrón-Cedeño et al. (2023) Alberto Barrón-Cedeño, Firoj Alam, Andrea Galassi, Giovanni Da San Martino, Preslav Nakov, Tamer Elsayed, Dilshod Azizov, Tommaso Caselli, Gullal S. Cheema, Fatima Haouari, Maram Hasanain, Mucahid Kutlu, Chengkai Li, Federico Ruggeri, Julia Maria Struß, and Wajdi Zaghouani. 2023. Overview of the CLEF–2023 CheckThat! Lab on Checkworthiness, Subjectivity, Political Bias, Factuality, and Authority of News Articles and Their Source. In Experimental IR Meets Multilinguality, Multimodality, and Interaction, Avi Arampatzis, Evangelos Kanoulas, Theodora Tsikrika, Stefanos Vrochidis, Anastasia Giachanou, Dan Li, Mohammad Aliannejadi, Michalis Vlachos, Guglielmo Faggioli, and Nicola Ferro (Eds.). Springer Nature Switzerland, Cham, 251–275. - Barrón-Cedeño et al. (2020) Alberto Barrón-Cedeño, Tamer Elsayed, Preslav Nakov, Giovanni Da San Martino, Maram Hasanain, Reem Suwaileh, Fatima Haouari, Nikolay Babulkov, Bayan Hamdan, Alex Nikolov, Shaden Shaar, and Zien Sheikh Ali. 2020. Overview of CheckThat! 2020: Automatic Identification and Verification of Claims in Social Media. In Experimental IR Meets Multilinguality, Multimodality, and Interaction - 11th International Conference of the CLEF Association, CLEF 2020, Thessaloniki, Greece, September 22-25, 2020, Proceedings (Lecture Notes in Computer Science, Vol. 12260), Avi Arampatzis, Evangelos Kanoulas, Theodora Tsikrika, Stefanos Vrochidis, Hideo Joho, Christina Lioma, Carsten Eickhoff, Aurélie Névéol, Linda Cappellato, and Nicola Ferro (Eds.). Springer, 215–236. https://doi.org/10.1007/978-3-030-58219-7_17 - Beltagy et al. (2020) Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The Long-Document Transformer. arXiv:2004.05150 [cs.CL] https://arxiv.org/abs/2004.05150 - Benson and Cruickshank (2024) Seth P Benson and Iain J Cruickshank. 2024. Developing a Natural Language Understanding Model to Characterize Cable News Bias. IEEE Access (2024). - Berendt et al. (2021) Bettina Berendt, Peter Burger, Rafael Hautekiet, Jan Jagers, Alexander Pleijter, and Peter Van Aelst. 2021. FactRank: Developing automated claim detection for Dutch-language fact-checkers. Online Social Networks and Media 22 (2021), 100113. https://doi.org/10.1016/j.osnem.2020.100113 - Bhatnagar et al. (2022) Varad Bhatnagar, Diptesh Kanojia, and Kameswari Chebrolu. 2022. Harnessing Abstractive Summarization for Fact-Checked Claim Detection. In Proceedings of the 29th International Conference on Computational Linguistics, Nicoletta Calzolari, Chu-Ren Huang, Hansaem Kim, James Pustejovsky, Leo Wanner, Key-Sun Choi, Pum-Mo Ryu, Hsin-Hsi Chen, Lucia Donatelli, Heng Ji, Sadao Kurohashi, Patrizia Paggio, Nianwen Xue, Seokhwan Kim, Younggyun Hahm, Zhong He, Tony Kyungil Lee, Enrico Santus, Francis Bond, and Seung-Hoon Na (Eds.). International Committee on Computational Linguistics, Gyeongju, Republic of Korea, 2934–2945. https://aclanthology.org/2022.coling-1.259 - Bhuiyan et al. (2020) Md Momen Bhuiyan, Amy X. Zhang, Connie Moon Sehat, and Tanushree Mitra. 2020. Investigating Differences in Crowdsourced News Credibility Assessment: Raters, Tasks, and Expert Criteria. Proc. ACM Hum.-Comput. Interact. 4, CSCW2, Article 93 (oct 2020), 26 pages. https://doi.org/10.1145/3415164 - Bin-Habtoor and Zaher (2012) A. Bin-Habtoor and Mahmoud Zaher. 2012. A Survey on Plagiarism Detection Systems. International Journal of Computer Theory and Engineering (01 2012), 185–188. https://doi.org/10.7763/IJCTE.2012.V4.447 - Biyani et al. (2014) Prakhar Biyani, Sumit Bhatia, Cornelia Caragea, and Prasenjit Mitra. 2014. Using non-lexical features for identifying factual and opinionative threads in online forums. Knowledge-Based Systems 69 (2014), 170–178. https://doi.org/10.1016/j.knosys.2014.04.048 - Bosco et al. (2018) Cristina Bosco, Felice Dell’Orletta, Fabio Poletto, Manuela Sanguinetti, and Maurizio Tesconi. 2018. Overview of the EVALITA 2018 Hate Speech Detection Task. In Proceedings of the Sixth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2018) co-located with the Fifth Italian Conference on Computational Linguistics (CLiC-it 2018), Turin, Italy. - Botella-Gil et al. (2024) Beatriz Botella-Gil, Alba Bonet-Jover, Robiert Sepúlveda-Torres, Patricio Martínez-Barco, and Estela Saquete. 2024. Exploring the Relationship between News Reliability and Violent Comments in Digital Media. In Proceedings of the 40th International Conference of the Spanish Society for Natural Language Processing. - Botnevik et al. (2020) Bjarte Botnevik, Eirik Sakariassen, and Vinay Setty. 2020. BRENDA: Browser Extension for Fake News Detection. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval (Virtual Event, China) (SIGIR ’20). Association for Computing Machinery, New York, NY, USA, 2117–2120. https://doi.org/10.1145/3397271.3401396 - Bourgeade et al. (2023) Tom Bourgeade, Alessandra Teresa Cignarella, Simona Frenda, Mario Laurent, Wolfgang Schmeisser-Nieto, Farah Benamara, Cristina Bosco, Véronique Moriceau, Viviana Patti, and Mariona Taulé. 2023. A Multilingual Dataset of Racial Stereotypes in Social Media Conversational Threads. In Findings of the Association for Computational Linguistics: EACL 2023. 674–684. - Breum et al. (2024) Simon Martin Breum, Daniel Vædele Egdal, Victor Gram Mortensen, Anders Giovanni Møller, and Luca Maria Aiello. 2024. The Persuasive Power of Large Language Models. Proceedings of the International AAAI Conference on Web and Social Media 18, 1 (May 2024), 152–163. https://doi.org/10.1609/icwsm.v18i1.31304 - Briscoe et al. (2013) Erica J. Briscoe, D. Scott Appling, Rudolph L. Mappus, and Heather Hayes. 2013. Determining credibility from social network structure. In Proceedings of the 2013 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (Niagara, Ontario, Canada) (ASONAM ’13). Association for Computing Machinery, New York, NY, USA, 1418–1424. https://doi.org/10.1145/2492517.2492574 - Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Vol. 33. Curran Associates, Inc., 1877–1901. https://proceedings.neurips.cc/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf - Buchanan et al. (2021) Ben Buchanan, Andrew Lohn, Micah Musser, and Katerina Sedova. 2021. Truth, Lies, and Automation: How Language Models Could Change Disinformation. Technical Report. Center for Security and Emerging Technology. https://doi.org/10.51593/2021CA003 - Burkart and Huber (2021) Nadia Burkart and Marco F Huber. 2021. A survey on the explainability of supervised machine learning. Journal of Artificial Intelligence Research 70 (2021), 245–317. - Cao et al. (2022) Jinjuan Cao, Zhong Qian, and Peifeng Li. 2022. End-to-End Event Factuality Identification with Cross-Lingual Information. In 2022 International Joint Conference on Neural Networks (IJCNN). 1–8. https://doi.org/10.1109/IJCNN55064.2022.9892869 - Cao et al. (2021) Pengfei Cao, Yubo Chen, Yuqing Yang, Kang Liu, and Jun Zhao. 2021. Uncertain Local-to-Global Networks for Document-Level Event Factuality Identification. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (Eds.). Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 2636–2645. https://doi.org/10.18653/v1/2021.emnlp-main.207 - Castillo et al. (2011) Carlos Castillo, Marcelo Mendoza, and Barbara Poblete. 2011. Information credibility on twitter. In Proceedings of the 20th International Conference on World Wide Web (Hyderabad, India) (WWW ’11). Association for Computing Machinery, New York, NY, USA, 675–684. https://doi.org/10.1145/1963405.1963500 - Casula et al. (2024) Camilla Casula, Elisa Leonardelli, and Sara Tonelli. 2024. Don’t Augment, Rewrite? Assessing Abusive Language Detection with Synthetic Data. In Findings of the Association for Computational Linguistics ACL 2024, Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association for Computational Linguistics, Bangkok, Thailand and virtual meeting, 11240–11247. https://doi.org/10.18653/v1/2024.findings-acl.669 - Cegin et al. (2024) Jan Cegin, Branislav Pecher, Jakub Simko, Ivan Srba, Maria Bielikova, and Peter Brusilovsky. 2024. Effects of diversity incentives on sample diversity and downstream model performance in LLM-based text augmentation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association for Computational Linguistics, Bangkok, Thailand, 13148–13171. https://doi.org/10.18653/v1/2024.acl-long.710 - Cekinel et al. (2024) Recep Firat Cekinel, Çağrı Çöltekin, and Pinar Karagoz. 2024. Cross-Lingual Learning vs. Low-Resource Fine-Tuning: A Case Study with Fact-Checking in Turkish. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue (Eds.). ELRA and ICCL, Torino, Italia, 4127–4142. https://aclanthology.org/2024.lrec-main.368 - Chang et al. (2021) Yung-Sheng Chang, Yan Zhang, and Jacek Gwizdka. 2021. The effects of information source and eHealth literacy on consumer health information credibility evaluation behavior. Computers in Human Behavior 115 (2021), 106629. https://doi.org/10.1016/j.chb.2020.106629 - Chaturvedi et al. (2018) Iti Chaturvedi, Erik Cambria, Roy E. Welsch, and Francisco Herrera. 2018. Distinguishing between facts and opinions for sentiment analysis: Survey and challenges. Information Fusion 44 (2018), 65–77. https://doi.org/10.1016/j.inffus.2017.12.006 - Cheema et al. (2022) Gullal Singh Cheema, Sherzod Hakimov, Abdul Sittar, Eric Müller-Budack, Christian Otto, and Ralph Ewerth. 2022. MM-Claims: A Dataset for Multimodal Claim Detection in Social Media. In Findings of the Association for Computational Linguistics: NAACL 2022, Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz (Eds.). Association for Computational Linguistics, Seattle, United States, 962–979. https://doi.org/10.18653/v1/2022.findings-naacl.72 - Chen and Shu (2023) Canyu Chen and Kai Shu. 2023. Can llm-generated misinformation be detected? arXiv preprint arXiv:2309.13788 (2023). - Chen and Yang (2023) Jiaao Chen and Diyi Yang. 2023. Unlearn What You Want to Forget: Efficient Unlearning for LLMs. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, Singapore, 12041–12052. https://doi.org/10.18653/v1/2023.emnlp-main.738 - Chen et al. (2020) Wei-Fan Chen, Khalid Al Khatib, Henning Wachsmuth, and Benno Stein. 2020. Analyzing Political Bias and Unfairness in News Articles at Different Levels of Granularity. In Proceedings of the Fourth Workshop on Natural Language Processing and Computational Social Science, David Bamman, Dirk Hovy, David Jurgens, Brendan O’Connor, and Svitlana Volkova (Eds.). Association for Computational Linguistics, Online, 149–154. https://doi.org/10.18653/v1/2020.nlpcss-1.16 - Chen et al. (2015) Yimin Chen, Niall J. Conroy, and Victoria L. Rubin. 2015. Misleading Online Content: Recognizing Clickbait as ”False News”. In Proceedings of the 2015 ACM on Workshop on Multimodal Deception Detection (Seattle, Washington, USA) (WMDD ’15). Association for Computing Machinery, New York, NY, USA, 15–19. https://doi.org/10.1145/2823465.2823467 - Chernyavskiy et al. (2020) Anton Chernyavskiy, Dmitry Ilvovsky, and Preslav Nakov. 2020. Aschern at SemEval-2020 Task 11: It Takes Three to Tango: RoBERTa, CRF, and Transfer Learning. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, Aurelie Herbelot, Xiaodan Zhu, Alexis Palmer, Nathan Schneider, Jonathan May, and Ekaterina Shutova (Eds.). International Committee for Computational Linguistics, Barcelona (online), 1462–1468. https://doi.org/10.18653/v1/2020.semeval-1.191 - Choi et al. (2023) Wonchan Choi, Besiki Stvilia, and Hyun Seung Lee. 2023. Developing a platform-specific framework for web credibility assessment: A case of social Q&A sites. Information Processing & Management 60, 3 (2023), 103321. https://doi.org/10.1016/j.ipm.2023.103321 - Chong et al. (2010) Miranda Chong, Lucia Specia, and Ruslan Mitkov. 2010. Using natural language processing for automatic plagiarism detection. In Proc. of 4th International Plagiarism Conference, Northrumbia University Newcastle-upon-Tyne, UK. - Choudhary et al. (2024) Monika Choudhary, Satyendra Singh Chouhan, and Santosh Singh Rathore. 2024. Beyond Text: Multimodal Credibility Assessment Approaches for Online User-Generated Content. ACM Trans. Intell. Syst. Technol. (June 2024). https://doi.org/10.1145/3673236 Just Accepted. - Chua et al. (2023) Hui Na Chua, Qaisar Khan, Muhammed Basheer Jasser, and Richard T. K Wong. 2023. Problem Understanding of Fake News Detection from a Data Mining Perspective. In 2023 IEEE 13th International Conference on Control System, Computing and Engineering (ICCSCE). 297–302. https://doi.org/10.1109/ICCSCE58721.2023.10237152 - Ciobanu et al. (2014) Amelia Ciobanu, Tudor Culda, Cristian Negrescu, and Dumitru Stanomir. 2014. Analysis of reverberation time blind estimation used in audio forensics. In 2014 11th International Symposium on Electronics and Telecommunications (ISETC). 1–4. https://doi.org/10.1109/ISETC.2014.7010810 - Cortis and Davis (2022) Keith Cortis and Brian Davis. 2022. Baseline English and Maltese-English Classification Models for Subjectivity Detection, Sentiment Analysis, Emotion Analysis, Sarcasm Detection, and Irony Detection. In Proceedings of the 1st Annual Meeting of the ELRA/ISCA Special Interest Group on Under-Resourced Languages. 161–168. - Crothers et al. (2023) Evan N. Crothers, Nathalie Japkowicz, and Herna L. Viktor. 2023. Machine-Generated Text: A Comprehensive Survey of Threat Models and Detection Methods. IEEE Access 11 (2023), 70977–71002. https://doi.org/10.1109/ACCESS.2023.3294090 - Cui and Lee (2020) Limeng Cui and Dongwon Lee. 2020. CoAID: COVID-19 Healthcare Misinformation Dataset. arXiv:2006.00885 [cs.SI] https://arxiv.org/abs/2006.00885 - Cui et al. (2019) Limeng Cui, Kai Shu, Suhang Wang, Dongwon Lee, and Huan Liu. 2019. dEFEND: A System for Explainable Fake News Detection. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management (Beijing, China) (CIKM ’19). Association for Computing Machinery, New York, NY, USA, 2961–2964. https://doi.org/10.1145/3357384.3357862 - Cunningham et al. (2002) Hamish Cunningham, Diana Maynard, Kalina Bontcheva, and Valentin Tablan. 2002. GATE: an architecture for development of robust HLT applications. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics (Philadelphia, Pennsylvania) (ACL ’02). Association for Computational Linguistics, USA, 168–175. https://doi.org/10.3115/1073083.1073112 - Da San Martino et al. (2019a) Giovanni Da San Martino, Alberto Barrón-Cedeño, and Preslav Nakov. 2019a. Findings of the NLP4IF-2019 Shared Task on Fine-Grained Propaganda Detection. In Proceedings of the Second Workshop on Natural Language Processing for Internet Freedom: Censorship, Disinformation, and Propaganda, Anna Feldman, Giovanni Da San Martino, Alberto Barrón-Cedeño, Chris Brew, Chris Leberknight, and Preslav Nakov (Eds.). Association for Computational Linguistics, Hong Kong, China, 162–170. https://doi.org/10.18653/v1/D19-5024 - Da San Martino et al. (2020a) Giovanni Da San Martino, Alberto Barrón-Cedeño, Henning Wachsmuth, Rostislav Petrov, and Preslav Nakov. 2020a. SemEval-2020 Task 11: Detection of Propaganda Techniques in News Articles. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, Aurelie Herbelot, Xiaodan Zhu, Alexis Palmer, Nathan Schneider, Jonathan May, and Ekaterina Shutova (Eds.). International Committee for Computational Linguistics, Barcelona (online), 1377–1414. https://doi.org/10.18653/v1/2020.semeval-1.186 - Da San Martino et al. (2021) Giovanni Da San Martino, Stefano Cresci, Alberto Barrón-Cedeño, Seunghak Yu, Roberto Di Pietro, and Preslav Nakov. 2021. A survey on computational propaganda detection. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence (Yokohama, Yokohama, Japan) (IJCAI’20). Article 672, 7 pages. - Da San Martino et al. (2020b) Giovanni Da San Martino, Shaden Shaar, Yifan Zhang, Seunghak Yu, Alberto Barrón-Cedeño, and Preslav Nakov. 2020b. Prta: A System to Support the Analysis of Propaganda Techniques in the News. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, Asli Celikyilmaz and Tsung-Hsien Wen (Eds.). Association for Computational Linguistics, Online, 287–293. https://doi.org/10.18653/v1/2020.acl-demos.32 - Da San Martino et al. (2019b) Giovanni Da San Martino, Seunghak Yu, Alberto Barrón-Cedeño, Rostislav Petrov, and Preslav Nakov. 2019b. Fine-Grained Analysis of Propaganda in News Article. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (Eds.). Association for Computational Linguistics, Hong Kong, China, 5636–5646. https://doi.org/10.18653/v1/D19-1565 - Das et al. (2023) Anubrata Das, Houjiang Liu, Venelin Kovatchev, and Matthew Lease. 2023. The state of human-centered NLP technology for fact-checking. Information Processing & Management 60, 2 (2023), 103219. https://doi.org/10.1016/j.ipm.2022.103219 - Derczynski et al. (2017) Leon Derczynski, Kalina Bontcheva, Maria Liakata, Rob Procter, Geraldine Wong Sak Hoi, and Arkaitz Zubiaga. 2017. SemEval-2017 Task 8: RumourEval: Determining rumour veracity and support for rumours. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017). 69–76. - Dhar and Das (2021) Rudra Dhar and Dipankar Das. 2021. Leveraging Expectation Maximization for Identifying Claims in Low Resource Indian Languages. In Proceedings of the 18th International Conference on Natural Language Processing (ICON), Sivaji Bandyopadhyay, Sobha Lalitha Devi, and Pushpak Bhattacharyya (Eds.). NLP Association of India (NLPAI), National Institute of Technology Silchar, Silchar, India, 307–312. https://aclanthology.org/2021.icon-main.37 - Dimitrov et al. (2021) Dimitar Dimitrov, Bishr Bin Ali, Shaden Shaar, Firoj Alam, Fabrizio Silvestri, Hamed Firooz, Preslav Nakov, and Giovanni Da San Martino. 2021. SemEval-2021 Task 6: Detection of Persuasion Techniques in Texts and Images. In Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), Alexis Palmer, Nathan Schneider, Natalie Schluter, Guy Emerson, Aurelie Herbelot, and Xiaodan Zhu (Eds.). Association for Computational Linguistics, Online, 70–98. https://doi.org/10.18653/v1/2021.semeval-1.7 - Edwards and Camacho-Collados (2024) Aleksandra Edwards and Jose Camacho-Collados. 2024. Language Models for Text Classification: Is In-Context Learning Enough?. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue (Eds.). ELRA and ICCL, Torino, Italia, 10058–10072. https://aclanthology.org/2024.lrec-main.879 - Ek and Ghanimifard (2019) Adam Ek and Mehdi Ghanimifard. 2019. Synthetic Propaganda Embeddings To Train A Linear Projection. In Proceedings of the Second Workshop on Natural Language Processing for Internet Freedom: Censorship, Disinformation, and Propaganda, Anna Feldman, Giovanni Da San Martino, Alberto Barrón-Cedeño, Chris Brew, Chris Leberknight, and Preslav Nakov (Eds.). Association for Computational Linguistics, Hong Kong, China, 155–161. https://doi.org/10.18653/v1/D19-5023 - El Ballouli et al. (2017) Rim El Ballouli, Wassim El-Hajj, Ahmad Ghandour, Shady Elbassuoni, Hazem Hajj, and Khaled Shaban. 2017. CAT: Credibility Analysis of Arabic Content on Twitter. In Proceedings of the Third Arabic Natural Language Processing Workshop, Nizar Habash, Mona Diab, Kareem Darwish, Wassim El-Hajj, Hend Al-Khalifa, Houda Bouamor, Nadi Tomeh, Mahmoud El-Haj, and Wajdi Zaghouani (Eds.). Association for Computational Linguistics, Valencia, Spain, 62–71. https://doi.org/10.18653/v1/W17-1308 - Elmimouni et al. (2022) Houda Elmimouni, Andrea Forte, and Jonathan Morgan. 2022. Why People Trust Wikipedia Articles: Credibility Assessment Strategies Used by Readers. In Proceedings of the 18th International Symposium on Open Collaboration (OpenSym ’22). Association for Computing Machinery, New York, NY, USA, Article 9, 10 pages. https://doi.org/10.1145/3555051.3555052 - Elngar et al. (2021) Ahmed A Elngar, Mohamed Gamal, Amar Fathy, Basma Moustafa, Omar Mahmoud, and Mohamed Shaban. 2021. Plagiarism detection algorithm model based on NLP technology. Journal of Cybersecurity and Information Management 5, 1 (2021), 43–61. - Enriquez et al. (2023) Daniel Enriquez, Gage Christensen, Hayden Donovan, Jared Lam, Noah Wong, Sergiu Dascalu, David Feil-Seifer, and Emily Hand. 2023. Authorship Verification for Hired Plagiarism Detection. In Proceedings of the 9th International Conference on Applied Computing & Information Technology (Virtual Event, USA) (ACIT ’22). Association for Computing Machinery, New York, NY, USA, 19–24. https://doi.org/10.1145/3543895.3543928 - Entman (2007) Robert M Entman. 2007. Framing bias: Media in the distribution of power. Journal of communication 57, 1 (2007), 163–173. - Esteves et al. (2018) Diego Esteves, Aniketh Janardhan Reddy, Piyush Chawla, and Jens Lehmann. 2018. Belittling the Source: Trustworthiness Indicators to Obfuscate Fake News on the Web. In Proceedings of the First Workshop on Fact Extraction and VERification (FEVER). 50–59. - Fan et al. (2019) Lisa Fan, Marshall White, Eva Sharma, Ruisi Su, Prafulla Kumar Choubey, Ruihong Huang, and Lu Wang. 2019. In Plain Sight: Media Bias Through the Lens of Factual Reporting. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (Eds.). Association for Computational Linguistics, Hong Kong, China, 6343–6349. https://doi.org/10.18653/v1/D19-1664 - Feng et al. (2023) KJ Kevin Feng, Nick Ritchie, Pia Blumenthal, Andy Parsons, and Amy X Zhang. 2023. Examining the Impact of Provenance-Enabled Media on Trust and Accuracy Perceptions. Proceedings of the ACM on Human-Computer Interaction 7, CSCW2 (2023), 1–42. - Fersini et al. (2018) Elisabetta Fersini, Paolo Rosso, and Maria Anzovino. 2018. Overview of the Task on Automatic Misogyny Identification at IberEval 2018. In IberEval@SEPLN (CEUR Workshop Proceedings, Vol. 2150). CEUR-WS.org, 214–228. - Frick et al. (2023) Raphael Antonius Frick, Inna Vogel, and Jeong-Eun Choi. 2023. Fraunhofer SIT at CheckThat!-2023: Enhancing the Detection of Multimodal and Multigenre Check-Worthiness Using Optical Character Recognition and Model Souping.. In CLEF (Working Notes). 337–350. - Fuhr et al. (2018) Norbert Fuhr, Anastasia Giachanou, Gregory Grefenstette, Iryna Gurevych, Andreas Hanselowski, Kalervo Jarvelin, Rosie Jones, YiquN Liu, Josiane Mothe, Wolfgang Nejdl, Isabella Peters, and Benno Stein. 2018. An Information Nutritional Label for Online Documents. SIGIR Forum 51, 3 (Feb. 2018), 46–66. https://doi.org/10.1145/3190580.3190588 - Gao et al. (2015) Qin Gao, Ye Tian, and Mengyuan Tu. 2015. Exploring factors influencing Chinese user’s perceived credibility of health and safety information on Weibo. Computers in Human Behavior 45 (2015), 21–31. https://doi.org/10.1016/j.chb.2014.11.071 - Gencheva et al. (2017) Pepa Gencheva, Preslav Nakov, Lluís Màrquez, Alberto Barrón-Cedeño, and Ivan Koychev. 2017. A Context-Aware Approach for Detecting Worth-Checking Claims in Political Debates. In Proceedings of the International Conference Recent Advances in Natural Language Processing, RANLP 2017, Ruslan Mitkov and Galia Angelova (Eds.). INCOMA Ltd., Varna, Bulgaria, 267–276. https://doi.org/10.26615/978-954-452-049-6_037 - Ghadery et al. (2021) Erfan Ghadery, Damien Sileo, and Marie-Francine Moens. 2021. LIIR at SemEval-2021 task 6: Detection of Persuasion Techniques In Texts and Images using CLIP features. In Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), Alexis Palmer, Nathan Schneider, Natalie Schluter, Guy Emerson, Aurelie Herbelot, and Xiaodan Zhu (Eds.). Association for Computational Linguistics, Online, 1015–1019. https://doi.org/10.18653/v1/2021.semeval-1.139 - Goldstein et al. (2023) Josh A. Goldstein, Girish Sastry, Micah Musser, Renee DiResta, Matthew Gentzel, and Katerina Sedova. 2023. Generative Language Models and Automated Influence Operations: Emerging Threats and Potential Mitigations. https://doi.org/10.48550/arXiv.2301.04246 arXiv:2301.04246 [cs]. - Gorrell et al. (2019) Genevieve Gorrell, Elena Kochkina, Maria Liakata, Ahmet Aker, Arkaitz Zubiaga, Kalina Bontcheva, and Leon Derczynski. 2019. Semeval-2019 task 7: Rumoureval 2019: Determining rumour veracity and support for rumours. In Proceedings of the 13th International Workshop on Semantic Evaluation: NAACL HLT 2019. Association for Computational Linguistics, 845–854. - Greškovičová et al. (2022) Katarína Greškovičová, Radomír Masaryk, Nikola Synak, and Vladimíra Čavojová. 2022. Superlatives, clickbaits, appeals to authority, poor grammar, or boldface: Is editorial style related to the credibility of online health messages? Frontiers in Psychology 13 (2022). https://doi.org/10.3389/fpsyg.2022.940903 - Grieve and Woodfield (2023) Jack Grieve and Helena Woodfield. 2023. The Language of Fake News. Cambridge University Press. - Guo et al. (2022) Zhijiang Guo, Michael Schlichtkrull, and Andreas Vlachos. 2022. A survey on automated fact-checking. Transactions of the Association for Computational Linguistics 10 (2022), 178–206. - Gupta and Srikumar (2021) Ashim Gupta and Vivek Srikumar. 2021. X-Fact: A New Benchmark Dataset for Multilingual Fact Checking. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (Eds.). Association for Computational Linguistics, Online, 675–682. https://doi.org/10.18653/v1/2021.acl-short.86 - Gupta et al. (2021a) Kshitij Gupta, Devansh Gautam, and Radhika Mamidi. 2021a. Volta at SemEval-2021 Task 6: Towards Detecting Persuasive Texts and Images using Textual and Multimodal Ensemble. In Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), Alexis Palmer, Nathan Schneider, Natalie Schluter, Guy Emerson, Aurelie Herbelot, and Xiaodan Zhu (Eds.). Association for Computational Linguistics, Online, 1075–1081. https://doi.org/10.18653/v1/2021.semeval-1.149 - Gupta et al. (2021b) Shreya Gupta, Parantak Singh, Megha Sundriyal, Md. Shad Akhtar, and Tanmoy Chakraborty. 2021b. LESA: Linguistic Encapsulation and Semantic Amalgamation Based Generalised Claim Detection from Online Content. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, Paola Merlo, Jorg Tiedemann, and Reut Tsarfaty (Eds.). Association for Computational Linguistics, Online, 3178–3188. https://doi.org/10.18653/v1/2021.eacl-main.277 - Gupta and Sharma (2021) Vansh Gupta and Raksha Sharma. 2021. NLPIITR at SemEval-2021 Task 6: RoBERTa Model with Data Augmentation for Persuasion Techniques Detection. In Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), Alexis Palmer, Nathan Schneider, Natalie Schluter, Guy Emerson, Aurelie Herbelot, and Xiaodan Zhu (Eds.). Association for Computational Linguistics, Online, 1061–1067. https://doi.org/10.18653/v1/2021.semeval-1.147 - Haddaway et al. (2022) Neal Haddaway, Matthew Page, Chris Pritchard, and Luke McGuinness. 2022. PRISMA2020: An R package and Shiny app for producing PRISMA 2020‐compliant flow diagrams, with interactivity for optimised digital transparency and Open Synthesis. Campbell Systematic Reviews 18 (03 2022). https://doi.org/10.1002/cl2.1230 - Hadj Ameur and Aliane (2021) Mohamed Seghir Hadj Ameur and Hassina Aliane. 2021. AraCOVID19-MFH: Arabic COVID-19 Multi-label Fake News & Hate Speech Detection Dataset. Procedia Computer Science 189 (2021), 232–241. https://doi.org/10.1016/j.procs.2021.05.086 AI in Computational Linguistics. - Hamborg et al. (2017) Felix Hamborg, Norman Meuschke, and Bela Gipp. 2017. Matrix-based news aggregation: exploring different news perspectives. In Proceedings of the 17th ACM/IEEE Joint Conference on Digital Libraries (Toronto, Ontario, Canada) (JCDL ’17). IEEE Press, 69–78. https://doi.org/10.1109/JCDL.2017.7991561 - Hamborg et al. (2020) Felix Hamborg, Anastasia Zhukova, and Bela Gipp. 2020. Automated identification of media bias by word choice and labeling in news articles. In Proceedings of the 18th Joint Conference on Digital Libraries (Champaign, Illinois) (JCDL ’19). IEEE Press, 196–205. https://doi.org/10.1109/JCDL.2019.00036 - Hammer et al. (2014) Hugo Lewi Hammer, Per Erik Solberg, and Lilja Øvrelid. 2014. Sentiment classification of online political discussions: a comparison of a word-based and dependency-based method. In Proceedings of the 5th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, Alexandra Balahur, Erik van der Goot, Ralf Steinberger, and Andres Montoyo (Eds.). Association for Computational Linguistics, Baltimore, Maryland, 90–96. https://doi.org/10.3115/v1/W14-2616 - Hangloo and Arora (2022) Sakshini Hangloo and Bhavna Arora. 2022. Combating multimodal fake news on social media: methods, datasets, and future perspective. Multimedia Systems 28, 8 (2022), 2391–2422. https://doi.org/10.1007/s00530-022-00966-y - Hans et al. (2024) Abhimanyu Hans, Avi Schwarzschild, Valeriia Cherepanova, Hamid Kazemi, Aniruddha Saha, Micah Goldblum, Jonas Geiping, and Tom Goldstein. 2024. Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated Text. arXiv:2401.12070 [cs.CL] https://arxiv.org/abs/2401.12070 - Hansen et al. (2020) Christian Hansen, Casper Hansen, Jakob Grue Simonsen, Birger Larsen, Stephen Alstrup, and Christina Lioma. 2020. Factuality Checking in News Headlines with Eye Tracking. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval (Virtual Event, China) (SIGIR ’20). Association for Computing Machinery, New York, NY, USA, 2013–2016. https://doi.org/10.1145/3397271.3401221 - Haouari et al. (2021) Fatima Haouari, Maram Hasanain, Reem Suwaileh, and Tamer Elsayed. 2021. ArCOV19-Rumors: Arabic COVID-19 Twitter Dataset for Misinformation Detection. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, Nizar Habash, Houda Bouamor, Hazem Hajj, Walid Magdy, Wajdi Zaghouani, Fethi Bougares, Nadi Tomeh, Ibrahim Abu Farha, and Samia Touileb (Eds.). Association for Computational Linguistics, Kyiv, Ukraine (Virtual), 72–81. https://aclanthology.org/2021.wanlp-1.8 - Hardalov et al. (2022) Momchil Hardalov, Anton Chernyavskiy, Ivan Koychev, Dmitry Ilvovsky, and Preslav Nakov. 2022. CrowdChecked: Detecting Previously Fact-Checked Claims in Social Media. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Yulan He, Heng Ji, Sujian Li, Yang Liu, and Chua-Hui Chang (Eds.). Association for Computational Linguistics, Online only, 266–285. https://aclanthology.org/2022.aacl-main.22 - Harris (1997) Robert Harris. 1997. Evaluating Internet Research Sources. http://www.virtualsalt.com/evaluating-internet-research-sources/. Accessed: 2024-10-01. - Hasanain et al. (2023) Maram Hasanain, Firoj Alam, Hamdy Mubarak, Samir Abdaljalil, Wajdi Zaghouani, Preslav Nakov, Giovanni Da San Martino, and Abed Freihat. 2023. ArAIEval Shared Task: Persuasion Techniques and Disinformation Detection in Arabic Text. In Proceedings of ArabicNLP 2023, Hassan Sawaf, Samhaa El-Beltagy, Wajdi Zaghouani, Walid Magdy, Ahmed Abdelali, Nadi Tomeh, Ibrahim Abu Farha, Nizar Habash, Salam Khalifa, Amr Keleg, Hatem Haddad, Imed Zitouni, Khalil Mrini, and Rawan Almatham (Eds.). Association for Computational Linguistics, Singapore (Hybrid), 483–493. https://doi.org/10.18653/v1/2023.arabicnlp-1.44 - Hassan et al. (2017) Naeemul Hassan, Fatma Arslan, Chengkai Li, and Mark Tremayne. 2017. Toward Automated Fact-Checking: Detecting Check-worthy Factual Claims by ClaimBuster. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (Halifax, NS, Canada) (KDD ’17). Association for Computing Machinery, New York, NY, USA, 1803–1812. https://doi.org/10.1145/3097983.3098131 - Hassan et al. (2015) Naeemul Hassan, Chengkai Li, and Mark Tremayne. 2015. Detecting Check-worthy Factual Claims in Presidential Debates. In Proceedings of the 24th ACM International on Conference on Information and Knowledge Management (Melbourne, Australia) (CIKM ’15). Association for Computing Machinery, New York, NY, USA, 1835–1838. https://doi.org/10.1145/2806416.2806652 - Hassell et al. (2020) Hans J. G. Hassell, John B. Holbein, and Matthew R. Miles. 2020. There is no liberal media bias in which news stories political journalists choose to cover. Science Advances 6 (2020). https://api.semanticscholar.org/CorpusID:215516592 - Hawke ([n. d.]) Sandro Hawke. [n. d.]. Technological Approaches to Improving Credibility Assessment on the Web: Final Community Group Report 11 October 2018. https://www.w3.org/2018/10/credibility-tech/. - Hellmueller and Trilling (2012) Lea Hellmueller and Damian Trilling. 2012. The Credibility of Credibility Measures: A Meta-Analysis in Leading Communication Journals, 1951 to 2011. In World Association for Public Opinion Research (WAPOR). Hongkong. - Heuer and Glassman (2024) Hendrik Heuer and Elena L. Glassman. 2024. Reliability Criteria for News Websites. ACM Trans. Comput.-Hum. Interact. 31, 2, Article 21 (jan 2024), 33 pages. https://doi.org/10.1145/3635147 - Hossain et al. (2021) Tashin Hossain, Jannatun Naim, Fareen Tasneem, Radiathun Tasnia, and Abu Nowshed Chy. 2021. CSECU-DSG at SemEval-2021 Task 6: Orchestrating Multimodal Neural Architectures for Identifying Persuasion Techniques in Texts and Images. In Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), Alexis Palmer, Nathan Schneider, Natalie Schluter, Guy Emerson, Aurelie Herbelot, and Xiaodan Zhu (Eds.). Association for Computational Linguistics, Online, 1088–1095. https://doi.org/10.18653/v1/2021.semeval-1.151 - Hovland and Weiss (1951) Carl I. Hovland and Walter Weiss. 1951. The Influence of Source Credibility on Communication Effectiveness*. Public Opinion Quarterly 15, 4 (01 1951), 635–650. https://doi.org/10.1086/266350 arXiv:https://academic.oup.com/poq/article-pdf/15/4/635/5422100/15-4-635.pdf - Hromadka et al. (2023) Timo Hromadka, Timotej Smolen, Tomas Remis, Branislav Pecher, and Ivan Srba. 2023. KInITVeraAI at SemEval-2023 Task 3: Simple yet Powerful Multilingual Fine-Tuning for Persuasion Techniques Detection. In Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023), Atul Kr. Ojha, A. Seza Doğruöz, Giovanni Da San Martino, Harish Tayyar Madabushi, Ritesh Kumar, and Elisa Sartori (Eds.). Association for Computational Linguistics, Toronto, Canada, 629–637. https://doi.org/10.18653/v1/2023.semeval-1.86 - Hu et al. (2024) Bo Hu, Zhendong Mao, and Yongdong Zhang. 2024. An Overview of Fake News Detection: From a New Perspective. Fundamental Research (Feb. 2024). https://doi.org/10.1016/j.fmre.2024.01.017 - Huang et al. (2019) Dongchen Huang, Yige Zhu, and Eni Mustafaraj. 2019. How Dependable are ”First Impressions” to Distinguish between Real and Fake NewsWebsites?. In Proceedings of the 30th ACM Conference on Hypertext and Social Media (Hof, Germany) (HT ’19). Association for Computing Machinery, New York, NY, USA, 201–210. https://doi.org/10.1145/3342220.3343670 - Huang et al. (2022) Kung-Hsiang Huang, ChengXiang Zhai, and Heng Ji. 2022. CONCRETE: Improving Cross-lingual Fact-checking with Cross-lingual Retrieval. In Proceedings of the 29th International Conference on Computational Linguistics, Nicoletta Calzolari, Chu-Ren Huang, Hansaem Kim, James Pustejovsky, Leo Wanner, Key-Sun Choi, Pum-Mo Ryu, Hsin-Hsi Chen, Lucia Donatelli, Heng Ji, Sadao Kurohashi, Patrizia Paggio, Nianwen Xue, Seokhwan Kim, Younggyun Hahm, Zhong He, Tony Kyungil Lee, Enrico Santus, Francis Bond, and Seung-Hoon Na (Eds.). International Committee on Computational Linguistics, Gyeongju, Republic of Korea, 1024–1035. https://aclanthology.org/2022.coling-1.86 - Hughes et al. (2024) Emelia May Hughes, Renee Wang, Prerna Juneja, Tony W Li, Tanushree Mitra, and Amy X. Zhang. 2024. Viblio: Introducing Credibility Signals and Citations to Video-Sharing Platforms. In Proceedings of the CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’24). Association for Computing Machinery, New York, NY, USA, Article 807, 20 pages. https://doi.org/10.1145/3613904.3642490 - Hussein et al. (2022) Ahmed Samir Hussein, Abu Bakr Soliman Mohammad, Mohamed Ibrahim, Laila Hesham Afify, and Samhaa R. El-Beltagy. 2022. NGU CNLP atWANLP 2022 Shared Task: Propaganda Detection in Arabic. In Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP), Houda Bouamor, Hend Al-Khalifa, Kareem Darwish, Owen Rambow, Fethi Bougares, Ahmed Abdelali, Nadi Tomeh, Salam Khalifa, and Wajdi Zaghouani (Eds.). Association for Computational Linguistics, Abu Dhabi, United Arab Emirates (Hybrid), 545–550. https://doi.org/10.18653/v1/2022.wanlp-1.66 - Iqbal et al. (2024) Hasan Iqbal, Yuxia Wang, Minghan Wang, Georgi Georgiev, Jiahui Geng, Iryna Gurevych, and Preslav Nakov. 2024. OpenFactCheck: A Unified Framework for Factuality Evaluation of LLMs. arXiv preprint arXiv:2408.11832 (2024). - Ivanov et al. (2024) Petar Ivanov, Ivan Koychev, Momchil Hardalov, and Preslav Nakov. 2024. Detecting Check-Worthy Claims in Political Debates, Speeches, and Interviews Using Audio Data. In ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 12011–12015. https://doi.org/10.1109/ICASSP48485.2024.10447064 - Jahanbakhsh and Karger (2024) Farnaz Jahanbakhsh and David R Karger. 2024. A Browser Extension for in-place Signaling and Assessment of Misinformation. In Proceedings of the CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’24). Association for Computing Machinery, New York, NY, USA, Article 946, 21 pages. https://doi.org/10.1145/3613904.3642473 - Jahanbakhsh et al. (2022) Farnaz Jahanbakhsh, Amy X. Zhang, and David R. Karger. 2022. Leveraging Structured Trusted-Peer Assessments to Combat Misinformation. Proc. ACM Hum.-Comput. Interact. 6, CSCW2, Article 524 (nov 2022), 40 pages. https://doi.org/10.1145/3555637 - Jaradat et al. (2018) Israa Jaradat, Pepa Gencheva, Alberto Barrón-Cedeño, Lluís Màrquez, and Preslav Nakov. 2018. ClaimRank: Detecting Check-Worthy Claims in Arabic and English. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations, Yang Liu, Tim Paek, and Manasi Patwardhan (Eds.). Association for Computational Linguistics, New Orleans, Louisiana, 26–30. https://doi.org/10.18653/v1/N18-5006 - Jaradat et al. (2024) Israa Jaradat, Haiqi Zhang, and Chengkai Li. 2024. On Detecting Cherry-picking in News Coverage Using Large Language Models. arXiv e-prints (2024), arXiv–2401. - Jawahar et al. (2020) Ganesh Jawahar, Muhammad Abdul-Mageed, and Laks Lakshmanan, V.S. 2020. Automatic Detection of Machine Generated Text: A Critical Survey. In Proceedings of the 28th International Conference on Computational Linguistics, Donia Scott, Nuria Bel, and Chengqing Zong (Eds.). International Committee on Computational Linguistics, Barcelona, Spain (Online), 2296–2309. https://doi.org/10.18653/v1/2020.coling-main.208 - Jeronimo et al. (2020) Caio LM Jeronimo, Claudio EC Campelo, Leandro Balby Marinho, Allan Sales, Adriano Veloso, and Roberta Viola. 2020. Computing with subjectivity lexicons. In Proceedings of the Twelfth Language Resources and Evaluation Conference. 3272–3280. - Ji et al. (2023b) Jiaojiao Ji, Yuqi Zhu, and Naipeng Chao. 2023b. A comparison of misinformation feature effectiveness across issues and time on Chinese social media. Information Processing & Management 60, 2 (2023), 103210. https://doi.org/10.1016/j.ipm.2022.103210 - Ji et al. (2023a) Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023a. Survey of Hallucination in Natural Language Generation. ACM Comput. Surv. 55, 12, Article 248 (March 2023), 38 pages. https://doi.org/10.1145/3571730 - Jia et al. (2022) Chenyan Jia, Alexander Boltz, Angie Zhang, Anqing Chen, and Min Kyung Lee. 2022. Understanding Effects of Algorithmic vs. Community Label on Perceived Accuracy of Hyper-partisan Misinformation. Proc. ACM Hum.-Comput. Interact. 6, CSCW2, Article 371 (nov 2022), 27 pages. https://doi.org/10.1145/3555096 - Jiang et al. (2023) Ye Jiang, Xingyi Song, Carolina Scarton, Iknoor Singh, Ahmet Aker, and Kalina Bontcheva. 2023. Categorising Fine-to-Coarse Grained Misinformation: An Empirical Study of the COVID-19 Infodemic. In Proceedings of the 14th International Conference on Recent Advances in Natural Language Processing, Ruslan Mitkov and Galia Angelova (Eds.). INCOMA Ltd., Shoumen, Bulgaria, Varna, Bulgaria, 556–567. https://aclanthology.org/2023.ranlp-1.61 - Jin et al. (2024) Yiping Jin, Leo Wanner, and Alexander Shvets. 2024. GPT-HateCheck: Can LLMs Write Better Functional Tests for Hate Speech Detection?. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue (Eds.). ELRA and ICCL, Torino, Italia, 7867–7885. https://aclanthology.org/2024.lrec-main.694/ - Juneja et al. (2024) Prerna Juneja, Wenjuan Zhang, Alison Marie Smith-Renner, Hemank Lamba, Joel Tetreault, and Alex Jaimes. 2024. Dissecting users’ needs for search result explanations. In Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI ’24). Association for Computing Machinery, New York, NY, USA, Article 841, 17 pages. https://doi.org/10.1145/3613904.3642059 - Jurkiewicz et al. (2020) Dawid Jurkiewicz, Łukasz Borchmann, Izabela Kosmala, and Filip Graliński. 2020. ApplicaAI at SemEval-2020 Task 11: On RoBERTa-CRF, Span CLS and Whether Self-Training Helps Them. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, Aurelie Herbelot, Xiaodan Zhu, Alexis Palmer, Nathan Schneider, Jonathan May, and Ekaterina Shutova (Eds.). International Committee for Computational Linguistics, Barcelona (online), 1415–1424. https://doi.org/10.18653/v1/2020.semeval-1.187 - Kakol et al. (2017) Michal Kakol, Radoslaw Nielek, and Adam Wierzbicki. 2017. Understanding and predicting web content credibility using the content credibility corpus. Information Processing & Management 53, 5 (2017), 1043–1061. - Kang et al. (2012) Byungkyu Kang, John O’Donovan, and Tobias Höllerer. 2012. Modeling topic specific credibility on twitter. In Proceedings of the 2012 ACM International Conference on Intelligent User Interfaces (Lisbon, Portugal) (IUI ’12). Association for Computing Machinery, New York, NY, USA, 179–188. https://doi.org/10.1145/2166966.2166998 - Kangur et al. (2024) Uku Kangur, Roshni Chakraborty, and Rajesh Sharma. 2024. Who Checks the Checkers? Exploring Source Credibility in Twitter’s Community Notes. arXiv preprint arXiv:2406.12444 (2024). - Karamolegkou et al. (2023) Antonia Karamolegkou, Jiaang Li, Li Zhou, and Anders Søgaard. 2023. Copyright Violations and Large Language Models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 7403–7412. - Kartal and Kutlu (2020) Yavuz Selim Kartal and Mucahid Kutlu. 2020. TrClaim-19: The First Collection for Turkish Check-Worthy Claim Detection with Annotator Rationales. In Proceedings of the 24th Conference on Computational Natural Language Learning, Raquel Fernández and Tal Linzen (Eds.). Association for Computational Linguistics, Online, 386–395. https://doi.org/10.18653/v1/2020.conll-1.31 - Kazemi et al. (2023) Ashkan Kazemi, Artem Abzaliev, Naihao Deng, Rui Hou, Scott Hale, Veronica Perez-Rosas, and Rada Mihalcea. 2023. Query Rewriting for Effective Misinformation Discovery. In Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), Jong C. Park, Yuki Arase, Baotian Hu, Wei Lu, Derry Wijaya, Ayu Purwarianti, and Adila Alfa Krisnadhi (Eds.). Association for Computational Linguistics, Nusa Dua, Bali, 398–407. https://doi.org/10.18653/v1/2023.ijcnlp-main.26 - Kazemi et al. (2021) Ashkan Kazemi, Kiran Garimella, Devin Gaffney, and Scott Hale. 2021. Claim Matching Beyond English to Scale Global Fact-Checking. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (Eds.). Association for Computational Linguistics, Online, 4504–4517. https://doi.org/10.18653/v1/2021.acl-long.347 - Khosla et al. (2020) Sopan Khosla, Rishabh Joshi, Ritam Dutt, Alan W Black, and Yulia Tsvetkov. 2020. LTIatCMU at SemEval-2020 Task 11: Incorporating Multi-Level Features for Multi-Granular Propaganda Span Identification. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, Aurelie Herbelot, Xiaodan Zhu, Alexis Palmer, Nathan Schneider, Jonathan May, and Ekaterina Shutova (Eds.). International Committee for Computational Linguistics, Barcelona (online), 1756–1763. https://doi.org/10.18653/v1/2020.semeval-1.230 - Kiili et al. (2007) Carita Kiili, Leena Laurinen, and Miika Marttunen. 2007. How students evaluate credibility and relevance of information on the internet? IADIS International Conference on Cognition and Exploratory Learning in Digital Age, CELDA 2007 (01 2007). - Kim and Bethard (2020) Moonsung Kim and Steven Bethard. 2020. TTUI at SemEval-2020 Task 11: Propaganda Detection with Transfer Learning and Ensembles. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, Aurelie Herbelot, Xiaodan Zhu, Alexis Palmer, Nathan Schneider, Jonathan May, and Ekaterina Shutova (Eds.). International Committee for Computational Linguistics, Barcelona (online), 1829–1834. https://doi.org/10.18653/v1/2020.semeval-1.240 - Kiousis (2001) Spiro Kiousis. 2001. Public Trust or Mistrust? Perceptions of Media Credibility in the Information Age. Mass Communication and Society 4, 4 (Nov. 2001), 381–403. https://doi.org/10.1207/S15327825MCS0404_4 - Klemperer and Brady (2006) V. Klemperer and M. Brady. 2006. Language of the Third Reich: LTI: Lingua Tertii Imperii. Bloomsbury Academic. https://books.google.sk/books?id=kwsleqxx_SMC - Kondo and Abe (2023) Akihiro Kondo and Hironobu Abe. 2023. Travel blogger credibility scoring method for recommendation system. In 2023 17th International Conference on Ubiquitous Information Management and Communication (IMCOM). 1–4. https://doi.org/10.1109/IMCOM56909.2023.10035643 - Krieger et al. (2022) Jan-David Krieger, Timo Spinde, Terry Ruas, Juhi Kulshrestha, and Bela Gipp. 2022. A domain-adaptive pre-training approach for language bias detection in news. In Proceedings of the 22nd ACM/IEEE Joint Conference on Digital Libraries (Cologne, Germany) (JCDL ’22). Association for Computing Machinery, New York, NY, USA, Article 3, 7 pages. https://doi.org/10.1145/3529372.3530932 - Lamsiyah et al. (2023) Salima Lamsiyah, Abdelkader El Mahdaouy, Hamza Alami, Ismail Berrada, and Christoph Schommer. 2023. UL & UM6P at ArAIEval Shared Task: Transformer-based model for Persuasion Techniques and Disinformation detection in Arabic. In Proceedings of ArabicNLP 2023, Hassan Sawaf, Samhaa El-Beltagy, Wajdi Zaghouani, Walid Magdy, Ahmed Abdelali, Nadi Tomeh, Ibrahim Abu Farha, Nizar Habash, Salam Khalifa, Amr Keleg, Hatem Haddad, Imed Zitouni, Khalil Mrini, and Rawan Almatham (Eds.). Association for Computational Linguistics, Singapore (Hybrid), 558–564. https://doi.org/10.18653/v1/2023.arabicnlp-1.55 - Lawson et al. (2020) Patrick Lawson, Carl J. Pearson, Aaron Crowson, and Christopher B. Mayhorn. 2020. Email phishing and signal detection: How persuasion principles and personality influence response patterns and accuracy. Applied Ergonomics 86 (2020), 103084. https://doi.org/10.1016/j.apergo.2020.103084 - Lazaridou et al. (2017) Konstantina Lazaridou, Ralf Krestel, and Felix Naumann. 2017. Identifying Media Bias by Analyzing Reported Speech. In 2017 IEEE International Conference on Data Mining (ICDM). 943–948. https://doi.org/10.1109/ICDM.2017.119 - Lee et al. (2015) Kenton Lee, Yoav Artzi, Yejin Choi, and Luke Zettlemoyer. 2015. Event Detection and Factuality Assessment with Non-Expert Supervision. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, Lluís Màrquez, Chris Callison-Burch, and Jian Su (Eds.). Association for Computational Linguistics, Lisbon, Portugal, 1643–1648. https://doi.org/10.18653/v1/D15-1189 - Leite et al. (2024) João A. Leite, Olesya Razuvayevskaya, Kalina Bontcheva, and Carolina Scarton. 2024. Weakly Supervised Veracity Classification with LLM-Predicted Credibility Signals. arXiv:2309.07601 [cs.CL] https://arxiv.org/abs/2309.07601 - Leonardelli et al. (2021) Elisa Leonardelli, Stefano Menini, Alessio Palmero Aprosio, Marco Guerini, and Sara Tonelli. 2021. Agreeing to Disagree: Annotating Offensive Language Datasets with Annotators’ Disagreement. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 10528–10539. - Lewis et al. (2020) Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Vol. 33. Curran Associates, Inc., 9459–9474. https://proceedings.neurips.cc/paper_files/paper/2020/file/6b493230205f780e1bc26945df7481e5-Paper.pdf - Li and Goldwasser (2021) Chang Li and Dan Goldwasser. 2021. MEAN: Multi-head Entity Aware Attention Networkfor Political Perspective Detection in News Media. In Proceedings of the Fourth Workshop on NLP for Internet Freedom: Censorship, Disinformation, and Propaganda, Anna Feldman, Giovanni Da San Martino, Chris Leberknight, and Preslav Nakov (Eds.). Association for Computational Linguistics, Online, 66–75. https://doi.org/10.18653/v1/2021.nlp4if-1.10 - Li et al. (2024c) Chunyang Li, Hao Peng, Xiaozhi Wang, Yunjia Qi, Lei Hou, Bin Xu, and Juanzi Li. 2024c. MAVEN-Fact: A Large-scale Event Factuality Detection Dataset. arXiv preprint arXiv:2407.15352 (2024). - Li et al. (2020b) Xiaojia Li, Yun Zhang, and Zhong Qian. 2020b. An End-to-End Approach for Document-level Event Factuality Identification in Chinese. In 2020 International Conference on Asian Language Processing (IALP). 215–220. https://doi.org/10.1109/IALP51396.2020.9310484 - Li et al. (2020a) Yichuan Li, Bohan Jiang, Kai Shu, and Huan Liu. 2020a. MM-COVID: A Multilingual and Multimodal Data Repository for Combating COVID-19 Disinformation. arXiv:2011.04088 [cs.SI] https://arxiv.org/abs/2011.04088 - Li et al. (2024a) Yafu Li, Qintong Li, Leyang Cui, Wei Bi, Zhilin Wang, Longyue Wang, Linyi Yang, Shuming Shi, and Yue Zhang. 2024a. MAGE: Machine-generated Text Detection in the Wild. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association for Computational Linguistics, Bangkok, Thailand, 36–53. https://doi.org/10.18653/v1/2024.acl-long.3 - Li et al. (2024b) Yufeng Li, Rrubaa Panchendrarajan, and Arkaitz Zubiaga. 2024b. FactFinders at CheckThat! 2024: Refining Check-worthy Statement Detection with LLMs through Data Pruning. In Working Notes of the Conference and Labs of the Evaluation Forum (CLEF 2024), Grenoble, France, 9-12 September, 2024 (CEUR Workshop Proceedings, Vol. 3740), Guglielmo Faggioli, Nicola Ferro, Petra Galuscáková, and Alba García Seco de Herrera (Eds.). CEUR-WS.org, 520–537. https://ceur-ws.org/Vol-3740/paper-47.pdf - Lin et al. (2011) Chenghua Lin, Yulan He, and Richard Everson. 2011. Sentence Subjectivity Detection with Weakly-Supervised Learning. In Proceedings of 5th International Joint Conference on Natural Language Processing, Haifeng Wang and David Yarowsky (Eds.). Asian Federation of Natural Language Processing, Chiang Mai, Thailand, 1153–1161. https://aclanthology.org/I11-1129 - Lin et al. (2024a) Luyang Lin, Lingzhi Wang, Jinsong Guo, and Kam-Fai Wong. 2024a. Investigating bias in llm-based bias detection: Disparities between llms and human perception. arXiv preprint arXiv:2403.14896 (2024). - Lin et al. (2024b) Luyang Lin, Lingzhi Wang, Xiaoyan Zhao, Jing Li, and Kam-Fai Wong. 2024b. IndiVec: An Exploration of Leveraging Large Language Models for Media Bias Detection with Fine-Grained Bias Indicators. In Findings of the Association for Computational Linguistics: EACL 2024, Yvette Graham and Matthew Purver (Eds.). Association for Computational Linguistics, St. Julian’s, Malta, 1038–1050. https://aclanthology.org/2024.findings-eacl.70 - Lin et al. (2022) Szu-Yin Lin, Yun-Ching Kung, and Fang-Yie Leu. 2022. Predictive intelligence in harmful news identification by BERT-based ensemble learning model with text sentiment analysis. Information Processing & Management 59, 2 (2022), 102872. https://doi.org/10.1016/j.ipm.2022.102872 - Lin et al. (2016) Xialing Lin, Patric R. Spence, and Kenneth A. Lachlan. 2016. Social media and credibility indicators: The effect of influence cues. Computers in Human Behavior 63 (2016), 264–271. https://doi.org/10.1016/j.chb.2016.05.002 - Ling et al. (2024) Eunice Ling, Domenico de Pieri, Evenne Loh, Karen M Scott, Stephen C H Li, and Heather J Medbury. 2024. Evaluation of the Accuracy, Credibility, and Readability of Statin-Related Websites: Cross-Sectional Study. Interact J Med Res 13 (14 Mar 2024), e42849. https://doi.org/10.2196/42849 - Liu et al. (2019a) Siyi Liu, Lei Guo, Kate Mays, Margrit Betke, and Derry Tanti Wijaya. 2019a. Detecting Frames in News Headlines and Its Application to Analyzing News Framing Trends Surrounding U.S. Gun Violence. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), Mohit Bansal and Aline Villavicencio (Eds.). Association for Computational Linguistics, Hong Kong, China, 504–514. https://doi.org/10.18653/v1/K19-1047 - Liu et al. (2019b) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. RoBERTa: A Robustly Optimized BERT Pretraining Approach. CoRR abs/1907.11692 (2019). arXiv:1907.11692 http://arxiv.org/abs/1907.11692 - Lu et al. (2012) Hao Lu, Yanwu Yang, Runsheng Gan, and Nan Zhang. 2012. The research on micro-blog public opinion index and the application of prototype system. In Proceedings of 2012 9th IEEE International Conference on Networking, Sensing and Control. 405–410. https://doi.org/10.1109/ICNSC.2012.6204953 - Lu et al. (2024) Qiang Lu, Yunfei Long, Xia Sun, Jun Feng, and Hao Zhang. 2024. Fact-sentiment incongruity combination network for multimodal sarcasm detection. Information Fusion 104 (2024), 102203. https://doi.org/10.1016/j.inffus.2023.102203 - Lu et al. (2022) Zhuoran Lu, Patrick Li, Weilong Wang, and Ming Yin. 2022. The effects of ai-based credibility indicators on the detection and spread of misinformation under social influence. Proceedings of the ACM on Human-Computer Interaction 6, CSCW2 (2022), 1–27. - Lucas et al. (2022) Jason Lucas, Limeng Cui, Thai Le, and Dongwon Lee. 2022. Detecting False Claims in Low-Resource Regions: A Case Study of Caribbean Islands. In Proceedings of the Workshop on Combating Online Hostile Posts in Regional Languages during Emergency Situations, Tanmoy Chakraborty, Md. Shad Akhtar, Kai Shu, H. Russell Bernard, Maria Liakata, Preslav Nakov, and Aseem Srivastava (Eds.). Association for Computational Linguistics, Dublin, Ireland, 95–102. https://doi.org/10.18653/v1/2022.constraint-1.11 - Lucas et al. (2023) Jason Lucas, Adaku Uchendu, Michiharu Yamashita, Jooyoung Lee, Shaurya Rohatgi, and Dongwon Lee. 2023. Fighting Fire with Fire: The Dual Role of LLMs in Crafting and Detecting Elusive Disinformation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, Singapore, 14279–14305. https://doi.org/10.18653/v1/2023.emnlp-main.883 - Lunardi et al. (2024) Riccardo Lunardi, David La Barbera, and Kevin Roitero. 2024. The Elusiveness of Detecting Political Bias in Language Models. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management. 3922–3926. - Lurie and Mustafaraj (2018) Emma Lurie and Eni Mustafaraj. 2018. Investigating the Effects of Google’s Search Engine Result Page in Evaluating the Credibility of Online News Sources. In Proceedings of the 10th ACM Conference on Web Science (Amsterdam, Netherlands) (WebSci ’18). Association for Computing Machinery, New York, NY, USA, 107–116. https://doi.org/10.1145/3201064.3201095 - Maab et al. (2023) Iffat Maab, Edison Marrese-Taylor, and Yutaka Matsuo. 2023. An Effective Approach for Informational and Lexical Bias Detection. In Proceedings of the Sixth Fact Extraction and VERification Workshop (FEVER), Mubashara Akhtar, Rami Aly, Christos Christodoulopoulos, Oana Cocarascu, Zhijiang Guo, Arpit Mittal, Michael Schlichtkrull, James Thorne, and Andreas Vlachos (Eds.). Association for Computational Linguistics, Dubrovnik, Croatia, 66–77. https://doi.org/10.18653/v1/2023.fever-1.7 - Macagno (2022) Fabrizio Macagno. 2022. Argumentation profiles and the manipulation of common ground. The arguments of populist leaders on Twitter. Journal of Pragmatics 191 (2022), 67–82. https://doi.org/10.1016/j.pragma.2022.01.022 - Macko et al. (2025) Dominik Macko, Jakub Kopál, Robert Moro, and Ivan Srba. 2025. MultiSocial: Multilingual Benchmark of Machine-Generated Text Detection of Social-Media Texts. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (Eds.). Association for Computational Linguistics, Vienna, Austria, 727–752. https://doi.org/10.18653/v1/2025.acl-long.36 - Macko et al. (2023) Dominik Macko, Robert Moro, Adaku Uchendu, Jason Lucas, Michiharu Yamashita, Matúš Pikuliak, Ivan Srba, Thai Le, Dongwon Lee, Jakub Simko, and Maria Bielikova. 2023. MULTITuDE: Large-Scale Multilingual Machine-Generated Text Detection Benchmark. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, Singapore, 9960–9987. https://doi.org/10.18653/v1/2023.emnlp-main.616 - Macko et al. (2024) Dominik Macko, Robert Moro, Adaku Uchendu, Ivan Srba, Jason S Lucas, Michiharu Yamashita, Nafis Irtiza Tripto, Dongwon Lee, Jakub Simko, and Maria Bielikova. 2024. Authorship Obfuscation in Multilingual Machine-Generated Text Detection. In Findings of the Association for Computational Linguistics: EMNLP 2024, Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (Eds.). Association for Computational Linguistics, Miami, Florida, USA, 6348–6368. https://doi.org/10.18653/v1/2024.findings-emnlp.369 - Maks and Vossen (2012) Isa Maks and Piek Vossen. 2012. Building a fine-grained subjectivity lexicon from a web corpus. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Mehmet Uğur Doğan, Bente Maegaard, Joseph Mariani, Asuncion Moreno, Jan Odijk, and Stelios Piperidis (Eds.). European Language Resources Association (ELRA), Istanbul, Turkey, 3070–3076. http://www.lrec-conf.org/proceedings/lrec2012/pdf/1018_Paper.pdf - Mansour et al. (2022) Watheq Mansour, Tamer Elsayed, and Abdulaziz Al-Ali. 2022. Did I See It Before? Detecting Previously-Checked Claims over Twitter. In Advances in Information Retrieval, Matthias Hagen, Suzan Verberne, Craig Macdonald, Christin Seifert, Krisztian Balog, Kjetil Nørvåg, and Vinay Setty (Eds.). Springer International Publishing, Cham, 367–381. - Martín et al. (2022) Alejandro Martín, Javier Huertas-Tato, Álvaro Huertas-García, Guillermo Villar-Rodríguez, and David Camacho. 2022. FacTeR-Check: Semi-automated fact-checking through semantic similarity and natural language inference. Knowledge-Based Systems 251 (2022), 109265. https://doi.org/10.1016/j.knosys.2022.109265 - Matz et al. (2024) Sandra C Matz, Jacob D Teeny, Sumer S Vaid, Heinrich Peters, Gabriella M Harari, and Moran Cerf. 2024. The potential of generative AI for personalized persuasion at scale. Scientific Reports 14, 1 (2024), 4692. - Mesgar and Strube (2018) Mohsen Mesgar and Michael Strube. 2018. A Neural Local Coherence Model for Text Quality Assessment. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun’ichi Tsujii (Eds.). Association for Computational Linguistics, Brussels, Belgium, 4328–4339. https://doi.org/10.18653/v1/D18-1464 - Miller (1939) Clyde R Miller. 1939. The techniques of propaganda. From “how to detect and analyze propaganda,” an address given at town hall. The Center for learning (1939). - Mitra and Gilbert (2021) Tanushree Mitra and Eric Gilbert. 2021. CREDBANK: A Large-Scale Social Media Corpus With Associated Credibility Annotations. Proceedings of the International AAAI Conference on Web and Social Media 9, 1 (Aug. 2021), 258–267. https://doi.org/10.1609/icwsm.v9i1.14625 - Mittal and Nakov (2022) Shubham Mittal and Preslav Nakov. 2022. IITD at WANLP 2022 Shared Task: Multilingual Multi-Granularity Network for Propaganda Detection. In Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP), Houda Bouamor, Hend Al-Khalifa, Kareem Darwish, Owen Rambow, Fethi Bougares, Ahmed Abdelali, Nadi Tomeh, Salam Khalifa, and Wajdi Zaghouani (Eds.). Association for Computational Linguistics, Abu Dhabi, United Arab Emirates (Hybrid), 529–533. https://doi.org/10.18653/v1/2022.wanlp-1.63 - Mittal et al. (2023) Shubham Mittal, Megha Sundriyal, and Preslav Nakov. 2023. Lost in Translation, Found in Spans: Identifying Claims in Multilingual Social Media. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, Singapore, 3887–3902. https://doi.org/10.18653/v1/2023.emnlp-main.236 - Mogadala and Varma (2012) Aditya Mogadala and Vasudeva Varma. 2012. Language independent sentence-level subjectivity analysis with feature selection. In Proceedings of the 26th Pacific Asia Conference on Language, Information, and Computation. 171–180. - Mohanty (2025) Suvendu Mohanty. 2025. Fine-Grained Bias Detection in LLM: Enhancing detection mechanisms for nuanced biases. arXiv preprint arXiv:2503.06054 (2025). - Molina et al. (2021) Maria D Molina, S Shyam Sundar, Thai Le, and Dongwon Lee. 2021. ”Fake News” Is Not Simply False Information: A Concept Explication and Taxonomy of Online Content. American Behavioral Scientist 65, 2 (2021), 180–212. https://doi.org/10.1177/0002764219878224 - Morio et al. (2020) Gaku Morio, Terufumi Morishita, Hiroaki Ozaki, and Toshinori Miyoshi. 2020. Hitachi at SemEval-2020 Task 11: An Empirical Study of Pre-Trained Transformer Family for Propaganda Detection. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, Aurelie Herbelot, Xiaodan Zhu, Alexis Palmer, Nathan Schneider, Jonathan May, and Ekaterina Shutova (Eds.). International Committee for Computational Linguistics, Barcelona (online), 1739–1748. https://doi.org/10.18653/v1/2020.semeval-1.228 - Mosquera and Moreda (2021) Alejandro Mosquera and Paloma Moreda. 2021. SMILE: An Informality Classification Tool for Helping to Assess Quality and Credibility in Web 2.0 Texts. Proceedings of the International AAAI Conference on Web and Social Media 6, 3 (Aug. 2021), 2–7. https://doi.org/10.1609/icwsm.v6i3.14347 - Mourad and Darwish (2013) Ahmed Mourad and Kareem Darwish. 2013. Subjectivity and sentiment analysis of modern standard Arabic and Arabic microblogs. In Proceedings of the 4th workshop on computational approaches to subjectivity, sentiment and social media analysis. 55–64. - Mueller et al. (2024) Felix B Mueller, Rebekka Görge, Anna K Bernzen, Janna C Pirk, and Maximilian Poretschkin. 2024. LLMs and Memorization: On Quality and Specificity of Copyright Compliance. arXiv preprint arXiv:2405.18492 (2024). - Mujahid et al. (2025) Zain Muhammad Mujahid, Dilshod Azizov, Maha Tufail Agro, and Preslav Nakov. 2025. Profiling News Media for Factuality and Bias Using LLMs and the Fact-Checking Methodology of Human Experts. arXiv preprint arXiv:2506.12552 (2025). - Mukund and Srihari (2010) Smruthi Mukund and Rohini K Srihari. 2010. A vector space model for subjectivity classification in Urdu aided by co-training. In Coling 2010: Posters. 860–868. - Multiple authors and Hawke (2019) Multiple authors and Sandro Hawke. 2019. Credibility Signals: Unofficial Draft 26 November 2019. https://credweb.org/signals-20191126. - Nakov et al. (2024) Preslav Nakov, Jisun An, Haewoon Kwak, Muhammad Arslan Manzoor, Zain Muhammad Mujahid, and Husrev Taha Sencar. 2024. A Survey on Predicting the Factuality and the Bias of News Media. (Aug. 2024), 15947–15962. https://doi.org/10.18653/v1/2024.findings-acl.944 - Nakov et al. (2022a) Preslav Nakov, Alberto Barrón-Cedeño, Giovanni Da San Martino, Firoj Alam, Rubén Míguez, Tommaso Caselli, Mücahid Kutlu, Wajdi Zaghouani, Chengkai Li, Shaden Shaar, Hamdy Mubarak, Alex Nikolov, and Yavuz Selim Kartal. 2022a. Overview of the CLEF-2022 CheckThat! Lab Task 1 on Identifying Relevant Claims in Tweets. In Proceedings of the Working Notes of CLEF 2022 - Conference and Labs of the Evaluation Forum, Bologna, Italy, September 5th - to - 8th, 2022 (CEUR Workshop Proceedings, Vol. 3180), Guglielmo Faggioli, Nicola Ferro, Allan Hanbury, and Martin Potthast (Eds.). CEUR-WS.org, 368–392. https://ceur-ws.org/Vol-3180/paper-28.pdf - Nakov et al. (2022b) Preslav Nakov, Giovanni Da San Martino, Firoj Alam, Shaden Shaar, Hamdy Mubarak, and Nikolay Babulkov. 2022b. Overview of the CLEF-2022 CheckThat! Lab Task 2 on Detecting Previously Fact-Checked Claims. In Proceedings of the Working Notes of CLEF 2022 - Conference and Labs of the Evaluation Forum, Vol. 3180. CEUR-WS, 11. https://ceur-ws.org/Vol-3180/paper-29.pdf - Nakov et al. (2021) Preslav Nakov, Giovanni Da San Martino, Tamer Elsayed, Alberto Barrón-Cedeño, Rubén Míguez, Shaden Shaar, Firoj Alam, Fatima Haouari, Maram Hasanain, Watheq Mansour, Bayan Hamdan, Zien Sheikh Ali, Nikolay Babulkov, Alex Nikolov, Gautam Kishore Shahi, Julia Maria Struß, Thomas Mandl, Mücahid Kutlu, and Yavuz Selim Kartal. 2021. Overview of the CLEF-2021 CheckThat! Lab on Detecting Check-Worthy Claims, Previously Fact-Checked Claims, and Fake News. In Experimental IR Meets Multilinguality, Multimodality, and Interaction - 12th International Conference of the CLEF Association, CLEF 2021, Virtual Event, September 21-24, 2021, Proceedings (Lecture Notes in Computer Science, Vol. 12880), K. Selçuk Candan, Bogdan Ionescu, Lorraine Goeuriot, Birger Larsen, Henning Müller, Alexis Joly, Maria Maistro, Florina Piroi, Guglielmo Faggioli, and Nicola Ferro (Eds.). Springer, 264–291. https://doi.org/10.1007/978-3-030-85251-1_19 - Ness et al. (2023) Eric Ness, Arooj Fatima, and Mahdi Maktabdar Oghaz. 2023. Data Driven Model to Investigate Political Bias in Mainstream Media. IEEE Access 11 (2023), 41880–41893. https://doi.org/10.1109/ACCESS.2023.3270630 - Nielsen and McConville (2022) Dan S. Nielsen and Ryan McConville. 2022. MuMiN: A Large-Scale Multilingual Multimodal Fact-Checked Misinformation Social Network Dataset. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval (Madrid, Spain) (SIGIR ’22). Association for Computing Machinery, New York, NY, USA, 3141–3153. https://doi.org/10.1145/3477495.3531744 - Olteanu et al. (2013) Alexandra Olteanu, Stanislav Peshterliev, Xin Liu, and Karl Aberer. 2013. Web Credibility: Features Exploration and Credibility Prediction. In Advances in Information Retrieval, Pavel Serdyukov, Pavel Braslavski, Sergei O. Kuznetsov, Jaap Kamps, Stefan Rüger, Eugene Agichtein, Ilya Segalovich, and Emine Yilmaz (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 557–568. - Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Vol. 35. Curran Associates, Inc., 27730–27744. https://proceedings.neurips.cc/paper_files/paper/2022/file/b1efde53be364a73914f58805a001731-Paper-Conference.pdf - O’Grady (2006) Laura O’Grady. 2006. Future directions for depicting credibility in health care web sites. International Journal of Medical Informatics 75, 1 (2006), 58–65. https://doi.org/10.1016/j.ijmedinf.2005.07.035 Health and the Internet for All. - Page et al. (2021) Matthew Page, David Moher, Patrick Bossuyt, Isabelle Boutron, Tammy Hoffmann, Cynthia Mulrow, Larissa Shamseer, Jennifer Tetzlaff, Elie Akl, Sue Brennan, Roger Chou, Julie Glanville, Jeremy Grimshaw, Asbjørn Hróbjartsson, Manoj Lalu, Tianjing Li, Elizabeth Loder, Evan Mayo-Wilson, Steve Mcdonald, and Joanne Mckenzie. 2021. PRISMA 2020 explanation and elaboration: Updated guidance and exemplars for reporting systematic reviews. BMJ 372 (03 2021), n160. https://doi.org/10.1136/bmj.n160 - Pal et al. (2023) Surya Kant Pal, Omar Junior Raffik, Rita Roy, Vishwakarma Bipin Lalman, Shristee Srivastava, and Bhoomi Sharma. 2023. Automatic Plagiarism Detection Using Natural Language Processing. In 2023 10th International Conference on Computing for Sustainable Global Development (INDIACom). 218–222. - Panchendrarajan and Zubiaga (2024) Rrubaa Panchendrarajan and Arkaitz Zubiaga. 2024. Claim detection for automated fact-checking: A survey on monolingual, multilingual and cross-lingual research. Natural Language Processing Journal 7 (2024), 100066. https://doi.org/10.1016/j.nlp.2024.100066 - Pang and Lee (2004) Bo Pang and Lillian Lee. 2004. A Sentimental Education: Sentiment Analysis Using Subjectivity Summarization Based on Minimum Cuts. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04). Barcelona, Spain, 271–278. https://doi.org/10.3115/1218955.1218990 - Papcunová et al. (2023) Jana Papcunová, Marcel Martončik, Denisa Fedáková, Michal Kentoš, Miroslava Bozogáňová, Ivan Srba, Robert Moro, Matúš Pikuliak, Marián Šimko, and Matúš Adamkovič. 2023. Hate speech operationalization: a preliminary examination of hate speech indicators and their structure. Complex & Intelligent Systems 9, 3 (01 Jun 2023), 2827–2842. https://doi.org/10.1007/s40747-021-00561-0 - Pastorino et al. (2024) Valeria Pastorino, Jasivan A Sivakumar, and Nafise Sadat Moosavi. 2024. Decoding News Narratives: A Critical Analysis of Large Language Models in Framing Detection. arXiv preprint arXiv:2402.11621 (2024). - Patricia Aires et al. (2019) Victoria Patricia Aires, Fabiola G. Nakamura, and Eduardo F. Nakamura. 2019. A Link-based Approach to Detect Media Bias in News Websites. In Companion Proceedings of The 2019 World Wide Web Conference (San Francisco, USA) (WWW ’19). Association for Computing Machinery, New York, NY, USA, 742–745. https://doi.org/10.1145/3308560.3316460 - Patwari et al. (2017) Ayush Patwari, Dan Goldwasser, and Saurabh Bagchi. 2017. TATHYA: A Multi-Classifier System for Detecting Check-Worthy Statements in Political Debates. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management (Singapore, Singapore) (CIKM ’17). Association for Computing Machinery, New York, NY, USA, 2259–2262. https://doi.org/10.1145/3132847.3133150 - Pauli et al. (2023) Amalie Pauli, Rafael Sarabia, Leon Derczynski, and Ira Assent. 2023. TeamAmpa at SemEval-2023 Task 3: Exploring Multilabel and Multilingual RoBERTa Models for Persuasion and Framing Detection. In Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023), Atul Kr. Ojha, A. Seza Doğruöz, Giovanni Da San Martino, Harish Tayyar Madabushi, Ritesh Kumar, and Elisa Sartori (Eds.). Association for Computational Linguistics, Toronto, Canada, 847–855. https://doi.org/10.18653/v1/2023.semeval-1.117 - Pauli et al. (2024) Amalie Brogaard Pauli, Isabelle Augenstein, and Ira Assent. 2024. Measuring and Benchmarking Large Language Models’ Capabilities to Generate Persuasive Language. https://doi.org/10.48550/arXiv.2406.17753 arXiv:2406.17753. - Pavlopoulos et al. (2021) John Pavlopoulos, Jeffrey Sorensen, Léo Laugier, and Ion Androutsopoulos. 2021. SemEval-2021 Task 5: Toxic Spans Detection. In Proceedings of the 15th International Workshop on Semantic Evaluation, SemEval@ACL/IJCNLP 2021, Virtual Event / Bangkok, Thailand, August 5-6, 2021, Alexis Palmer, Nathan Schneider, Natalie Schluter, Guy Emerson, Aurélie Herbelot, and Xiaodan Zhu (Eds.). Association for Computational Linguistics, 59–69. https://doi.org/10.18653/V1/2021.SEMEVAL-1.6 - Pecher et al. (2024a) Branislav Pecher, Jan Cegin, Robert Belanec, Jakub Simko, Ivan Srba, and Maria Bielikova. 2024a. Fighting Randomness with Randomness: Mitigating Optimisation Instability of Fine-Tuning using Delayed Ensemble and Noisy Interpolation. In Findings of the Association for Computational Linguistics: EMNLP 2024, Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (Eds.). Association for Computational Linguistics, Miami, Florida, USA, 11005–11044. https://doi.org/10.18653/v1/2024.findings-emnlp.644 - Pecher et al. (2024b) Branislav Pecher, Ivan Srba, and Maria Bielikova. 2024b. Comparing Specialised Small and General Large Language Models on Text Classification: 100 Labelled Samples to Achieve Break-Even Performance. arXiv:2402.12819 [cs.CL] https://arxiv.org/abs/2402.12819 - Pecher et al. (2024c) Branislav Pecher, Ivan Srba, and Maria Bielikova. 2024c. On Sensitivity of Learning with Limited Labelled Data to the Effects of Randomness: Impact of Interactions and Systematic Choices. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (Eds.). Association for Computational Linguistics, Miami, Florida, USA, 522–556. https://doi.org/10.18653/v1/2024.emnlp-main.32 - Pecher et al. (2024d) Branislav Pecher, Ivan Srba, and Maria Bielikova. 2024d. A Survey on Stability of Learning with Limited Labelled Data and its Sensitivity to the Effects of Randomness. ACM Comput. Surv. 57, 1, Article 19 (Oct. 2024), 40 pages. https://doi.org/10.1145/3691339 - Peng et al. (2025) Qiwei Peng, Robert Moro, Michal Gregor, Ivan Srba, Simon Ostermann, Marian Simko, Juraj Podrouzek, Matúš Mesarčík, Jaroslav Kopčan, and Anders Søgaard. 2025. SemEval-2025 Task 7: Multilingual and Crosslingual Fact-Checked Claim Retrieval. In Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025), Sara Rosenthal, Aiala Rosá, Debanjan Ghosh, and Marcos Zampieri (Eds.). Association for Computational Linguistics, Vienna, Austria, 2498–2511. https://aclanthology.org/2025.semeval-1.323/ - Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP). 1532–1543. - Pérez-Rosas et al. (2018) Verónica Pérez-Rosas, Bennett Kleinberg, Alexandra Lefevre, and Rada Mihalcea. 2018. Automatic Detection of Fake News. In Proceedings of the 27th International Conference on Computational Linguistics, Emily M. Bender, Leon Derczynski, and Pierre Isabelle (Eds.). Association for Computational Linguistics, Santa Fe, New Mexico, USA, 3391–3401. https://aclanthology.org/C18-1287 - Petroni et al. (2019) Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language Models as Knowledge Bases?. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (Eds.). Association for Computational Linguistics, Hong Kong, China, 2463–2473. https://doi.org/10.18653/v1/D19-1250 - Piccolo et al. (2021) Lara Piccolo, Azizah C Blackwood, Tracie Farrell, and Martino Mensio. 2021. Agents for fighting misinformation spread on Twitter: design challenges. In Proceedings of the 3rd Conference on Conversational User Interfaces. 1–7. - Pikuliak et al. (2023) Matúš Pikuliak, Ivan Srba, Robert Moro, Timo Hromadka, Timotej Smoleň, Martin Melišek, Ivan Vykopal, Jakub Simko, Juraj Podroužek, and Maria Bielikova. 2023. Multilingual Previously Fact-Checked Claim Retrieval. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, Singapore, 16477–16500. https://doi.org/10.18653/v1/2023.emnlp-main.1027 - Pisarevskaya and Zubiaga (2025) Dina Pisarevskaya and Arkaitz Zubiaga. 2025. Zero-shot and Few-shot Learning with Instruction-following LLMs for Claim Matching in Automated Fact-checking. In Proceedings of the 31st International Conference on Computational Linguistics, Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend Al-Khalifa, Barbara Di Eugenio, and Steven Schockaert (Eds.). Association for Computational Linguistics, Abu Dhabi, UAE, 9721–9736. https://aclanthology.org/2025.coling-main.650/ - Piskorski et al. (2023) Jakub Piskorski, Nicolas Stefanovitch, Giovanni Da San Martino, and Preslav Nakov. 2023. SemEval-2023 Task 3: Detecting the Category, the Framing, and the Persuasion Techniques in Online News in a Multi-lingual Setup. In Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023), Atul Kr. Ojha, A. Seza Doğruöz, Giovanni Da San Martino, Harish Tayyar Madabushi, Ritesh Kumar, and Elisa Sartori (Eds.). Association for Computational Linguistics, Toronto, Canada, 2343–2361. https://doi.org/10.18653/v1/2023.semeval-1.317 - Pitler and Nenkova (2008) Emily Pitler and Ani Nenkova. 2008. Revisiting readability: a unified framework for predicting text quality. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (Honolulu, Hawaii) (EMNLP ’08). Association for Computational Linguistics, USA, 186–195. - Podgurski et al. (2022) León Podgurski, Karolina Zaczynska, and Georg Rehm. 2022. Evaluating Web Content Using the W3C Credibility Signals. https://doi.org/10.3233/SSW220005 - Przybyła et al. (2024) Piotr Przybyła, Alexander Shvets, and Horacio Saggion. 2024. Verifying the robustness of automatic credibility assessment. Natural Language Processing (2024), 1–29. https://doi.org/10.1017/nlp.2024.54 - Przybyła and Soto (2021) Piotr Przybyła and Axel J. Soto. 2021. When classification accuracy is not enough: Explaining news credibility assessment. Information Processing & Management 58, 5 (2021), 102653. https://doi.org/10.1016/j.ipm.2021.102653 - Qian et al. (2015) Zhong Qian, Peifeng Li, and Qiaoming Zhu. 2015. A two-step approach for event factuality identification. In 2015 International Conference on Asian Language Processing (IALP). 103–106. https://doi.org/10.1109/IALP.2015.7451542 - Qian et al. (2019) Zhong Qian, Peifeng Li, Qiaoming Zhu, and Guodong Zhou. 2019. Document-Level Event Factuality Identification via Adversarial Neural Network. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Jill Burstein, Christy Doran, and Thamar Solorio (Eds.). Association for Computational Linguistics, Minneapolis, Minnesota, 2799–2809. https://doi.org/10.18653/v1/N19-1287 - Qian et al. (2022) Zhong Qian, Heng Zhang, Peifeng Li, Qiaoming Zhu, and Guodong Zhou. 2022. Document-level event factuality identification via machine reading comprehension frameworks with transfer learning. In Proceedings of the 29th International Conference on Computational Linguistics. 2622–2632. - Qiao et al. (2023) Shuofei Qiao, Yixin Ou, Ningyu Zhang, Xiang Chen, Yunzhi Yao, Shumin Deng, Chuanqi Tan, Fei Huang, and Huajun Chen. 2023. Reasoning with Language Model Prompting: A Survey. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (Eds.). Association for Computational Linguistics, Toronto, Canada, 5368–5393. https://doi.org/10.18653/v1/2023.acl-long.294 - Qureshi and Malick (2023) Khubaib Ahmed Qureshi and Rauf Ahmed Shams Malick. 2023. Explainable tweet credibility ranker: A comprehensive credibility solution. Computers and Electrical Engineering 112 (2023), 109028. https://doi.org/10.1016/j.compeleceng.2023.109028 - Qureshi et al. (2021) Khubaib Ahmed Qureshi, Rauf Ahmed Shams Malick, and Muhammad Sabih. 2021. Social Media and Microblogs Credibility: Identification, Theory Driven Framework, and Recommendation. IEEE Access 9 (2021), 137744–137781. https://doi.org/10.1109/ACCESS.2021.3114417 - Ramponi et al. (2025) Alan Ramponi, Marco Rovera, Robert Moro, and Sara Tonelli. 2025. Multilingual vs Crosslingual Retrieval of Fact-Checked Claims: A Tale of Two Approaches. arXiv:2505.22118 [cs.CL] https://arxiv.org/abs/2505.22118 - Ramponi and Tonelli (2022) Alan Ramponi and Sara Tonelli. 2022. Features or Spurious Artifacts? Data-centric Baselines for Fair and Robust Hate Speech Detection. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 3027–3040. - Razuvayevskaya et al. (2024) Olesya Razuvayevskaya, Ben Wu, João A. Leite, Freddy Heppell, Ivan Srba, Carolina Scarton, Kalina Bontcheva, and Xingyi Song. 2024. Comparison between parameter-efficient techniques and full fine-tuning: A case study on multilingual news article classification. PLOS ONE 19, 5 (05 2024), 1–26. https://doi.org/10.1371/journal.pone.0301738 - Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (Eds.). Association for Computational Linguistics, Hong Kong, China, 3982–3992. https://doi.org/10.18653/v1/D19-1410 - Ridnik et al. (2021) Tal Ridnik, Emanuel Ben-Baruch, Nadav Zamir, Asaf Noy, Itamar Friedman, Matan Protter, and Lihi Zelnik-Manor. 2021. Asymmetric Loss for Multi-Label Classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). 82–91. - Riegelsberger et al. (2005) Jens Riegelsberger, M. Angela Sasse, and John D. McCarthy. 2005. Do people trust their eyes more than ears? media bias in detecting cues of expertise. In CHI ’05 Extended Abstracts on Human Factors in Computing Systems (Portland, OR, USA) (CHI EA ’05). Association for Computing Machinery, New York, NY, USA, 1745–1748. https://doi.org/10.1145/1056808.1057012 - Rieh and Danielson (2007) Soo Young Rieh and David R. Danielson. 2007. Credibility: A Multidisciplinary Framework. Annual Review of Information Science and Technology 41, 1 (Jan. 2007), 307–364. https://doi.org/10.1002/aris.2007.1440410114 - Rijo and Waldzus (2023) Angela Rijo and Sven Waldzus. 2023. That’s interesting! The role of epistemic emotions and perceived credibility in the relation between prior beliefs and susceptibility to fake-news. Computers in Human Behavior 141 (2023), 107619. https://doi.org/10.1016/j.chb.2022.107619 - Robertson and Zaragoza (2009) Stephen Robertson and Hugo Zaragoza. 2009. The Probabilistic Relevance Framework: BM25 and Beyond. Foundations and Trends® in Information Retrieval 3, 4 (2009), 333–389. https://doi.org/10.1561/1500000019 - Rodrigo-Ginés et al. (2023) Francisco-Javier Rodrigo-Ginés, Jorge Carrillo-de Albornoz, and Laura Plaza. 2023. A systematic review on media bias detection: What is media bias, how it is expressed, and how to detect it. Expert Systems with Applications (2023), 121641. - Roele (2021) Cees Roele. 2021. WVOQ at SemEval-2021 Task 6: BART for Span Detection and Classification. In Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), Alexis Palmer, Nathan Schneider, Natalie Schluter, Guy Emerson, Aurelie Herbelot, and Xiaodan Zhu (Eds.). Association for Computational Linguistics, Online, 270–274. https://doi.org/10.18653/v1/2021.semeval-1.32 - Rogiers et al. (2024) Alexander Rogiers, Sander Noels, Maarten Buyl, and Tijl De Bie. 2024. Persuasion with Large Language Models: a Survey. https://doi.org/10.48550/arXiv.2411.06837 arXiv:2411.06837 [cs]. - Rovera et al. (2025) Marco Rovera, Serena Cristoforetti, and Sara Tonelli. 2025. ModaFact: Multi-paradigm Evaluation for Joint Event Modality and Factuality Detection. In Proceedings of the 31st International Conference on Computational Linguistics, Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend Al-Khalifa, Barbara Di Eugenio, and Steven Schockaert (Eds.). Association for Computational Linguistics, Abu Dhabi, UAE, 6378–6396. https://aclanthology.org/2025.coling-main.425/ - Rubin and Liddy (2006) Victoria L. Rubin and Elizabeth D. Liddy. 2006. Assessing Credibility of Weblogs. In AAAI Spring Symposium: Computational Approaches to Analyzing Weblogs. https://api.semanticscholar.org/CorpusID:28691245 - Sakketou et al. (2022) Flora Sakketou, Joan Plepi, Riccardo Cervero, Henri-Jacques Geiss, Paolo Rosso, and Lucie Flek. 2022. Factoid: A new dataset for identifying misinformation spreaders and political bias. arXiv preprint arXiv:2205.06181 (2022). - Salek Faramarzi et al. (2023) Noushin Salek Faramarzi, Fateme Hashemi Chaleshtori, Hossein Shirazi, Indrakshi Ray, and Ritwik Banerjee. 2023. Claim Extraction and Dynamic Stance Detection in COVID-19 Tweets. In Companion Proceedings of the ACM Web Conference 2023 (Austin, TX, USA) (WWW ’23 Companion). Association for Computing Machinery, New York, NY, USA, 1059–1068. https://doi.org/10.1145/3543873.3587643 - Sales et al. (2021) Allan Sales, Albin Zehe, Leandro Balby Marinho, Adriano Veloso, Andreas Hotho, and Janna Omeliyanenko. 2021. Assessing Media Bias in Cross-Linguistic and Cross-National Populations. In Proceedings of the International AAAI Conference on Web and Social Media, Vol. 15. 561–572. - Salminen et al. (2020) Joni Salminen, Maximilian Hopf, Shammur A Chowdhury, Soon-gyo Jung, Hind Almerekhi, and Bernard J Jansen. 2020. Developing an online hate classifier for multiple social media platforms. Human-centric Computing and Information Sciences 10 (2020), 1–34. - Sap et al. (2022) Maarten Sap, Swabha Swayamdipta, Laura Vianna, Xuhui Zhou, Yejin Choi, and Noah A Smith. 2022. Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 5884–5906. - Saurí and Pustejovsky (2009) Roser Saurí and James Pustejovsky. 2009. FactBank: a corpus annotated with event factuality. Language resources and evaluation 43 (2009), 227–268. - Savchev (2022) Aleksandar Savchev. 2022. AI Rational at CheckThat!-2022: Using transformer models for tweet classification. In Proceedings of the Working Notes of CLEF 2022 - Conference and Labs of the Evaluation Forum, Bologna, Italy, September 5th - to - 8th, 2022 (CEUR Workshop Proceedings, Vol. 3180), Guglielmo Faggioli, Nicola Ferro, Allan Hanbury, and Martin Potthast (Eds.). CEUR-WS.org, 656–659. https://ceur-ws.org/Vol-3180/paper-52.pdf - Savinova and Del Prado (2023) Elena Savinova and Fermin Moscoso Del Prado. 2023. Analyzing Subjectivity Using a Transformer-Based Regressor Trained on Naïve Speakers’ Judgements. In Proceedings of the 13th Workshop on Computational Approaches to Subjectivity, Sentiment, & Social Media Analysis. 305–314. - Sawinski et al. (2023) Marcin Sawinski, Krzysztof Wecel, Ewelina Ksiezniak, Milena Strózyna, Wlodzimierz Lewoniewski, Piotr Stolarski, and Witold Abramowicz. 2023. OpenFact at CheckThat!-2023: Head-to-Head GPT vs. BERT - A Comparative Study of Transformers Language Models for the Detection of Check-worthy Claims. In Working Notes of the Conference and Labs of the Evaluation Forum (CLEF 2023), Thessaloniki, Greece, September 18th to 21st, 2023 (CEUR Workshop Proceedings, Vol. 3497), Mohammad Aliannejadi, Guglielmo Faggioli, Nicola Ferro, and Michalis Vlachos (Eds.). CEUR-WS.org, 453–472. https://ceur-ws.org/Vol-3497/paper-040.pdf - Schlicht et al. (2023) Ipek Baris Schlicht, Lucie Flek, and Paolo Rosso. 2023. Multilingual Detection of Check-Worthy Claims Using World Languages and Adapter Fusion. In Advances in Information Retrieval: 45th European Conference on Information Retrieval, ECIR 2023, Dublin, Ireland, April 2–6, 2023, Proceedings, Part I (Dublin, Ireland). Springer-Verlag, Berlin, Heidelberg, 118–133. https://doi.org/10.1007/978-3-031-28244-7_8 - Schwarz and Morris (2011) Julia Schwarz and Meredith Morris. 2011. Augmenting web pages and search results to support credibility assessment. In Proceedings of the SIGCHI conference on human factors in computing systems. 1245–1254. - Self and Roberts (2019) Charles C. Self and Chris Roberts. 2019. Credibility. In An Integrated Approach to Communication Theory and Research (3 ed.). Routledge. - Shaar et al. (2022a) Shaden Shaar, Firoj Alam, Giovanni Da San Martino, and Preslav Nakov. 2022a. The Role of Context in Detecting Previously Fact-Checked Claims. In Findings of the Association for Computational Linguistics: NAACL 2022, Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz (Eds.). Association for Computational Linguistics, Seattle, United States, 1619–1631. https://doi.org/10.18653/v1/2022.findings-naacl.122 - Shaar et al. (2020) Shaden Shaar, Nikolay Babulkov, Giovanni Da San Martino, and Preslav Nakov. 2020. That is a Known Lie: Detecting Previously Fact-Checked Claims. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (Eds.). Association for Computational Linguistics, Online, 3607–3618. https://doi.org/10.18653/v1/2020.acl-main.332 - Shaar et al. (2022b) Shaden Shaar, Nikola Georgiev, Firoj Alam, Giovanni Da San Martino, Aisha Mohamed, and Preslav Nakov. 2022b. Assisting the Human Fact-Checkers: Detecting All Previously Fact-Checked Claims in a Document. In Findings of the Association for Computational Linguistics: EMNLP 2022, Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (Eds.). Association for Computational Linguistics, Abu Dhabi, United Arab Emirates, 2069–2080. https://doi.org/10.18653/v1/2022.findings-emnlp.151 - Shaar et al. (2021) Shaden Shaar, Fatima Haouari, Watheq Mansour, Maram Hasanain, Nikolay Babulkov, Firoj Alam, and Preslav Nakov. 2021. Overview of the CLEF-2021 CheckThat! Lab Task 2 on Detecting Previously Fact-Checked Claims in Tweets and Political Debates. In Proceedings of the Working Notes of CLEF 2021 - Conference and Labs of the Evaluation Forum, Vol. 2936. CEUR-WS, 13. http://ceur-ws.org/Vol-2936/paper-29.pdf - Sharma et al. (2019) Karishma Sharma, Feng Qian, He Jiang, Natali Ruchansky, Ming Zhang, Yan Liu, Feng Qian was, Feng Qian, He Jiang, Natali Ruchansky, Yan Liu, Ming Zhang, Laks V. S. Lakshmanan, Michael Simpson, Saravanan Thirumuruganathan, Karishma Sharma, Feng Qian, He Jiang, Natali Ruchansky, Ming Zhang, and Yan Liu. 2019. Combating Fake News: A Survey on Identification and Mitigation Techniques. ACM Transactions on Intelligent Systems and Technology 10, 3 (2019), 21. https://doi.org/10.1145/3305260 arXiv:1901.06437 - Sheikh Ali et al. (2023) Zien Sheikh Ali, Watheq Mansour, Fatima Haouari, Maram Hasanain, Tamer Elsayed, and Abdulaziz Al-Ali. 2023. Tahaqqaq: A Real-Time System for Assisting Twitter Users in Arabic Claim Verification. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval (Taipei, Taiwan) (SIGIR ’23). Association for Computing Machinery, New York, NY, USA, 3170–3174. https://doi.org/10.1145/3539618.3591815 - Sheikhi et al. (2023) Ghazaal Sheikhi, Samia Touileb, and Sohail Khan. 2023. Automated Claim Detection for Fact-checking: A Case Study using Norwegian Pre-trained Language Models. In Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa), Tanel Alumäe and Mark Fishel (Eds.). University of Tartu Library, Tórshavn, Faroe Islands, 1–9. https://aclanthology.org/2023.nodalida-1.1 - Sheng et al. (2021) Qiang Sheng, Juan Cao, Xueyao Zhang, Xirong Li, and Lei Zhong. 2021. Article Reranking by Memory-Enhanced Key Sentence Matching for Detecting Previously Fact-Checked Claims. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (Eds.). Association for Computational Linguistics, Online, 5468–5481. https://doi.org/10.18653/v1/2021.acl-long.425 - Shokri et al. (2024) Mohammad Shokri, Vivek Sharma, Elena Filatova, Shweta Jain, and Sarah Levitan. 2024. Subjectivity Detection in English News using Large Language Models. In Proceedings of the 14th Workshop on Computational Approaches to Subjectivity, Sentiment, & Social Media Analysis, Orphée De Clercq, Valentin Barriere, Jeremy Barnes, Roman Klinger, João Sedoc, and Shabnam Tafreshi (Eds.). Association for Computational Linguistics, Bangkok, Thailand, 215–226. https://doi.org/10.18653/v1/2024.wassa-1.17 - Shu et al. (2020) Kai Shu, Deepak Mahudeswaran, Suhang Wang, Dongwon Lee, and Huan Liu. 2020. FakeNewsNet: A Data Repository with News Content, Social Context, and Spatiotemporal Information for Studying Fake News on Social Media. Big Data 8, 3 (2020), 171–188. https://doi.org/10.1089/big.2020.0062 arXiv:https://doi.org/10.1089/big.2020.0062 PMID: 32491943. - Shu et al. (2017) Kai Shu, Amy Sliva, Suhang Wang, Jiliang Tang, and Huan Liu. 2017. Fake News Detection on Social Media: A Data Mining Perspective. ACM SIGKDD Explorations Newsletter 19, 1 (Sept. 2017), 22–36. https://doi.org/10.1145/3137597.3137600 arXiv:1708.01967 - Shusas and Forte (2024) Erica Shusas and Andrea Forte. 2024. Trust and Transparency: An Exploratory Study on Emerging Adults’ Interpretations of Credibility Indicators on Social Media Platforms. In Extended Abstracts of the 2024 CHI Conference on Human Factors in Computing Systems (CHI EA ’24). Association for Computing Machinery, New York, NY, USA, Article 343, 7 pages. https://doi.org/10.1145/3613905.3650801 - Simchon et al. (2024) Almog Simchon, Matthew Edwards, and Stephan Lewandowsky. 2024. The persuasive effects of political microtargeting in the age of generative artificial intelligence. PNAS Nexus 3, 2 (01 2024), pgae035. https://doi.org/10.1093/pnasnexus/pgae035 arXiv:https://academic.oup.com/pnasnexus/article-pdf/3/2/pgae035/56683743/pgae035.pdf - Simon et al. (2023) Felix M Simon, Sacha Altay, and Hugo Mercier. 2023. Misinformation reloaded? Fears about the impact of generative AI on misinformation are overblown. Harvard Kennedy School Misinformation Review 4, 5 (2023). - Singh et al. (2024) Iknoor Singh, Carolina Scarton, Xingyi Song, and Kalina Bontcheva. 2024. Breaking Language Barriers with MMTweets: Advancing Cross-Lingual Debunked Narrative Retrieval for Fact-Checking. arXiv:2308.05680 [cs.CL] https://arxiv.org/abs/2308.05680 - Sitaula et al. (2020) Niraj Sitaula, Chilukuri K. Mohan, Jennifer Grygiel, Xinyi Zhou, and Reza Zafarani. 2020. Credibility-Based Fake News Detection. Springer International Publishing, Cham, 163–182. https://doi.org/10.1007/978-3-030-42699-6_9 - Smith et al. (2001) Jackie Smith, John D. McCarthy, Clark Mcphail, and Boguslaw Augustyn. 2001. From Protest to Agenda Building: Description Bias in Media Coverage of Protest Events in Washington, D.C. Social Forces 79 (2001), 1397 – 1423. https://api.semanticscholar.org/CorpusID:145456088 - Solaiman et al. (2019) Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert-Voss, Jeff Wu, Alec Radford, Gretchen Krueger, Jong Wook Kim, Sarah Kreps, Miles McCain, Alex Newhouse, Jason Blazakis, Kris McGuffie, and Jasmine Wang. 2019. Release Strategies and the Social Impacts of Language Models. arXiv:1908.09203 [cs.CL] https://arxiv.org/abs/1908.09203 - Solanki et al. (2021) Aastik Solanki, Adesh Kumar Jaiswal, Aditi Mishra, Anant Prakash Mishra, and Suruchi Sabherwal. 2021. Machine Learning-Based Identification of Collusive Users in Twitter Stream. In 2021 International Conference on Computational Performance Evaluation (ComPE). 797–802. https://doi.org/10.1109/ComPE53109.2021.9752090 - Soroka (2012) Stuart N Soroka. 2012. The gatekeeping function: Distributions of information in media and the real world. The Journal of Politics 74, 2 (2012), 514–528. - Spezzano et al. (2021) Francesca Spezzano, Anu Shrestha, Jerry Alan Fails, and Brian W. Stone. 2021. That’s Fake News! Reliability of News When Provided Title, Image, Source Bias & Full Article. Proc. ACM Hum.-Comput. Interact. 5, CSCW1, Article 109 (apr 2021), 19 pages. https://doi.org/10.1145/3449183 - Spiegel and Macko (2024) Michal Spiegel and Dominik Macko. 2024. KInIT at SemEval-2024 Task 8: Fine-tuned LLMs for Multilingual Machine-Generated Text Detection. In Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024), Atul Kr. Ojha, A. Seza Doğruöz, Harish Tayyar Madabushi, Giovanni Da San Martino, Sara Rosenthal, and Aiala Rosá (Eds.). Association for Computational Linguistics, Mexico City, Mexico, 558–564. https://doi.org/10.18653/v1/2024.semeval-1.84 - Spinde et al. (2021a) Timo Spinde, Christina Kreuter, Wolfgang Gaissmaier, Felix Hamborg, Bela Gipp, and Helge Giese. 2021a. Do You Think It’s Biased? How To Ask For The Perception Of Media Bias. In 2021 ACM/IEEE Joint Conference on Digital Libraries (JCDL). 61–69. https://doi.org/10.1109/JCDL52503.2021.00018 - Spinde et al. (2021b) Timo Spinde, Manuel Plank, Jan-David Krieger, Terry Ruas, Bela Gipp, and Akiko Aizawa. 2021b. Neural Media Bias Detection Using Distant Supervision With BABE - Bias Annotations By Experts. In Findings of the Association for Computational Linguistics: EMNLP 2021, Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (Eds.). Association for Computational Linguistics, Punta Cana, Dominican Republic, 1166–1177. https://doi.org/10.18653/v1/2021.findings-emnlp.101 - Spinde et al. (2023) Timo Spinde, Elisabeth Richter, Martin Wessel, Juhi Kulshrestha, and Karsten Donnay. 2023. What do Twitter comments tell about news article bias? Assessing the impact of news article bias on its perception on Twitter. Online Social Networks and Media 37-38 (2023), 100264. https://doi.org/10.1016/j.osnem.2023.100264 - Spinde et al. (2021c) Timo Spinde, Lada Rudnitckaia, Felix Hamborg, and Bela Gipp. 2021c. Identification of Biased Terms in News Articles by Comparison of Outlet-Specific Word Embeddings. In Diversity, Divergence, Dialogue, Katharina Toeppe, Hui Yan, and Samuel Kai Wah Chu (Eds.). Springer International Publishing, Cham, 215–224. - Spinde et al. (2021d) Timo Spinde, Lada Rudnitckaia, Jelena Mitrović, Felix Hamborg, Michael Granitzer, Bela Gipp, and Karsten Donnay. 2021d. Automated identification of bias inducing words in news articles using linguistic and context-oriented features. Information Processing & Management 58, 3 (2021), 102505. https://doi.org/10.1016/j.ipm.2021.102505 - Sundriyal et al. (2023) Megha Sundriyal, Tanmoy Chakraborty, and Preslav Nakov. 2023. From Chaos to Clarity: Claim Normalization to Empower Fact-Checking. In Findings of the Association for Computational Linguistics: EMNLP 2023, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, Singapore, 6594–6609. https://doi.org/10.18653/v1/2023.findings-emnlp.439 - Suwaileh et al. (2025) Reem Suwaileh, Maram Hasanain, Fatema Hubail, Wajdi Zaghouani, and Firoj Alam. 2025. ThatiAR: subjectivity detection in Arabic news sentences. In Proceedings of the International AAAI Conference on Web and Social Media, Vol. 19. 2587–2602. - Takkouche and Norman (2011) Bahi Takkouche and Guy Norman. 2011. PRISMA statement. Epidemiology 22, 1 (2011), 128. - Tchechmedjiev et al. (2019) Andon Tchechmedjiev, Pavlos Fafalios, Katarina Boland, Malo Gasquet, Matthäus Zloch, Benjamin Zapilko, Stefan Dietze, and Konstantin Todorov. 2019. ClaimsKG: A Knowledge Graph of Fact-Checked Claims. In The Semantic Web – ISWC 2019, Chiara Ghidini, Olaf Hartig, Maria Maleshkova, Vojtěch Svátek, Isabel Cruz, Aidan Hogan, Jie Song, Maxime Lefrançois, and Fabien Gandon (Eds.). Springer International Publishing, Cham, 309–324. - Tian et al. (2021) Junfeng Tian, Min Gui, Chenliang Li, Ming Yan, and Wenming Xiao. 2021. MinD at SemEval-2021 Task 6: Propaganda Detection using Transfer Learning and Multimodal Fusion. In Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), Alexis Palmer, Nathan Schneider, Natalie Schluter, Guy Emerson, Aurelie Herbelot, and Xiaodan Zhu (Eds.). Association for Computational Linguistics, Online, 1082–1087. https://doi.org/10.18653/v1/2021.semeval-1.150 - Tran (2020) Minh Tran. 2020. How biased are American media outlets? A framework for presentation bias regression. In 2020 IEEE International Conference on Big Data (Big Data). 4359–4364. https://doi.org/10.1109/BigData50022.2020.9377987 - Tufchi et al. (2023) Shivani Tufchi, Ashima Yadav, and Tanveer Ahmed. 2023. A Comprehensive Survey of Multimodal Fake News Detection Techniques: Advances, Challenges, and Opportunities. International Journal of Multimedia Information Retrieval 12, 2 (Aug. 2023), 28. https://doi.org/10.1007/s13735-023-00296-3 - Uchendu et al. (2023) Adaku Uchendu, Thai Le, and Dongwon Lee. 2023. Attribution and Obfuscation of Neural Text Authorship: A Data Mining Perspective. SIGKDD Explor. Newsl. 25, 1 (July 2023), 1–18. https://doi.org/10.1145/3606274.3606276 - Uchendu et al. (2021) Adaku Uchendu, Zeyu Ma, Thai Le, Rui Zhang, and Dongwon Lee. 2021. TuringBench: A Benchmark Environment for Turing Test in the Age of Neural Text Generation. In Findings of the Association for Computational Linguistics: EMNLP 2021. Association for Computational Linguistics, Punta Cana, Dominican Republic, 2001–2016. https://doi.org/10.18653/v1/2021.findings-emnlp.172 - Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Advances in Neural Information Processing Systems, I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.), Vol. 30. Curran Associates, Inc. https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf - Vo and Lee (2018) Nguyen Vo and Kyumin Lee. 2018. The Rise of Guardians: Fact-checking URL Recommendation to Combat Fake News. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval (Ann Arbor, MI, USA) (SIGIR ’18). Association for Computing Machinery, New York, NY, USA, 275–284. https://doi.org/10.1145/3209978.3210037 - Vo and Lee (2020) Nguyen Vo and Kyumin Lee. 2020. Where Are the Facts? Searching for Fact-checked Information to Alleviate the Spread of Fake News. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (Eds.). Association for Computational Linguistics, Online, 7717–7731. https://doi.org/10.18653/v1/2020.emnlp-main.621 - Voronov et al. (2024) Anton Voronov, Lena Wolf, and Max Ryabinin. 2024. Mind Your Format: Towards Consistent Evaluation of In-Context Learning Improvements. In Findings of the Association for Computational Linguistics ACL 2024, Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association for Computational Linguistics, Bangkok, Thailand and virtual meeting, 6287–6310. https://doi.org/10.18653/v1/2024.findings-acl.375 - Vykopal et al. (2025a) Ivan Vykopal, Martin Hyben, Robert Moro, Michal Gregor, and Jakub Simko. 2025a. A Generative-AI-Driven Claim Retrieval System Capable of Detecting and Retrieving Claims from Social Media Platforms in Multiple Languages. arXiv:2504.20668 [cs.CL] https://arxiv.org/abs/2504.20668 - Vykopal et al. (2025b) Ivan Vykopal, Matúš Pikuliak, Simon Ostermann, Tatiana Anikina, Michal Gregor, and Marián Šimko. 2025b. Large Language Models for Multilingual Previously Fact-Checked Claim Detection. arXiv:2503.02737 [cs.CL] https://arxiv.org/abs/2503.02737 - Vykopal et al. (2024) Ivan Vykopal, Matúš Pikuliak, Ivan Srba, Robert Moro, Dominik Macko, and Maria Bielikova. 2024. Disinformation Capabilities of Large Language Models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association for Computational Linguistics, Bangkok, Thailand, 14830–14847. https://doi.org/10.18653/v1/2024.acl-long.793 - Waldman and Devitt (1998) Paul Waldman and James Devitt. 1998. Newspaper photographs and the 1996 presidential election: The question of bias. Journalism & Mass Communication Quarterly 75, 2 (1998), 302–311. - Wang et al. (2023a) Ace Wang, Liz Maylin De Jesus Sanchez, Anya Wintner, Yuanxin Zhu, and Eni Mustafaraj. 2023a. Assessing Google Search’s New Features in Supporting Credibility judgments of Unknown Websites. In Proceedings of the 2023 Conference on Human Information Interaction and Retrieval (CHIIR ’23). Association for Computing Machinery, New York, NY, USA, 303–307. https://doi.org/10.1145/3576840.3578277 - Wang et al. (2023b) Cunxiang Wang, Xiaoze Liu, Yuanhao Yue, Xiangru Tang, Tianhang Zhang, Cheng Jiayang, Yunzhi Yao, Wenyang Gao, Xuming Hu, Zehan Qi, et al. 2023b. Survey on factuality in large language models: Knowledge, retrieval and domain-specificity. arXiv preprint arXiv:2310.07521 (2023). - Wang and Liu (2011) Dong Wang and Yang Liu. 2011. A cross-corpus study of unsupervised subjectivity identification based on calibrated em. In Proceedings of the 2nd Workshop on Computational Approaches to Subjectivity and Sentiment Analysis (WASSA 2.011). 161–167. - Wang (2017) William Yang Wang. 2017. “Liar, Liar Pants on Fire”: A New Benchmark Dataset for Fake News Detection. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). 422–426. - Wang et al. (2024a) Yuxia Wang, Jonibek Mansurov, Petar Ivanov, Jinyan Su, Artem Shelmanov, Akim Tsvigun, Osama Mohammed Afzal, Tarek Mahmoud, Giovanni Puccetti, and Thomas Arnold. 2024a. SemEval-2024 Task 8: Multidomain, Multimodel and Multilingual Machine-Generated Text Detection. In Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024), Atul Kr. Ojha, A. Seza Doğruöz, Harish Tayyar Madabushi, Giovanni Da San Martino, Sara Rosenthal, and Aiala Rosá (Eds.). Association for Computational Linguistics, Mexico City, Mexico, 2057–2079. https://doi.org/10.18653/v1/2024.semeval-1.279 - Wang et al. (2024b) Yuxia Wang, Jonibek Mansurov, Petar Ivanov, Jinyan Su, Artem Shelmanov, Akim Tsvigun, Osama Mohammed Afzal, Tarek Mahmoud, Giovanni Puccetti, Thomas Arnold, Alham Aji, Nizar Habash, Iryna Gurevych, and Preslav Nakov. 2024b. M4GT-Bench: Evaluation Benchmark for Black-Box Machine-Generated Text Detection. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association for Computational Linguistics, Bangkok, Thailand, 3964–3992. https://doi.org/10.18653/v1/2024.acl-long.218 - Wang et al. (2024c) Yuxia Wang, Minghan Wang, Hasan Iqbal, Georgi Georgiev, Jiahui Geng, and Preslav Nakov. 2024c. Openfactcheck: A unified framework for factuality evaluation of llms. arXiv preprint arXiv:2405.05583 (2024). - Waseem and Hovy (2016) Zeerak Waseem and Dirk Hovy. 2016. Hateful symbols or hateful people? predictive features for hate speech detection on twitter. In Proceedings of the NAACL student research workshop. 88–93. - Weerkamp and De Rijke (2008) Wouter Weerkamp and Maarten De Rijke. 2008. Credibility improves topical blog post retrieval. In Proceedings of ACL-08: HLT. 923–931. - Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Vol. 35. Curran Associates, Inc., 24824–24837. https://proceedings.neurips.cc/paper_files/paper/2022/file/9d5609613524ecf4f15af0f7b31abca4-Paper-Conference.pdf - Wessel and Horych (2024) Martin Wessel and Tomáš Horych. 2024. Beyond the Surface: Spurious Cues in Automatic Media Bias Detection. In Proceedings of the Fourth Workshop on Language Technology for Equality, Diversity, Inclusion. 21–30. - Wessel et al. (2023) Martin Wessel, Tomás Horych, Terry Ruas, Akiko Aizawa, Bela Gipp, and Timo Spinde. 2023. Introducing MBIB - The First Media Bias Identification Benchmark Task and Dataset Collection. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval (Taipei, Taiwan) (SIGIR ’23). Association for Computing Machinery, New York, NY, USA, 2765–2774. https://doi.org/10.1145/3539618.3591882 - Wiebe and Riloff (2005) Janyce Wiebe and Ellen Riloff. 2005. Creating subjective and objective sentence classifiers from unannotated texts. In International conference on intelligent text processing and computational linguistics. Springer, 486–497. - Wiebe et al. (2004) Janyce Wiebe, Theresa Wilson, Rebecca Bruce, Matthew Bell, and Melanie Martin. 2004. Learning subjective language. Computational linguistics 30, 3 (2004), 277–308. - Wiebe et al. (2005) Janyce Wiebe, Theresa Wilson, and Claire Cardie. 2005. Annotating expressions of opinions and emotions in language. Language resources and evaluation 39 (2005), 165–210. - Wiegand and Klakow (2011) Michael Wiegand and Dietrich Klakow. 2011. Convolution kernels for subjectivity detection. In Proceedings of the 18th Nordic Conference of Computational Linguistics (NODALIDA 2011). 254–261. - Wiegand et al. (2019) Michael Wiegand, Josef Ruppenhofer, and Thomas Kleinbauer. 2019. Detection of abusive language: the problem of biased datasets. In Proceedings of the 2019 conference of the North American Chapter of the Association for Computational Linguistics: human language technologies, volume 1 (long and short papers). 602–608. - Wiegand et al. (2018) Michael Wiegand, Melanie Siegel, and Josef Ruppenhofer. 2018. Overview of the GermEval 2018 Shared Task on the Identification of Offensive Language. In Proceedings of GermEval 2018, 14th Conference on Natural Language Processing (KONVENS 2018). - Williams et al. (2021) Evan M. Williams, Paul Rodrigues, and Sieu Tran. 2021. Accenture at CheckThat! 2021: Interesting claim identification and ranking with contextually sensitive lexical training data augmentation. In Proceedings of the Working Notes of CLEF 2021 - Conference and Labs of the Evaluation Forum, Bucharest, Romania, September 21st - to - 24th, 2021 (CEUR Workshop Proceedings, Vol. 2936), Guglielmo Faggioli, Nicola Ferro, Alexis Joly, Maria Maistro, and Florina Piroi (Eds.). CEUR-WS.org, 659–669. https://ceur-ws.org/Vol-2936/paper-55.pdf - Winarto et al. (2023) Tora Sangputra Yopie Winarto, Kevin Wijaya, Muhammad Abdullah Faqih, Simeon Yuda Prasetyo, and Yohan Muliono. 2023. Tackling Clickbait with Machine Learning: A Comparative Study of Binary Classification Models for YouTube Title. Procedia Computer Science 227 (2023), 282–290. https://doi.org/10.1016/j.procs.2023.10.526 8th International Conference on Computer Science and Computational Intelligence (ICCSCI 2023). - Wu et al. (2023) Ben Wu, Olesya Razuvayevskaya, Freddy Heppell, João A. Leite, Carolina Scarton, Kalina Bontcheva, and Xingyi Song. 2023. SheffieldVeraAI at SemEval-2023 Task 3: Mono and Multilingual Approaches for News Genre, Topic and Persuasion Technique Classification. In Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023), Atul Kr. Ojha, A. Seza Doğruöz, Giovanni Da San Martino, Harish Tayyar Madabushi, Ritesh Kumar, and Elisa Sartori (Eds.). Association for Computational Linguistics, Toronto, Canada, 1995–2008. https://doi.org/10.18653/v1/2023.semeval-1.275 - Wu et al. (2025) Junchao Wu, Shu Yang, Runzhe Zhan, Yulin Yuan, Lidia Sam Chao, and Derek Fai Wong. 2025. A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions. Computational Linguistics 51, 1 (March 2025), 275–338. https://doi.org/10.1162/coli_a_00549 - Wu et al. (2024) Junchao Wu, Shu Yang, Runzhe Zhan, Yulin Yuan, Derek F. Wong, and Lidia S. Chao. 2024. A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions. arXiv:2310.14724 [cs.CL] https://arxiv.org/abs/2310.14724 - Wu et al. (2020) Lianwei Wu, Yuan Rao, Ambreen Nazir, and Haolin Jin. 2020. Discovering differential features: Adversarial learning for information credibility evaluation. Inf. Sci. 516, C (apr 2020), 453–473. https://doi.org/10.1016/j.ins.2019.12.040 - Wuehrl et al. (2023) Amelie Wuehrl, Lara Grimminger, and Roman Klinger. 2023. An Entity-based Claim Extraction Pipeline for Real-world Biomedical Fact-checking. In Proceedings of the Sixth Fact Extraction and VERification Workshop (FEVER), Mubashara Akhtar, Rami Aly, Christos Christodoulopoulos, Oana Cocarascu, Zhijiang Guo, Arpit Mittal, Michael Schlichtkrull, James Thorne, and Andreas Vlachos (Eds.). Association for Computational Linguistics, Dubrovnik, Croatia, 29–37. https://doi.org/10.18653/v1/2023.fever-1.3 - Xu et al. (2023) Lingling Xu, Haoran Xie, Si-Zhao Joe Qin, Xiaohui Tao, and Fu Lee Wang. 2023. Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment. arXiv:2312.12148 [cs.CL] https://arxiv.org/abs/2312.12148 - Xu et al. (2024) Rongwu Xu, Brian S. Lin, Shujian Yang, Tianqi Zhang, Weiyan Shi, Tianwei Zhang, Zhixuan Fang, Wei Xu, and Han Qiu. 2024. The Earth is Flat because…: Investigating LLMs’ Belief towards Misinformation via Persuasive Conversation. https://doi.org/10.48550/arXiv.2312.09085 arXiv:2312.09085. - Yan et al. (2019) Hao Yan, Sanmay Das, Allen Lavoie, Sirui Li, and Betsy Sinclair. 2019. The Congressional Classification Challenge: Domain Specificity and Partisan Intensity. In Proceedings of the 2019 ACM Conference on Economics and Computation (Phoenix, AZ, USA) (EC ’19). Association for Computing Machinery, New York, NY, USA, 71–89. https://doi.org/10.1145/3328526.3329582 - Yang et al. (2020) Fumeng Yang, Zhuanyi Huang, Jean Scholtz, and Dustin L. Arendt. 2020. How do visual explanations foster end users’ appropriate trust in machine learning?. In Proceedings of the 25th International Conference on Intelligent User Interfaces (Cagliari, Italy) (IUI ’20). Association for Computing Machinery, New York, NY, USA, 189–201. https://doi.org/10.1145/3377325.3377480 - Ye and Skiena (2019) Junting Ye and Steven Skiena. 2019. MediaRank: Computational Ranking of Online News Sources. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (Anchorage, AK, USA) (KDD ’19). Association for Computing Machinery, New York, NY, USA, 2469–2477. https://doi.org/10.1145/3292500.3330709 - Yi et al. (2022) Qingqing Yi, Zhong Qian, Peifeng Li, and Qiaoming Zhu. 2022. Chinese Sentence-level Event Factuality Identification with Recursive Neural Network. In 2022 International Joint Conference on Neural Networks (IJCNN). 1–8. https://doi.org/10.1109/IJCNN55064.2022.9892209 - Yoosuf and Yang (2019) Shehel Yoosuf and Yin Yang. 2019. Fine-Grained Propaganda Detection with Fine-Tuned BERT. In Proceedings of the Second Workshop on Natural Language Processing for Internet Freedom: Censorship, Disinformation, and Propaganda, Anna Feldman, Giovanni Da San Martino, Alberto Barrón-Cedeño, Chris Brew, Chris Leberknight, and Preslav Nakov (Eds.). Association for Computational Linguistics, Hong Kong, China, 87–91. https://doi.org/10.18653/v1/D19-5011 - Zade et al. (2023) Himanshu Zade, Megan Woodruff, Erika Johnson, Mariah Stanley, Zhennan Zhou, Minh Tu Huynh, Alissa Elizabeth Acheson, Gary Hsieh, and Kate Starbird. 2023. Tweet Trajectory and AMPS-based Contextual Cues can Help Users Identify Misinformation. Proc. ACM Hum.-Comput. Interact. 7, CSCW1, Article 103 (apr 2023), 27 pages. https://doi.org/10.1145/3579536 - Zampieri et al. (2019) Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019. SemEval-2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval). In Proceedings of the 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics, Minneapolis, Minnesota, USA, 75–86. https://doi.org/10.18653/v1/S19-2010 - Zampieri et al. (2020) Marcos Zampieri, Preslav Nakov, Sara Rosenthal, Pepa Atanasova, Georgi Karadzhov, Hamdy Mubarak, Leon Derczynski, Zeses Pitenis, and Çağrı Çöltekin. 2020. SemEval-2020 Task 12: Multilingual Offensive Language Identification in Social Media (OffensEval 2020). In Proceedings of SemEval. - Zellers et al. (2019) Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019. Defending Against Neural Fake News. In Advances in Neural Information Processing Systems, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (Eds.), Vol. 32. Curran Associates, Inc. https://proceedings.neurips.cc/paper_files/paper/2019/file/3e9f0fc9b2f89e043bc6233994dfcf76-Paper.pdf - Zhang et al. (2018) Amy X. Zhang, Aditya Ranganathan, Sarah Emlen Metz, Scott Appling, Connie Moon Sehat, Norman Gilmore, Nick B. Adams, Emmanuel Vincent, Jennifer Lee, Martin Robbins, Ed Bice, Sandro Hawke, David Karger, and An Xiao Mina. 2018. A Structured Response to Misinformation: Defining and Annotating Credibility Indicators in News Articles. In Companion Proceedings of the The Web Conference 2018 (Lyon, France) (WWW ’18). International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, CHE, 603–612. https://doi.org/10.1145/3184558.3188731 - Zhang and Clough (2020) Canyu Zhang and Paul D. Clough. 2020. Investigating clickbait in Chinese social media: A study of WeChat. Online Social Networks and Media 19 (2020), 100095. https://doi.org/10.1016/j.osnem.2020.100095 - Zhang et al. (2023) Heng Zhang, Peifeng Li, Zhong Qian, and Xiaoxu Zhu. 2023. Incorporating factuality inference to identify document-level event factuality. In Findings of the Association for Computational Linguistics: ACL 2023. 13990–14002. - Zhang et al. (2019) Yifan Zhang, Giovanni Da San Martino, Alberto Barrón-Cedeño, Salvatore Romeo, Jisun An, Haewoon Kwak, Todor Staykovski, Israa Jaradat, Georgi Karadzhov, Ramy Baly, Kareem Darwish, James Glass, and Preslav Nakov. 2019. Tanbih: Get To Know What You Are Reading. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations, Sebastian Padó and Ruihong Huang (Eds.). Association for Computational Linguistics, Hong Kong, China, 223–228. https://doi.org/10.18653/v1/D19-3038 - Zhao et al. (2016) Liang Zhao, Ting Hua, Chang-Tien Lu, and Ing-Ray Chen. 2016. A topic-focused trust model for Twitter. Computer Communications 76 (2016), 1–11. https://doi.org/10.1016/j.comcom.2015.08.001 - Zhou and Zafarani (2020) Xinyi Zhou and Reza Zafarani. 2020. A Survey of Fake News: Fundamental Theories, Detection Methods, and Opportunities. Comput. Surveys 53, 5 (Oct. 2020), 1–40. https://doi.org/10.1145/3395046 arXiv:1812.00315 - Zugecova et al. (2025) Aneta Zugecova, Dominik Macko, Ivan Srba, Robert Moro, Jakub Kopál, Katarína Marcinčinová, and Matúš Mesarčík. 2025. Evaluation of LLM Vulnerabilities to Being Misused for Personalized Disinformation Generation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (Eds.). Association for Computational Linguistics, Vienna, Austria, 780–797. https://doi.org/10.18653/v1/2025.acl-long.38 - İŞ and TUNCER (2018) Hafzullah İŞ and Taner TUNCER. 2018. Confidence Index Analysis of Twitter Users Timeline. In 2018 International Conference on Artificial Intelligence and Data Processing (IDAP). 1–8. https://doi.org/10.1109/IDAP.2018.8620917