2308.12215

# SoK: Machine Learning for Misinformation Detection **Authors**: - Madelyne Xiao (Princeton University) - Jonathan Mayer (Princeton University) \addbibresource references.bib \floatsetup [table]capposition=top Abstract We examine the disconnect between scholarship and practice in applying machine learning to trust and safety problems, using misinformation detection as a case study. We survey literature on automated detection of misinformation across a corpus of 248 well-cited papers in the field. We then examine subsets of papers for data and code availability, design missteps, reproducibility, and generalizability. Our paper corpus includes published work in security, natural language processing, and computational social science. Across these disparate disciplines, we identify common errors in dataset and method design. In general, detection tasks are often meaningfully distinct from the challenges that online services actually face. Datasets and model evaluation are often non-representative of real-world contexts, and evaluation frequently is not independent of model training. We demonstrate the limitations of current detection methods in a series of three representative replication studies. Based on the results of these analyses and our literature survey, we conclude that the current state-of-the-art in fully-automated misinformation detection has limited efficacy in detecting human-generated misinformation. We offer recommendations for evaluating applications of machine learning to trust and safety problems and recommend future directions for research. 1 INTRODUCTION Online services face a daunting task: There is an unceasing deluge of user-generated content, on the order of hundreds of thousands of posts per minute on popular social media platforms [dewan_2015]. Some of that content is false, hateful, harassing, extremist, or otherwise problematic. How can platforms reliably and proactively identify these “trust and safety” issues? Machine learning has proved an attractive approach in the academic literature, leading to large bodies of scholarship on misinformation detection [alam_survey_2021], toxic speech classification [andročec_2021], and other core trust and safety challenges (e.g., [lee_2020]). The conceptual appeal of machine learning is that it could address the massive scale of user-generated content on large platforms and the capacity constraints of small platforms. Recent work claims impressive performance statistics: In the literature review that we conduct for this work, among publications that report performance metrics, about 70% of papers report over 80% accuracy on at least one detection task; some of these works report near-perfect performance [baly_predicting_nodate, dhamani_using_2019]. In recent years, news items from major tech companies have tempered these expectations. In October of 2023, in a Bluesky post commenting on Twitter’s user-driven Community Notes program, Twitter’s former head of trust and safety stated that large-scale automated detection of misinformation remains a hard problem, and that no generalizable automated solutions are currently available [roth_2023]. In January of 2025, Meta announced that it would terminate its third-party fact-checking program in favor of a Community Notes-like system of user-driven content moderation [chan_meta_2025]. These disclosures accord with our observation that, in practice, trust and safety functions at online services remain heavily manual: driven by user reports and carried out by human moderators. In this work, we investigate the disconnect between scholarship and practice in applications of machine learning to trust and safety problems. Our project is inspired by recent research that has identified shortcomings in machine learning applications for many problem domains, including information security [arp_ml_2022, Jacobs2022]. We use misinformation detection as a case study for trust and safety problems because the topic has recently generated a rich literature with diverse methods and claimed successes. Misinformation detection has substantive complexities that are common for trust and safety problems: linguistic and cultural nuance, sensitivity to context, and rapidly evolving circumstances. We seek to answer four discrete research questions, which collectively shed light on the research-practice gap in automated misinformation detection. - How well-suited are misinformation detection methods in the academic literature to the needs of online services, specifically social media platforms that host user-generated content? - Are there identifiable missteps related to target selection, dataset curation, feature selection, and evaluations of method performance in ML-driven misinformation detection studies? - How reproducible are published ML-driven misinformation detection methods? - How generalizable are published ML-driven misinformation detection methods to out-of-domain data (i.e., to data types and topics not present in training data)? We address these research questions in three ways. First, we conduct a broad literature review and synthesis of the full paper corpus (248 papers), with a focus on detection targets and evaluation. We provide an in-depth review of a subset (87 papers) of the full paper set, with a focus on methods: dataset design, feature engineering, and model selection. Second, we attempt to obtain code and data for a subset of prior work. Third, we test several representative approaches for replication and generalizability. We arrive at the following results by applying these methods. 1. Detection tasks in scholarship are often steps removed from the misinformation content moderation challenges that platforms face. Detection targets may be of limited consequence, or may be more readily and accurately identified through manual means. 1. Methods frequently target proxies for the presence of misleading content; these approaches are easy to evade. Datasets used in publications are often non-representative of real-world contexts. Model evaluation often lacks independence from training and rarely involves close emulation of a real-world deployment. 1. Data and code availability problems pervade the literature, inhibiting replication. Where these are available, we are generally able to replicate prior work. 1. Prior work has poor generalizability when classifying content beyond what was included in training data. Through this work, we hope to underscore 1) the prevalence and severity of reproducibility and ML-driven method design issues in the existing misinformation detection literature, 2) the need for careful and preemptive evaluation of ML-driven methods at the point of problem formulation, and 3) the importance of method explainability and data accessibility. Based on these observations, we provide recommendations for future work that proposes ML models to address trust and safety concerns. We contribute information taxonomies, our annotated corpus of 248 research papers, recent datasets, and frameworks for evaluating ML-driven content moderation tasks. 2 MOTIVATION The detection of misinformation and “influence operations” (IOs) is a topic of convergent interest for security, social science, and AI/ML researchers. Methods developed in recent years for the detection of influence operations (e.g., astroturfing, coordinated misinformation campaigns) resemble techniques previously employed by security researchers for the detection of botnets, advanced persistent threats (APTs), and malware [milajerdi2019holmes, king2006subvirt, stone2009botnet]. As such, we believe that the security community is uniquely well-positioned to shepherd the responsible development of misinformation detection methods. In addition, misinformation research stands to benefit a good deal from the formal structure of security research methodology: In a survey study from 2022 conducted by Mirza et al., human fact-checkers and journalists expressed interest in adopting rigorous threat modeling practices similar to those found in academic security literature [mirza2023tactics]. 3 PRELIMINARIES Definitions. In the absence of agreed-upon definitions of misinformation, disinformation, malinformation, and influence operations, we refrain from advancing singular definitions of the same While certain taxonomies [alam_survey_2021, zhou_survey_2020, oshikawa_survey_2018] provide relative definitions of these terms—distinguishing misinformation from disinformation, for instance, via the presence or absence of intent—we emphasize that, to the best of our knowledge, there is no robust invariant for identifying a statement as (mis)informative on the basis of semantics alone.. Instead, for each paper we review, we consider the paper authors’ working definitions of these terms (if such definitions are provided) and evaluate model performance with respect to the definitions set forth [wu_misinformation_2019, gelfert_fake_2018, haiden_definitional_2018, klein_fake_2017, tandoc_defining_2018]. We limit our analysis to text-based English-language misinformation. We acknowledge that the nature of misinformation narratives and misinformation spread varies with language and geography. We note that disinformation is commonly used to refer to incorrect information written with the intent to deceive, while misinformation refers to incorrect information in general; for the sake of completeness and concision, we refer to intentionally and unintentionally false information as misinformation for this work unless otherwise qualified. We occasionally use false rumors, false news, and false information interchangeably, but avoid fake news, a politically charged term [habgood_coote_2018]. We use “influence operations” (IOs) or “coordinated campaigns” in our discussion of collaborative attempts to disseminate misinformation across networks [jackson_2023]. Feasibility of detection. Most of the work we review presupposes that automated detection of misinformation is possible at all. We note that this is, in itself, a strong assumption to make: Published work in psychology, linguistics, and philosophy of language has previously called into question the feasibility of using semantic and syntactic features to determine the veracity of text statements and the usefulness of binary true/false classifications of information, particularly in the absence of unified definitions of misinformation, disinformation, and information [soe_2017, vrij_pitfalls_2008]. On the other hand, some researchers maintain that misinformative texts are characterized by indelible “fingerprints” that distinguish them from non-misinformative texts: for instance, emotional language and reduced lexical diversity [amado_undeutsch_2015, carrasco_2022]. This debate motivated our discussion of non-textual feature sets for detection. Taxonomies. Through an iterative process of reading and inductive coding of our papers, we develop a taxonomy of five “information scopes.” As we read each paper in our corpus, we noted the operative unit of misinformation detection for the classifier or method described—the smallest semantic or organizational unit of information that the method attempted to classify as true or false—and sorted each paper into emergent categories. Our final taxonomy comprises five information scopes: claims, news articles, social media accounts, networks, and websites. We consider these scopes in the context of online services—for instance, news articles and websites whose links might be posted to a social media platform. We label certain works as members of multiple scopes when they make classification decisions about more than one type of detection unit: works targeting social media posts, for instance, are generally categorized as falling within ( C and U) or ( C and N) in Table 4. - Claims. The smallest semantic unit of fact or misinformation, comprising a subject, predicate, and object, at minimum. - Articles. News-oriented writing of length 100 words or more. - Users. All data and metadata associated with a single user’s account as defined by a social media platform (e.g., an account corresponding to a handle on Twitter/X; or a page or profile corresponding to one business, individual, or organization on Facebook). - Networks. A set of users and interactions between these users as represented by a social graph. - Websites. A news site, including its hosting infrastructure and text and image contents. We describe errors in method design with respect to the step of the development pipeline in which they occur. We break this process down as follows: target selection, dataset curation, model choice, feature set selection and model evaluation. Development step definitions follow: - Target selection. The stated versus actual objective(s) of the classification task that paper authors describe: for instance, detection of emotional valence of a text (angry, sad, happy); verification of semantic accuracy (true, false); or characterizing degree of virality (e.g., as measured by number of reshares on a post). - Dataset curation. The source, size, and contents of datasets used for model training and testing. Provenance information includes temporal labels for 1) the date of the dataset’s production and 2) the date of dataset access by paper authors. - Model choice. The choice of ML model used by paper authors for their detection task, as well as their motivation for this choice. - Feature selection. The choice of feature(s) that paper authors use to train their ML models, as well as their motivation for this choice. - Model evaluation. Paper authors’ approach to benchmarking method performance after initial training. This includes 1) choice of test dataset; 2) performance statistics (e.g., ROC values, true and false positive rates); and 3) ecological validity of test cases in relation to proposed deployment setting. Paper organization. In Section 2, we situate this project within existing security literature and academic literature that critiques machine learning-driven methods. We motivate our choice of automated misinformation detection as a representative case study and highlight the particular relevance of this problem to the security community. In Section 3, we present the information taxonomies developed from our inductive coding of papers. We discuss findings from our reading and coding of papers in Section 4. Our systematization of literature progresses along two axes: 1) from unimodal to increasingly multi-modal methods, and 2) in order of operations of method development. Specifically, we discuss issues pertaining to method-target fit, dataset curation and feature selection, model selection, and method evaluation (RQ1, RQ2). In Section 5, we illustrate the issues identified in our literature review with a series of replication studies, with a focus on reproducibility and generalizability of results (RQ3, RQ4). In Section 6, we conclude with 1) a discussion of findings from our literature review and replication studies and 2) recommendations for evaluating ML-driven interventions for trust and safety. 4 Systematization of Literature Paper selection. To seed our corpus, we manually curated a selection of 23 highly-cited survey papers that provide comprehensive overviews of the state of automated misinformation detection at the time of writing [khan_benchmark_2021, fallis_functional_2014, zhou_survey_2020, alam_survey_2021, oshikawa_survey_2018, conroy_automatic_2015, sharma_combating_2019, islam_deep_2020, ahmed_detecting_2021, shu_fake_nodate, habib_false_2019, zhang_fake_2018, de_oliveira_identifying_2021, shu_mining_2020, miro-llinares_misinformation_2021, chen_news_2015, fernandez_online_2018, martin_recent_nodate, shelke_source_2019, guo_future_2019, thorne_automated_2018, zubiaga_analysing_2016, rubin_deception_2015]. We relied on these papers to ground our own understanding of existing approaches to automated misinformation detection, and to identify detection methods that have been well-received by the research community. We searched for survey papers in Google Scholar with queries “survey misinformation detection” and “survey fake news detection” and collected the most-frequently cited papers within the past 10 years. This time frame was naturally enforced by a lack of well-cited older publications, and was not fixed before we began our sampling process. We then inspected each paper’s reference section for related papers; we read the abstracts for these references in order to confirm relevance and fit. We supplemented this core corpus of highly-cited papers with publications surfaced by Google Scholar queries. We queried the following terms on Google Scholar: “ misinformation detection [ $x$ ]” and “ automated fact checking [ $x$ ],” where $x$ $∈\{$ claims, news articles, accounts, networks, websites, influence operations We include “influence operations” as a separate search term because this term of art is a relatively new one—we group most of these works with our network-scoped methods. $\}$ . We collected the 50 most highly-cited papers in the set resulting from the union of search results returned by both search queries for each $x$ . To counter potential bias toward older publications, we collected papers with the highest citation rates per year. We note that these search terms are deliberately over-inclusive; we manually review all potential works for relevance after the initial sampling step. After removal of out-of-scope works (see In- and out-of-scope work) from this set of 250 papers, 219 eligible papers remained. To ensure that security-oriented approaches to detection were represented in our corpus, we conducted a separate snowball sampling search for work published in security venues: Using the same keywords listed in the previous subsection, we oversampled publications from four A* [core] security research venues (USENIX Security, IEEE S&P, NDSS, and ACM CCS). This process resulted in the addition of 29 works, most of which address the detection of accounts and networks that spread misinformative content (e.g., botnets and trolls). Our final corpus comprises 248 papers published between 2009 and 2024, inclusive. “Full” and “focus” paper corpora. We conduct our literature survey at two different levels of granularity. For all papers in the full corpus (248 works), we note detection targets and model evaluation (steps 1 and 5 in Section 3). For a subset of these (87), we perform deep-coding of methods: we note dataset design choices, model selection, and feature sets (resp. steps 2, 3, and 4 in Section 3). We take inspiration from [wei_sociodem_2024, scheffler_e2ee_2023, warford_ux_2021], who perform a similar deep-coding of a subset of their whole-paper corpus. A partial summary of this deep-coding is available in Table 2. Our motivation for developing this focus set is practical: Many of the works we review do not include in-depth discussions of model choice, weights, and datasets. As such, each per-scope discussion is book-ended by overviews of the full paper corpus, with focused analyses of method design details in between. Targets C Claims: i. Content-based detection via distance calculations on semantic embeddings; ii. Content-based detection via search on knowledge graph topology; iii. “Checkability” or “checkworthiness.” A Articles: i. Syntactic and stylometric signals, including genre and sentiment; ii. Topic-aware detection of stance and relevance to known rumoring topics. U Users: i. Account metadata (bios, images, account age); ii. Single account behaviors (comments on posts, published posts). N Networks: i. Propagation patterns across social graphs; ii. Timestamped records of user-user interactions. W Websites: i. Text and URL/domain semantics; ii. Site visitor demographics; iii. Suspicious UI elements; iv. Hosting infrastructure (DNS certificate, site age). Datasets i. Dataset age; ii. Evidence of leakage (temporal leakage, feature leakage?); iii. Data dependencies (author, source, style); iv. Availability of information required to reproduce or reconstruct similar datasets (was non-public information required to produce ground-truth training sets?); v. Availability of original data. Models i. Distance calculations on semantic embeddings; ii. “Traditional” ML (SVM, RF, DT); iii. Deep learning (CNN, LSTM, GRU); iv. Graph cut algorithms; v. Stacked ensemble classifiers; vi. Graph clustering algorithms. Features i. Textual; ii. Network-based; iii. Author-, user- or source-based; iv. Infrastructural. Evaluation i. Testing in real time; ii. Generalizability of approach; iii. Evasion-resistance; iv. Robustness to distributional shifts in training or test data; v. False-positive/false-negative rates. Table 1: Taxonomy of targets, models, features, and evaluation codes. Alphanumeric abbreviations (e.g., “ C.i ”) are referenced throughout this work. Corpus curation and coding. As a first-pass relevancy check, we used automated keyword matching to confirm that all collected papers did, in fact, address misinformation and automated fact-checking methods; we read the abstracts of papers surfaced by this check to confirm relevance and fit. We then annotated research papers that passed this check in accordance with our codebook (see SI). One reader made three separate coding passes over the corpus, varying the objectives of her annotation on each pass: In the first pass, she specifically sought to identify taxonomies for classification and detection; on the second, she noted actual versus stated detection targets and approaches to evaluation; on the final pass, she noted method design approaches and errors. Two coders read and independently coded a random subset (30 papers) of the full corpus. Fleiss’s kappa for this subset was > 0.80, indicating strong agreement. In- and out-of-scope work. We consider a number of security-oriented approaches to misinformation detection in this work. Sybil and botnet detection methods that specifically target influence operations online are categorized as account- and network-scoped detection methods in our corpus. Commercial approaches to misinformation detection are in-scope for this project. In view of data accessibility issues, however, we are unable to provide an in-depth analysis of commercial methodologies in our main literature review, and instead include a market survey of commercial fact-checking providers in Appendix A. (In general, commercial vendors do not make code and training data publicly available.) LLM-powered detection is in-scope for this work. Accessibility to code bases for LLMs is similarly limited (this has been discussed at length in the popular press [mollman_2023, wong_2024]), and disallows testing and evaluation by researchers. As such, while we do briefly discuss LLM-powered detection approaches in our supplement, we defer a more extensive discussion to future work. Notation. In these sections, we denote information scope and pipeline steps with circled icons ( A, 1) and subtaxa within those categories with lowercase bolded Roman numerals (iv). We note that, while targets are generally unique to each information scope (and are referenced by an alphanumeric label (e.g., (A.i)), critiques of steps 2 - 5 are not scope-specific, and are always denoted by Arabic-Roman numeral pairs (e.g., (2.i)). We highlight each subtaxon in-text, with a different color corresponding to each development step. (The full taxonomy is in Table 1.) A summary of findings for each scope is available in the starred **Takeaways summary at the end of each subsection. 2 Datasets. Claim-scoped papers in our corpus that propose to perform fact verification (2.i) rely on outdated existing datasets of labeled statements in order to establish ground truth. LIAR [wang_liar_2017] We note that the LIAR dataset is actually a collection of 27.8K labeled PolitiFact statements—so, in a sense, PolitiFact is the dominant data source. and PolitiFact [politifact] were the most popular datasets among claim-scoped works, and, taken together, were used by approximately half of all papers within this scope [wang_learning_2019, soprano_many_2021]. LIAR is a static political news dataset that was published in 2017; on average, papers that cite LIAR were published two years after LIAR’s release. Topic detection and word frequency models trained on LIAR are likely ineffective in contemporary fact-checking contexts, and for non-political subject matter [debnath_hierarchical_2020, rasool_multi-label_2019]. Choice of ground truth site or labeled dataset can significantly influence the outcome of analysis: Bozarth et al. found that perceived prevalence of misinformation in a corpus of 2016 election news varied from 2% to 40%, depending on choice of ground-truth reference website [bozarth-truth-2020]. Additionally, misinformation taxonomies across reference sites are inconsistent: PolitiFact employs a six-point labeling scale (pants-on-fire; false; barely-true; half-true; mostly-true; true), FEVER employs a three-point scale (supported; refuted; notenoughinfo), and GossipCop employs an eleven-point scale (ratings from 0 to 10). This divergence is likely a symptom of definitional issues (Section 3). 3 Model selection. Model choice follows target choice for claim-scoped methods: for methods that pre-construct knowledge graphs or other reference databases, models perform some form of (3.i) shortest-path search on the KG topology [shiralkar_finding_2017, ciampaglia-wiki-2015], and approximate logical inference via transitive closure on graph edges. For methods that perform information retrieval at query time—e.g., to match corroborating sources to a claim to be checked—model training generally follows conversion of the text statement to a bag-of-words or TF-IDF embedding; choice of model is highly variable [bhutani_fake_2019, shu_defend_2019], and does not appear to be predictive of performance. For methods that perform detection of misinformative posts on social media, (3.v) stacked ensemble classifiers are a common approach to incorporating multiple feature modalities. | 1. Ajao et al. [ajao_sentiment_2019] | C A | Sentiment (A.i) | PHEME [pheme_2018] | LSTM, DT, RF, SVM | | | | | 0.86 (Acc.) | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | 2. Abulldah-All-Tanvir et al. [abdullah-all-tanvir_detecting_2019] | C A N | Content (C.i) | Twitter (API) | NB, RNN, LSTM, SVM, Logit | | | | | 0.89 (Acc.) | | 3. Bhutani et al. [bhutani_fake_2019] | C A | Content (C.i); sentiment (A.i) | Twitter (API), PolitiFact [politifact] | Naive Bayes, RF | | | | | 0.60 (AUC) | | 4. Bozarth et al. [bozarth-truth-2020] | C | Contents (C.i) | PolitiFact [politifact], Daily Dot, Zimdars, MBFC | LDA | | | | | n/a | | 5. Ciampaglia et al. [ciampaglia-wiki-2015] | C N | Shortest path search (C.ii) | DBpedia | kNN, RF | | | | | 0.97 (AUC) | | 1. Afroz et al. [afroz_detecting_2012] | A | Content (C.i); syntax (A.i) | Brennan-Greenstadt | SVM, J48 Decision Trees | | | | | 0.97 (F1) | | 2. Ahmed et al. [ahmed_detecting_2018] | A | Syntax (A.i) | Twitter, Kaggle, Horne and Adali [horne_this_2017] | SVM | | | | | 0.92 (Acc.) | | 3. Bourgonje et al. [bourgonje_clickbait_2017] | A | Stance (A.ii) | Fake News Challenge Data | Logit | | | | | 0.90 (Acc.) | | 4. Brasoveanu et al. [brasoveanu_semantic_2019] | A C | Sentiment (A.i); keywords (A.ii) | LIAR [wang_liar_2017] | CNN, LSTM, CN | | | | | 0.64 (Acc.) | | 5. Della Vedova et al. [della_vedova_automatic_2018] | A N | Content (C.i); virality (N.iv) | FakeNewsNet, Buzzfeed | Logit | | | | | 0.82 (Acc.) | | 1. Cao et al. [cao_aiding_2012] | U N | Acct. cred. (U.i); prop. (N.i) | Tuenti social network | Louvain clustering | | | | | 0.90+ (TP) | | 2. Danezis et al. [danezis2009sybilinfer] | U N | Acct. cred. (U.i); prop. (N.i) | LiveJournal data | Bayesian inf. | | | | | n/a* | | 3. Ezzeddine et al. [ezzeddine2023exposing] | U | Acct. behaviors (U.ii) | DATA | LSTM | | | | | 0.91 (AUC) | | 4. Hamdi et al. [hamdi_hybrid_2020] | U N | Account metadata (U.i); prop. (N.i) | CREDBANK | LDA, Bayes, Logit, SVM | | | | | 0.99 (AUC) | | 5. Helmstetter et al. [helmstetter-supervised-2018] | U N A | Acct metadata (U.i); post sharing data (U.ii) | Public site cred. lists | SVM, NB, DT, RF | | | | | 0.936 (F1) | | 1. Alizadeh et al. [alizadeh2020content] | N A U | Propagation (N.i); syntax (A.i); acct metadata (U.i) | Twitter (API), Reddit IRA troll list | RF | | | | | 0.70+ (F1) | | 2. Antoniadis et al. [antoniadis_model_2015] | U A | Acct metadata (U.i); syntax (A.i) | Hurricane Sandy tweet dataset | J48, RF, KNN, Bayes | | | | | 0.79 (Avg. Prec.) | | 3. Assenmacher et al. [assenmacher2020two] | N A | Propagation (N.i); topic det. (A.ii) | Twitter (API) | Clustering | | | | | not reported | | 4. Buntain et al. [buntain_automatically_2017] | N U A | Time (N.ii); acct metadata (U.i); sentiment (A.i) | CREDBANK, Buzzfeed | RF | | | | | 0.65 (Acc.) | | 5. Castillo et al. [castillo_information_2011] | U A | Syntax (A.i); user behavior (U.ii) | Twitter Monitor events | SVM, DT | | | | | 0.874 (P) | | 1. Asr et al. [torabi_asr_data_2018] | W A | Source rep. (W.i); Syntax (A.i) | BuzzfeedUSE, Snopes, Rashkin, Rubin | CNN, SVM, NB | | | | | not reported | | 2. Baly et al. [baly_predicting_nodate] | W A N | Source rep. (W.i); Site infra. (W.ii); (A.i) | MediaBiasFactCheck [mediabiasfactheck] | SVM | | | | | 0.66 (Acc.) | | 3. Baly et al. [baly-etal-2020-written] | W A N | Source rep. (W.i); Site infra. (W.ii); (A.i) | MediaBiasFactCheck [mediabiasfactheck] | SVM | | | | | 0.7152 (Acc.) | | 4. Castelo et al. [castelo_agnostic_2019] | W A | Site infra. (W.ii); syntax (A.i) | Celebrity, US-Election2016 | SVM, kNN, RF | | | | | 0.86 (Acc.) | | 5. Chen et al. [chen_proactive_2020] | W A | Hosting infra. (URL) (W.ii); syntax (A.i) | PoliticalFakeNews | Clustering | | | | | 0.97 (AUC) | Table 2: Focus corpus by scope and target. Coding of 25 papers in our focus set, sorted by information scope. Codes for the full focus set appear in Table 4. Values in parentheses in “Target” field correspond to highlighted subcategories presented in Section 4 (e.g., “C.i” denotes target (i) in “Claims,” Section 4.1). If authors present evaluation results for multiple models, we underline the most performant model and record its corresponding performance score. 4.1 Claims About 70% of paper authors within this scope cite the 2016 U.S. presidential election as motivation for the development of their methods [debnath_hierarchical_2020, hassan_data_2014, katsaros_which_2019]; about 30% cite COVID-19 misinformation [kou_hc-covid_2022, roitero_can_2021, glockner_missing_2022]. All works mention the speed and volume of social media misinformation in particular [kartal_too_2020]. 30% cite the relatively slow pace of manual fact-checking [ciampaglia-wiki-2015, hassan_data_2014, kartal_too_2020] —and the need for faster, automated approaches—as motivation. Claim-scoped papers form 15% of our corpus. 1 Detection targets. Across all information scopes, we find that claim-scoped detection methods are most consistent in their attempts to verify semantic contents of text statements. This is done in the following ways: (C.i) distance calculations on semantic embeddings to perform textual entailment or stance detection [saikh_novel_2019, borges_combining_2018]; and (C.ii) search on a knowledge graph topology [ciampaglia-wiki-2015, shiralkar_finding_2017] to determine if these reference sources corroborate or refute the claim to be checked. A small class of approaches explicitly detect (C.iii) “ checkworthiness,” We include these works in our corpus because they do take topic and source credibility into consideration in the process of ranking checkability. and are intended to surface checkable statements to human fact-checkers for manual verification [kartal_too_2020, shiralkar_finding_2017, hassan_data_2014]. Targets such as author credibility and language cues are proxy targets for the presence of misinformation: signals which may not be sufficient for determining text veracity in isolation, but which are indicative of (lack of) veracity by association with an external heuristic. About 60% of papers within the full corpus at this scope employ non-semantic targets, including the (A.i) syntactic and/or stylometric qualities of text [jadhav_fake_2019], (U.i) source reputation [rasool_multi-label_2019], or (U.ii) contextual indicators, such as commenter responses [tian_early_2020], to classify social media posts. 4 Feature selection. At the level of single claims, semantic feature analysis is limited to the (4.i) identification of structured statements as a precursor to knowledge graph (KG) construction. These claims take the form of subject-predicate-object (SPO) statements (e.g., “I like pie”) [ciampaglia-wiki-2015, magdy-stat-2010, shiralkar_finding_2017]. Detection versatility is determined by the size of the source dataset and the granularity of the relationships encoded by graph edges [shiralkar_finding_2017]. Supervised methods that detect linguistic cues employ (4.i) hand-crafted word or topic lists [kartal_too_2020]). Authors employing supervised methods claim that their approach permits highly customized targeting of specific rumoring narratives [kartal_too_2020], though this also assumes that the method developer has prior knowledge of the contents of test data; authors employing unsupervised methods claim that their approaches detect contextual and language features that cannot be easily extracted by common features such as word frequency or sentiment [wang_learning_2019]. Methods detecting social media data consider (4.ii) network and (4.iii) user interaction features (post likes, shares, comments) [paudel_lambretta_2023, gupta_tweetcred_2015, cui_same_2019]. 5 Evaluation. Though a majority of claim-scoped methods cite the speed of social media misinformation as motivation, (5.i) only one method within this scope reported results from testing in real time [hassan_toward_2017]. KG-based methods are (5.ii) non-generalizable by design: the approaches we survey require structured inputs for graph construction, and rely on published datasets for ground truth [ciampaglia-wiki-2015, shiralkar_finding_2017]. Though this approach permits semantic verification of statements, it is difficult to perform iterative updates to source databases in real-time, particularly in the types of online settings where claim-checking might be most usefully deployed (e.g., during breaking news events, where no source of ground truth is immediately available). We observe that a majority of works at this scope do not test on novel or out-of-domain data; we discuss overfitting issues in greater detail in the next section. **Takeaways: 1 The efficacy of claim-scoped methods is completely determined by the depth and breadth of coverage conferred by a reference database. 2 Datasets are frequently out-of-date, and taxonomies are inconsistent. 3 Though some inference is possible via transitive closure on knowledge graph edges, this capability is, in general, limited. 4 Knowledge-graph-based methods require structured inputs, which might not be readily available in a breaking news setting, or in scenarios where ground truth references are not yet available. 5 Few authors test in real-time, despite citing the slow pace of manual fact-checking as motivation. 4.2 Articles We consider all news-oriented writing of length 100 words or greater to fall within this scope. 25% of papers within this scope cite growing distrust of mainstream news media outlets as motivation for their methods, which promise to deliver fast labeling of news stories that appear on social media [rubin_deception_2015]. Text-based credibility classifiers have been shown to have limited efficacy, however: while unsupervised approaches can identify bias with high accuracy, this performance degrades in misinformation and credibility classification tasks [fairbanks2018credibility, potthast-etal-2018-stylometric]. This scope comprises 24% of our paper corpus. 1 Detection targets. In contrast to single claims, full-length news articles have sufficient text contents to make semantic verification difficult—and certain off-the-shelf NLP approaches practicable. These approaches are distinct from direct claim verification and qualify as proxy detection methods: For instance, Bhutani et al. associate strongly negative sentiment with the presence of false information [bhutani_fake_2019], and Horne et al. find that satire and misinformation share stylistic similarities [horne_this_2017]. All article-scoped methods target proxy signals, and adopt at least one of the following three approaches to detection: (A.i) NLP analysis of article contents to identify language features particular to writing styles heuristically associated with misinformation (genre detection, sentiment analysis) [mihalcea-strapparava-2009-lies]; and (A.ii) analysis of article contents and headlines to identify potentially clickbait-y titles or discussion related to known misinformation narratives (topic detection) [zhang_fake_2018, kartal_too_2020]. Respectively, these approaches 1) simultaneously assume and detect a heuristic (e.g., strong emotion indicating the presence of misinformation); and 2) assume prior knowledge of rumoring topics. 2 Datasets. While well-annotated, current datasets are in short supply across all information scopes, this deficit is particularly glaring at the article scope. This is due in large part to definitional ambiguities that prevent fine-grained labeling of longer texts for classifier training. As a result, 44% of paper authors within this scope (2.i) use public datasets released years prior to the start of their research [brasoveanu_semantic_2019, wang_liar_2017, horne_this_2017, shu_beyond_2019]. These datasets (LIAR [wang_liar_2017], Buzzfeed-Webis [potthast-etal-2018-stylometric], and PolitiFact [politifact]) include news links, speaker credibility scores, and other metadata that (2.ii) constitute serious sources of leakage for methods that use contextual features to infer true/false labels. The remaining 56% of authors curate their own article corpora by asking crowdworkers to generate misinformative text [perez-rosas_automatic_2017], selectively editing true news articles (e.g., via verb inversion or noun replacement) [mihalcea_lie_2009], or compiling articles from authoritative news sources and known satire sites [rubin_fake_2016]. These data curation techniques (2.iii) introduce additional dependencies and shortcuts to textual datasets for which such variables are already difficult to control. Style [potthast-etal-2018-stylometric] and genre [hosseinimotlagh_unsupervised_nodate], for instance, are emergent qualities of writing that cannot be easily marginalized out of a text embedding. 3 Model selection. Among papers that report testing with multiple models—including (3.ii) classical ML models [ahmed_detecting_2018], (3.iii) unsupervised NN models [brasoveanu_semantic_2019], and (3.v) stacked ensemble classifiers [ruchansky_csi_2017] —there is no clear correlation between model choice and actual performance. We note that, in instances where authors test on two- and multi (i.e., $>$ 2)-way classification tasks, performance declines sharply in the latter case [afroz_detecting_2012]. In those instances, reported performance scores are for two-way tasks. For this reason, as well, classical ML models (logit, SVM) oftentimes appear to be most performant. 4 Feature selection. Among supervised methods that disclose their feature sets, we find that (4.i) word frequency, sentiment, and genre were among the most commonly used features, and were collectively employed by 80% of works within this scope; these features can also be sources of dependency-induced noise. It is difficult to quantify the impact of dependencies related to voice, house style, and source on classifier performance, particularly in the case of unsupervised learning methods, which comprise 59% of methods at this scope. We evaluate an unsupervised learning method (and consider its performance in light of possible style-related dependencies) in our replication analysis of Nasir et al. (Section 5.1) [nasir_fake_2021]. 5 Evaluation. Misinformation detection methods scoped to full texts risk overfitting to single topics: 42% of authors select (5.ii) one or more narratives of interest (e.g., the 2016 presidential election, the Boston Marathon bombing), train a classifier on these topics, then test this classifier on a different set of texts that discuss the same topic [noauthor_rumors_2014, jin_detection_2017, nasir_fake_2021]. This approach, while valid for evaluating classifier performance on closed datasets, lacks ecological validity for the use cases that authors claim that their methods will address: Rapid topic identification and high-quality annotation of relevant articles are generally unavailable in breaking news scenarios on social media platforms [shu2020detecting]. 60% of authors at this scope compare the performance of their detection method to other published approaches or ML models, but (5.ii) neglect to test on novel datasets. These methods do well when tested on in-domain texts, and in comparison to a selection of older ML models; many report accuracy well above 80% [zhang_detecting_2019, silva_embracing_2021]. Only one paper within the article scope tested in an adversarial setting: Its authors found that, while stylometry-based misinformation detection had an accuracy rate greater than 80% on routine tasks, this score dropped to about 50% in adversarial cases [afroz_detecting_2012]. We demonstrate such a dropoff in Section 5.1. **Takeaways: 1 In the absence of semantic definitions of misinformation, proxy detection targets are common but easy to evade. 2 Well-labeled datasets are rare; those datasets that are available are at least several years old at time of writing. 3 Unsupervised methods show marginal improvements over classical ML models in some cases; it is unclear if these improvements are 1) significant or 2) sustainable across different datasets. 4 Detection methods at the article scope are uniquely susceptible to text-based dependencies that are difficult to control for. 5 Inflated performance scores can often be attributed to testing on same-topic news articles. 4.3 Users Evidence of foreign interference during the 2016 U.S. presidential election triggered a resurgent interest in malicious account detection [ezzeddine2023exposing, saeed2022trollmagnifier]. As such, 90% of security- and social-science-oriented works that we include within this scope (a dozen papers) explicitly discuss Russian trolls or other influence operations conducted by nation state actors and train classifiers on published lists of such accounts [saeed2022trollmagnifier, zannettou2019disinformation, mirza2023tactics, lukito2020coordinating, keller2020political]. Account-scoped papers formed approximately 15% of our corpus (40 papers). 1 Detection targets. Papers within this scope target source reputation, and (U.i) inspect account metadata, such bios, account age, and profile images; or distinguish suspicious accounts by (U.ii) a single user’s social behaviors, such as their comments on posts (n.b. this target is distinct from (N.i)). The security literature we review discusses trolls and bots deployed for astroturfing, misinformation campaigns, and IOs [saeed2022trollmagnifier, cresci_bots_2020, chu2012detecting, zhang2016rise, alizadeh2020content]. In the absence of rigorous definitions of these account types, however, actual detection targets are tautological: A troll or bot is an account that exhibits troll- or bot-like behavior, or that interacts with confirmed troll or bot accounts [saeed2022trollmagnifier, cresci_bots_2020]. 2 Datasets. All troll and bot account detection works we reviewed relied on published lists of “known” troll accounts for model training, but (2.iv) neglected to mention the heavily manual investigation required to produce these original lists [silverman_timberg_kao_merrill_2022, linvill2020troll]. Researchers who compiled some of these account lists, including a set of several hundred Twitter accounts with possible links to a known Russian troll farm (the Internet Research Agency, or IRA) manually examined suspicious accounts and tweet contents in order to produce detailed account and content taxonomies; notably, these classifications required external intelligence about account activity that was not published alongside account lists [linvill2020troll]. Two well-cited troll lists compiled by the U.S. government, comprising thousands of suspicious Twitter and Facebook accounts, were curated using proprietary non-public information [exhibit_b]. 3 Model selection. Detection proceeds via classifier training on a list of “known” suspicious accounts and application of this classifier to a dataset of novel accounts [saeed2022trollmagnifier, lukito2020coordinating]. We note that, regardless of model choice, if classifier training data and feature selection reflect a heuristic about suspicious behavior, the resulting classifier will simply learn this heuristic: The methods we review can be used to detect accounts whose behaviors conform to heuristic assumptions, but cannot be used to surface novel malicious behaviors, and are not resistant to attacks or evasion [wang2014man]; we explore this further in our replication analysis of Saeed et al. (Section 5.2) [saeed2022trollmagnifier]. A subset of methods at this scope and the network scope formulate detection as a (3.iv) graph cut or influence maximization problem, and describe approaches to identifying optimal cuts for isolating suspicious accounts [zhang_misinformation_2016, cao_aiding_2012]. 4 Feature selection. Methods that define suspicious accounts by intrinsic properties of these accounts (e.g., user handles and profile images) target the (4.iii) semantics of this account metadata, and detect evidence of manipulation in (e.g.) image metadata and bios; or text outputs, such as posts and links [helmstetter-supervised-2018, sansonetti_unreliable_2020]. Methods that define suspicious accounts by account activity target (4.ii) networked behaviors, such as liking and resharing statistics [pacheco_uncovering_2020]. Feature sets for some methods in the first category include demographic data for users, such as inferred political party affiliation or race [ribeiro_media_2018] (these n th order assumptions are dangerous to make [freelon_black_2020]; see our discussion about proxy signals, in Section 4.2). 5 Evaluation. We observe accuracy scores above 80% for (5.ii) confirmatory detection of like accounts for all methods that reference a seed list of known trolls [saeed2022trollmagnifier, addawood_linguistic_2019, ezzeddine2023exposing]; de novo detection of behaviors not represented within training data is not possible, by the self-admission of 10% of authors within this scope [saeed2022trollmagnifier, stone2009botnet]. As we discuss further in the next subsection (Networks), the increasingly hybrid nature of IOs requires more nuanced taxonomies for classification: extent of coordination, rather than existence, might be a more appropriate measure of possible manipulation. In a blog post on bot detection from 2021, Twitter “debunked” four common heuristics commonly used to identify bot accounts, including several detected by methods in our corpus, and described a “forensic team of investigators” who manually verify bot-ness of suspect accounts [Thread_2021]. Some authors of bot detection methods acknowledge that their approaches are (5.iii) trivially easy to evade if account holders 1) avoid interacting with known suspicious accounts or 2) vary their account identity and posting semantics [danezis2009sybilinfer, cresci_bots_2020]. **Takeaways: 1 Targets are frequently tautological. 2 Hand-annotated training datasets are the result of intensive fact-finding on the part of human researchers, and often require information that is not publicly available. 3 Classifiers can only detect accounts resembling those in seed lists. 4 Features that attempt to infer user credibility from demographic information risk reinforcing existing biases. 5 Current methods cannot detect novel malicious behaviors. 4.4 Networks Within the security literature, a growing awareness of hybrid networks, which employ a combination of automated and manual approaches to disseminate content, has encouraged a turn toward network-based bot detection methods, and away from detection of individual accounts [cresci_bots_2020, grimme2018changing]. We observe a parallel turn toward network-based methods in AI, ML, and NLP venues as a result of growing recognition of overfitting and generalizability issues in purely text-based detection methods (see Articles) [castelo_agnostic_2019]. The common assumption, across disciplines, is that coordinated networks leave more detectable evidence of manipulation than do individual accounts, and that these footprints should be identifiable regardless of attack type or rumoring topic [castelo_agnostic_2019, milajerdi2019holmes]. Network-scoped methods, including relevant security literature, form 20% of our corpus. 1 Detection targets. All methods at this scope identify patterns of user interaction and content propagation as targets; these methods associate virality with the existence of rumoring narratives [vosoughi-twitter-2018, tambuscio_fact-checking_2015, shu_beyond_2019] and temporally anomalous activity with evidence of coordination [stone2009botnet, cresci_bots_2020, lukito2020coordinating]. The corresponding targets for these approaches are (N.i) propagation patterns across social graph topologies and (N.ii) temporal records of user-user interactions. Within non-security misinformation literature, we note that virality assumptions disallow early detection of misinformation [monti2019fake, castillo_information_2011]. Similarly, within the security literature, anomalous patterns of account registration and user interaction serve as proxies for the presence of Sybils and botnets [yuan2019detecting, danezis2009sybilinfer]; early detection requires that authors formulate a priori assumptions about the nature of these patterns. 2 Datasets. We conducted an author outreach survey for works within this scope in an attempt to locate hard-to-find social media datasets. We found that (2.v) accessibility issues were exacerbated by the shutdown of the Twitter API [twitter]. In total, we attempted to locate datasets and code for 50 different papers (see Section 5 for our methodology). Thirty-six (72%) of these analyzed tweet corpora, and 42 (84%) of these targeted social media users and posting contents. We were able to independently source complete methods or data for fourteen (28%) of these. Of the 27 authors we eventually contacted about providing partial or dehydrated datasets, nine responded; six of those authors were able to provide method code or partial datasets. 3 Model selection. The network-scoped methods we review formulate detection as 1) a structured content classification problem [alizadeh2020content, ratkiewicz2011detecting], and/or 2) a clustering problem on social graphs [assenmacher2020two, sharma_combating_2019]. In the former case, authors employ an assortment of (3.ii, 3.iii) supervised and unsupervised models to detect suspicious language across multiple accounts. In the latter case, authors use (3.vi) Louvain or K-means clustering or K-nearest neighbors to detect neighborhoods of suspicious accounts, as determined by user-user interactions. Though methods in the latter category advertise themselves as content-agnostic, we note that published methods [liu_early_nodate, ma_detect_2017] access datasets of social media posts that were already sorted by rumoring topic or event [ma_rnns_2016, liu_realtime_2015]. 4 Feature selection. 55% of papers within this scope make normative theoretical assumptions about user behaviors: In keeping with an epidemiological model Some academics have argued that the disease metaphor for misinformation promotes an overly simplistic model of information spread and opinion formation [williams_2023, tay2024thinking]. of misinformation spread [jin_epidemiological_2013, vanderlinden_2022], Nguyen et al. assume a homogeneous population of newsreaders, with (4.ii) identical probabilities of “infection” and reinfection [nguyen_containment_2012]. Similarly, in the security literature, techniques for detecting bots and Sybils identify behaviors that align with heuristics determined a priori by researchers: These methods assume, for instance, that Sybils will form well-connected neighborhoods [yuan2019detecting], or (seemingly contradictorily) that compromised Sybils will refrain from connecting with additional Sybils, to avoid detection [danezis2009sybilinfer]. In fact, the landmark paper by Douceur that characterized Sybil attacks stated that such attacks cannot be prevented unless special assumptions are made about account behaviors [douceur2002sybil]. 5 Evaluation. Ferrara et al. [ferrara2016rise] called attention to the false positive rate problem in botnet detection in 2016, noting that classifiers for bot detection only work well in instances where there is a clear-cut distinction between bot and non-bot accounts. This distinction is becoming increasingly blurred, however. Sophisticated network-based attacks try to engage non-bot accounts in organic interactions with bot accounts (e.g., astroturfing attacks) [cresci_bots_2020, stone2009botnet], (5.v) rendering even positive detection results insufficient or meaningless: coordinated activity need not be inauthentic, and inauthentic activity need not be malicious. **Takeaways: 1 Pattern- and virality-based detection approaches disallow early detection of rumors. 2 Current social media data is difficult to obtain. 3 “Topic-agnostic” classifier design occurs downstream of topic-aware dataset design. 4 Epidemiological models of information spread make strong assumptions about opinion formation and user behaviors; feature sets reflect these a priori notions. 5 Though network-scoped methods are less susceptible to content-based dependencies than are content-aware methods, they cannot infer intent or authenticity of the behaviors they detect. 4.5 Websites Methods within this scope apply credibility, factuality, or (political) bias scores to whole news sites; authors claim that site-wide labels can be used to quickly infer the quality of individual news articles produced by these sites [hounsel-infra-2020, baly-etal-2020-written, castelo_agnostic_2019, chen_proactive_2020]. As with article-scoped methods, we noted intervention fit issues at the whole website scope. Asr et al. found that whole-source labels were insufficient proxies for the truthfulness of single news articles, and elided subject-specific variations in reporting quality [torabi_asr_data_2018]. (We discuss dataset distribution in greater depth in Datasets.) 1 Detection targets. In 50% of works that we review in this scope, authors reduce the task of whole-site credibility labeling to a significantly smaller, unimodal classification task: Chen et al. [chen_proactive_2020] (W.i) detect suspicious domain semantics (in essence, a text classification task on URLs); Ribeiro et al. [ribeiro_media_2018] (W.ii) infer site bias from site visitor demographics; Castillo et al. [castillo_information_2011] (W.iii) detect suspicious ad interfaces and markup features; Baly et al. [baly_predicting_nodate, baly-etal-2020-written] and Hounsel et al. [hounsel-infra-2020] present methods incorporating (W.iv) infrastructural features, though the overall performance of the method of Baly et al. is strongly determined by performance on the text classification task alone (thus, in practice, the method closely resembles the article-scoped detection methods we examine, and is susceptible to the same dependencies that we observe in that scope). We demonstrate this via an ablation analysis in our replication study of their method (see Section 5.3). 2 Datasets. For both training and testing, all methods scoped to whole website detection rely on published lists of websites with accompanying credibility scores [baly_predicting_nodate, baly-etal-2020-written, hounsel-infra-2020, castelo_agnostic_2019]; common reference sites include Media Bias/Fact Check, Snopes, and FactCheck.org [mediabiasfactheck, snopes, FactCheck]. As discussed in Section 4.1, however, these references do not have uniform taxonomies for classifying site credibility. Additionally, pre-labeled lists are 1) (2.i) biased towards older, more visible real news and fake news outlets [snopes], 2) (2.iii) are restricted to specific information domains [baly_predicting_nodate], or 3) (2.i) include inactive websites within their labeled datasets [mediabiasfactheck]. The median lifespan of a set of 283 misinformation news sites is 4 years, per a survey conducted by Chalkiadakis et al. in 2021 [chalkiadakis_rise_2021]. We note papers that perform website infrastructure analysis on contemporaneous snapshots (circa 2019) of websites in their corpora, even though the text-based features for the same analysis were drawn from datasets published in 2016 and 2017 [castelo_agnostic_2019]. This constitutes a serious source of (2.ii) temporal leakage. No works within this scope discuss approaches to accounting for uneven distributions in training data, or how they might account for shifts in baseline distributions during the lifecycle of an active website. Hounsel et al., for instance, train their classifier on a reference list in which 34% of misinformation training set sites were active and all websites in the real news training set were active, possibly resulting in overfitting to features specific to those inactive websites [hounsel-infra-2020]. 3 Model selection. Four of the seven methods we reviewed within this scope employed (3.ii) SVM classifiers; in two of those cases, SVM outperformed other, more complex unsupervised models [torabi_asr_data_2018, castelo_agnostic_2019]. These results accord with our earlier observation, in Section 4.2, that SVM classifiers are comparatively performant on two-way classification tasks. 4 Feature selection. Works within this scope employ multi-modal feature sets comprising a mix of (4.i) textual, (4.ii) network-based, and (4.iv) infrastructural signals: for instance, Baly et al. [baly-etal-2020-written] consider network traffic, URL semantics, and site contents; and Hounsel et al. [hounsel-infra-2020] consider TLS/SSL certificates, web hosting configurations, and domain registrations. None of the detection methods we reviewed discusses the computational costs of deploying their methods at scale. Though all works discuss their feature selection process (via leave-one-out and use-one-only evaluations), none describes a process for normalizing or weighting features according to dataset distribution or detection setting needs. 5 Evaluation. Methods within this scope that propose to perform whole-site labeling from analysis of a selection of news articles or infrastructural features are susceptible to distributional imbalances (e.g., between news verticals represented in an article corpus). Baly et al. train their model on (5.ii) political news websites only, and their credibility labels are strongly correlated with political bias scores. Hounsel et al. perform (5.i) testing in real time —one of the few studies we reviewed, and one of two studies at this scope, that did so [hounsel-infra-2020, chen_proactive_2020]. Both of these works report significant performance dropoffs between experimental and real-time tests, most likely as a result of distributional differences between real-world and experimental datasets (in practice, most websites do not host news-related content at all) [hounsel-infra-2020, chen_proactive_2020]. **Takeaways: 1 Methods claiming to classify news sites generally reduce this task to simpler, unimodal ones, such as URL classification. 2 The works we review do not consider shifts in feature distribution over time. 3 We note that SVM classifiers perform well on two-way classification tasks, and even outperform more sophisticated unsupervised models. 4 Feature normalization largely undiscussed. 5 Authors who conduct testing in separate real-time settings report significant performance dropoffs with respect to experimental results. 5 REPLICATION STUDIES We choose three distinct targets and scopes to replicate issues identified in our literature review. Our targets are 1) (A.i) syntax-based text features; 2) (U.ii) user behaviors; and 3) (W.iv) multimodal whole-website features, including hosted content and infrastructure. These works are highly representative of their respective information scopes: Nasir et al. [nasir_fake_2021] (Section 5.1) train a neural network on corpora of true and misinformative news articles; Saeed et al. [saeed2022trollmagnifier] (Section 5.2) use published lists of trolls to infer the presence of other troll accounts on Reddit; and Baly et al. [baly_predicting_nodate] (Section 5.3) examine a multimodal feature set comprising infrastructure, content, and network-based features in order to infer whole-site credibility. For each work, we evaluate replicability and generalizability (RQ3, RQ4). Where possible, we 1) replicate reported results; 2) inspect datasets for potential dependencies; 3) perform ablation analyses to understand individual feature performance; and 4) test on current data. Paper selection criteria and author outreach. We sorted our full text corpus by information scope. Within each scope, we sorted papers in order of decreasing citation count. We then proceeded as follows: 1. We attempted to source the full methods and datasets for the most cited paper within each information scope. 1. If we were unable to find this information during an independent web search, we reached out to the paper’s lead author(s) to request access. 1. If this request was unsuccessful—if the author did not respond, or confirmed that the dataset or code was no longer available—we returned to step 1 for the next most highly-cited paper in our corpus within that scope. Replication analyses. In order to reproduce results published in a selection of papers and perform cross-cutting analyses on out-of-domain and out-of-sample datasets, we conduct a series of replication analyses on a subset of papers. We chose representative methods from disparate information scopes, and which consider a variety of different feature types. We reproduced results reported in each paper on the datasets mentioned therein, contacting authors when necessary to obtain datasets and code. We evaluated reproducibility and generalizability as follows: - Reproducibility. We reproduce published results with code and data reported in the original publication. We evaluate availability of code and data and, where possible, compare our analysis outcomes with those reported in the original paper (RQ3). - Explainability. Toward understanding the contributions of specific feature types to overall classifier performance, and why certain approaches work, we perform feature ablation studies when appropriate. - Replicability and generalizability. Model perfor-mance on novel datasets is useful for determining the generalizability of existing detection methods to different contexts (RQ4). For models that were explicitly tested on specific misinformation narratives (e.g., 2020 stolen election narratives), on specific timeframes, or on specific types of misinformation (e.g., parody, satire), we develop updated datasets to test method performance on diverse information domains. 5.1 Detection of suspicious language In view of rampant data dependency issues identified in our survey of article-scoped literature, we reproduced results from a representative study published in 2021 by Nasir et al [nasir_fake_2021]. The authors propose a neural net-based approach to the classification of news articles. The method employs a hybrid deep learning model that combines convolutional and recurrent neural networks for the classification of real and fake news. The authors report results for tests on two datasets: the ISOT dataset (45,000 news stories, equally distributed across true and false categories, as labeled by PolitiFact) and the FA-KES dataset (804 news stories about the Syrian war, 426 true and 376 false) [isot, fa_kes]. We were able to replicate original paper results and run the method on updated datasets of news articles. Motivated by the prevalence of methods trained on few- or single-source datasets in the article scope, we test possible dependencies related to journalistic house style, as all true articles in the training corpus were sourced from Reuters. House style as a confounder. To investigate house style as a possible confounder for misinformation detection, we excerpted 100 articles from both Reuters and The New York Times, two news outlets with distinctive (and different) reporting styles. We randomly selected these articles from both outlets’ RSS feeds in May 2023. We sourced articles for both corpora from the following verticals: U.S. and world politics, economics, science, and entertainment. Excerpt lengths ranged from 100 to 300 words. These corpora, each comprising 100 true news stories, are labeled “Reuters- original ” and “NYTimes- original ” in Table 3. Table 3: Replication analysis of Nasir et al. (2021). We tested the method of Nasir et al. on both datasets discussed in the original paper, and on novel datasets from Reuters and The New York Times [nasir_fake_2021]. We then selectively edited 50% of the news articles within each corpus. We changed proper nouns, negated verbs, and altered reported statistics so that the factual content of these articles was no longer accurate but house style and tone were preserved. We call these altered text corpora “Reuters- modified ” and “NYTimes- modified ” in Table 3. The classifier had 0.727 accuracy on Reuters- original and 0.673 accuracy on NYTimes- original; this difference is not significant (p > 0.05). Additionally, the classifier had about 0.50 (random) accuracy on both modified datasets. The classifier’s false positive and false negative rates on modified and original Reuters and NYTimes corpora tell a more interesting story, however: Overwhelmingly, false NYTimes articles were classified as true (FPR = 0.02 and 0.735 for Reuters- modified and NYTimes- modified, respectively), while all true Reuters articles were classified as false (FNR = 1 and 0.286 for Reuters- modified and NYTimes- modified, respectively). These results indicate that the classifier was significantly more conservative in its assignment of true labels for the Reuters dataset than it was for the NYTimes dataset; additionally, within each modified corpus, the classifier did not effectively differentiate between true and untrue news articles. We note that classifier performance in an adversarial setting was no better than random, and that house style appears to have a significant impact on classifier sensitivity to misinformative texts. 5.2 Detection of suspicious accounts We reproduce results from a study published in 2022 by Saeed et al. [saeed2022trollmagnifier]. In summary, the authors propose a method, called TrollMagnifier, for the identification of Reddit accounts that exhibit troll-like behaviors. Like all other account-scoped methods we analyze in the security literature, TrollMagnifier is trained on posting and reply statistics for non-troll and known Russian troll accounts (as identified by Reddit) [saeed2022trollmagnifier]. Reproducibility of published results. Study authors provided us with pre-processed datasets and classifier code upon email request. The full Reddit Pushshift dataset is freely available online [baumgartner2020pushshift]. We were able to replicate original paper results using these materials. Original account handles were anonymized; as such, we were unable to verify if the accounts identified in the original study appeared to be troll-like. <details> <summary>x1.png Details</summary>  ### Visual Description ## Partial Dependence Plots: Feature Analysis ### Overview The image presents six partial dependence plots, each illustrating the relationship between a single feature (x1 through x6) and the model's predicted outcome. Each plot shows how the model's prediction changes as the feature value varies, while all other features are held constant. ### Components/Axes * **Y-axis (Vertical):** "Partial dependence", ranging from approximately 0.4 to 0.7. * **X-axis (Horizontal):** Feature values for x1, x2, x3, x4, x5, and x6. The ranges vary for each feature: * x1: 0.0 to 1.0 * x2: 0.0 to 1.0 * x3: 0.0 to 1.0 * x4: 0.0 to 0.1 * x5: 0.00 to 0.02 * x6: 0.0 to 0.4 * **Data Series:** Each plot contains a single blue line representing the partial dependence of the model's prediction on the respective feature. * **Tick Marks:** Small vertical black lines along the x-axis indicate the distribution of the feature values in the dataset. ### Detailed Analysis **Plot 1: x1** * **Trend:** The partial dependence increases sharply from x1 = 0.0 to approximately x1 = 0.2, then plateaus around a partial dependence of 0.7 for x1 values between 0.2 and 1.0. * **Values:** * At x1 = 0.0, partial dependence is approximately 0.45. * At x1 = 0.2, partial dependence reaches approximately 0.7. * From x1 = 0.2 to 1.0, partial dependence remains roughly constant at 0.7. **Plot 2: x2** * **Trend:** The partial dependence shows a sharp spike near x2 = 0.0, followed by a rapid decrease, and then a gradual decline as x2 increases. * **Values:** * At x2 = 0.0, partial dependence peaks at approximately 0.6. * At x2 = 0.1, partial dependence drops to approximately 0.4. * At x2 = 1.0, partial dependence is approximately 0.3. **Plot 3: x3** * **Trend:** The partial dependence starts with a sharp increase near x3 = 0.0, followed by a slight increase until x3 = 0.6, and then a slight decrease until x3 = 1.0. * **Values:** * At x3 = 0.0, partial dependence is approximately 0.48. * At x3 = 0.6, partial dependence reaches approximately 0.5. * At x3 = 1.0, partial dependence decreases to approximately 0.49. **Plot 4: x4** * **Trend:** The partial dependence increases sharply from x4 = 0.0 to approximately x4 = 0.01, then plateaus around a partial dependence of 0.54 for x4 values between 0.01 and 0.1. * **Values:** * At x4 = 0.0, partial dependence is approximately 0.5. * At x4 = 0.01, partial dependence reaches approximately 0.54. * From x4 = 0.01 to 0.1, partial dependence remains roughly constant at 0.54. **Plot 5: x5** * **Trend:** The partial dependence increases sharply from x5 = 0.00 to approximately x5 = 0.002, then plateaus around a partial dependence of 0.52 for x5 values between 0.002 and 0.02. * **Values:** * At x5 = 0.00, partial dependence is approximately 0.5. * At x5 = 0.002, partial dependence reaches approximately 0.52. * From x5 = 0.002 to 0.02, partial dependence remains roughly constant at 0.52. **Plot 6: x6** * **Trend:** The partial dependence shows a complex pattern with multiple peaks and valleys. It increases sharply near x6 = 0.0, then decreases, increases again, and finally decreases as x6 approaches 0.4. * **Values:** * At x6 = 0.0, partial dependence is approximately 0.5. * At x6 = 0.1, partial dependence reaches approximately 0.53. * At x6 = 0.25, partial dependence peaks at approximately 0.53. * At x6 = 0.4, partial dependence decreases to approximately 0.43. ### Key Observations * Features x1 and x4 exhibit a strong positive impact on the model's prediction up to a certain threshold, after which the effect plateaus. * Feature x2 has a strong negative impact on the model's prediction as its value increases. * Feature x6 has a more complex relationship with the model's prediction, showing non-linear behavior. * The tick marks on the x-axis indicate the distribution of the feature values. Denser tick marks suggest a higher concentration of data points in that region. ### Interpretation The partial dependence plots provide insights into how each feature influences the model's predictions. They help understand the marginal effect of each feature, holding all other features constant. This information can be used for: * **Feature Importance:** Identifying the most influential features in the model. * **Model Understanding:** Gaining insights into the model's behavior and how it makes predictions. * **Feature Engineering:** Guiding feature engineering efforts by identifying non-linear relationships and potential interactions between features. * **Model Debugging:** Identifying potential issues with the model, such as unexpected behavior or biases. For example, the plot for x1 suggests that increasing x1 up to a value of 0.2 significantly increases the model's prediction, but further increases have little effect. This could indicate a saturation effect or a non-linear relationship that the model has learned. Similarly, the plot for x2 suggests that increasing x2 has a negative impact on the model's prediction. The complex behavior of x6 suggests that it may interact with other features or have a non-linear relationship with the target variable. </details> Figure 1: Partial dependence plots for each TrollMagnifier feature. Respectively, x1 = “comments on posts that trolls commented on,” x2 = “comments on posts that trolls started,” x3 = “direct comment in reply to troll post,” x4 = “threaded comment in reply to troll comments on a troll post,” x5 = “same title post as troll,” x6 = “same title post as troll.” Tautological targets. As described in preceding sections (Section 4.3), suspicious account detection suffers from a lack of clear and consistent definitions. Troll accounts cannot be described by degree of automation (while some trolls are bot-like, many others are operated by humans) [cresci_bots_2020] or the nature of the information they spread (this might be political misinformation, ads, or anything in between) [yang2022botometer]; as such, the clearest definition that Saeed et al. implicitly offer is that a troll is an account that exhibits troll-like behavior: i.e., interacts with known troll accounts, or appears on the same posts or message threads as these accounts. The authors note in their own work that this approach cannot be used to detect novel trolling behaviors, and requires a seed list of known troll accounts for every new detection task. Implied versus actual data dimensionality. While the original TrollMagnifier paper strongly implied that the proposed method would leverage networked behaviors to identify troll accounts acting in coordination online, we found that, in actuality, the features under analysis lacked any sort of temporal component and were limited in scope. There were six features in all (described in full in Figure 2), each corresponding to an aggregate engagement statistic. Longitudinal data and timestamps were not available; as such, it was not possible to perform time series analysis. Account names were not available, disallowing construction of user graphs. Feature importance. In our partial dependence analysis, we find that feature x1—commenting statistics—was the sole feature that consistently produced classification accuracy greater than 0.6 (most other features had accuracy no better than random). Per our earlier observation that many account-scoped methods target behaviors that are difficult to distinguish from routine online activity, we recommend that feature engineering for account- and network-scoped methods reflect some intuition about the nature of actually suspicious behaviors. Furthermore, the performance of the current feature set suggests that manual classification might be as effective as (or even more effective than) an automated approach that detects a content-agnostic heuristic. 5.3 Detection of suspicious websites We reproduce results from a study published in 2018 by Baly et al [baly_predicting_nodate]. In summary: the authors propose a multimodal approach to the classification of news websites. This method is particularly representative of works within this information scope: 50% of works within this scope use a similar mixed-modalities approach to detecting misinformation websites, and Baly et al. include site-specific feature types, including domain and traffic-based features, in their analysis. Baly et al. analyzed website contents, associated social media accounts, and Wikipedia pages in order to perform two classification tasks: fact and political bias classification. The authors developed a dataset of 1066 websites manually labeled for their political leaning (extreme-left, left, left-center, center, right-center, right, and extreme-right) and degree of credibility (low, mixed, high). These labels were extracted from the Media Bias/Fact Check (MBFC) database [mediabiasfactheck]. We were able to reproduce original study results and perform ablation analyses on existing datasets. We were unable to run the method on an updated dataset, as feature extraction code was not available. Reproducibility of published results. All features extracted for the original analysis were captured in a series of json files. While we were able to readily reproduce results reported in the paper, certain elements of the dataset (follower counts on social media, Wikipedia page contents) were out of date. As we did not find documentation in the method repository for re-extraction of these features, we were restricted to conducting our tests on data that were already available. We binned bias labels into left, center, and right categories, as the seven-way taxonomy initially applied to the dataset by MBFC yielded small label classes. Classifier performance on the resulting three-way bias classification task accords with the results reported by Baly et al. on the same task. Multimodal features: help or hindrance? We performed an ablation study of the method on the EMNLP18 dataset and analyzed the method’s bias and fact-checking classification functions separately [baly_predicting_nodate]. Specifically, we stratified the original EMNLP18 dataset by political leaning and credibility, as labeled by MBFC, and analyzed the performance of 1) the full feature set, 2) individual features and 3) ablated feature sets (removing one feature type per test). Our results are summarized in Appendix A. We find that, on 11 out of 12 test datasets, classifier performance using only text-based features (articles and wikipedia, derived from articles randomly sampled from the website in question, and the site’s corresponding Wikipedia page, respectively) was comparable to performance on the full feature set. On five out six datasets, bias classification accuracy on text-only features actually outperformed bias classification on the whole feature set (see the bottom half of Appendix A), suggesting that the full-site classifier of Baly et al. was effectively a text content classifier. 6 DISCUSSION We focus our discussion of results on those issues identified by our literature review and investigated in greater depth in our replication analyses. Additionally, we provide recommendations for evaluating ML-driven trust and safety interventions and link these recommendations to major takeaways of the present study. RQ1: Fit. Very few methods that claimed to detect misinformation performed actual fact verification: Instead, they targeted proxy signals that were frequently steps away from promised detection targets. These differences were particularly noticeable in methods that relied heavily on text- and network-based features to perform classification. In those cases, semantic/syntactic signatures and propagation patterns served as proxies for the existence of misinformation. We demonstrated, through our own replication studies, that it is easy to circumvent approaches that rely on style-based cues to perform proxy detection. RQ2: Data curation and model explainability. Lack of access to current, well-annotated datasets remains a serious problem for current and future misinformation research. Across existing datasets, taxonomies for classifying misinformation were inconsistent. Testing on contemporaneous data was uncommon among those papers we annotated, and testing in real-time settings was even rarer. Proof-of-concept experiments oftentimes did not control for data dependencies. Authors describing black-box methods—particularly those employing neural nets or other forms of unsupervised learning—did not disclose feature sets retrieved by their methods. RQ3: Reproducibility. We noted widespread code and data availability issues: Fewer than 30% of our attempts to locate code and data, or obtain this information from authors, were successful. The code and datasets we were able to retrieve were frequently unusable or out of date. In fact, we were able to reproduce and replicate results on published and current data for only one of our replication studies. In that single case, we found that the article-scoped method performed no better than random (0.50 accuracy) on a current, mixed-domain dataset (Section 5.1). RQ4: Generalizability. Methods that were trained on single-source or single-domain datasets appeared to perform well on data from the same source, or within the same domain; these methods, unsurprisingly, performed poorly on multi-source or out-of-domain topics. These discrepancies are closely tied to undisclosed or uncontrolled-for data dependencies, which we discuss in RQ2. 6.1 Recommendations for future research We propose the following as directions for future research: Designing for intervention fit. All article-scoped methods detect proxy targets for misinformation. Some of these proxy targets, including house style and text sentiment, are easily circumvented (Section 5.1). At the point of task formulation, researchers might consider the following: What signals—or proxy signals—will the intervention detect? What are the potential risks of a false negative or false positive result (and are they worth the potential benefits of e.g., greater speed)? What is the actual detection target, and how closely does it approximate the classification task to be performed? In an adversarial setting, would a malicious actor be able to circumvent the automation? Understanding hybrid detection. While academic misinformation detection favors fully automated methods, commercial checking services and online platforms employ hybrid methods. Increasingly, online services are turning toward human-driven hybrid detection approaches, wherein human users make complex fact-checking decisions and automated methods amplify their decision-making power (e.g., by identifying content similar to already-flagged posts). We believe that exploration of the interplay between human and automated decision-making systems will prove a fruitful frontier for academic research. And, in fact, recent work [sehat2024misinformation, lykouris2024learning] frames the hybrid detection problem as one of triage and scheduling: how can automated approaches quickly surface urgent cases for further review by human moderators? Investigating distributional shifts. Fewer than 10% of methods within our whole corpus tested on contemporaneous data, and fewer than 5% tested in a real-time setting. To understand how an ML intervention might work in practice, researchers must understand how robust their methods are to changes in the distribution of available data during training versus testing. These shifts are particular to medium (for instance, whole site classifiers are susceptible to fluctuations in news vertical coverage and author). Notably, the few works in our corpus that tested in real-time reported precipitous performance dropoffs. Addressing nuanced detection challenges. We note that, in general, the detection methods we consider in this work elide subtleties that are particular to the medium under consideration: for instance, article-level detection methods generally apply broad ‘true’/‘false’ classifications to a text under analysis where only single sentences might be slightly inaccurate. Additionally, the language signals that most methods expressly detect are fairly unsubtle, and are likely verifiable via manual means. Suggestion, insinuation, and leading questions are powerful rhetorical tools that might render a newsreader more susceptible to actual misinformation, or that might suggest misinformative ideas via indirect means; no works within our literature corpus expressly targeted these forms of language, however. 7 Ethics considerations Our author outreach survey design was approved by the Princeton University IRB. All statistics published from those communications are in aggregate, with no personal identifiers attached to individual author responses. Datasets used for this work were already publicly available or were obtained with permission from study authors. 8 Open science Code and data used for our analyses, as well as our full paper corpus, are included in our online SI (available at https://anonymous.4open.science/r/sok_misinformation-41E8). Acknowledgments The authors would like to thank Anne Kohlbrenner for assisting with paper coding; Ben Kaiser and Sarah Scheffler for productive conversations, and for reviewing multiple drafts of this manuscript; and Mona Wang for retrieving data used for a replication study. \printbibliography Appendix Appendix A Commercial fact-checking services We include here a brief market survey of commercial and LLM-powered fact-checking and IO detection services. In general, these services fall into five categories: 1) media fact-checking organizations; 2) brand safety and suitability services; 3) trust & safety operations at large social media platforms, 4) threat detection operations, and 5) analytics organizations unaffiliated with a media outlet that offer research capacity to governments and businesses. We define each service category and (with the exception of the first category, which comprises human media workers and fact-checkers) discuss automated content moderation operations deployed by three prominent exemplars within each service category. In general, in instances where such information is made available, we observe that at-scale content moderation businesses at least employ human-labeled datasets to train classifiers, and some retain subject-area experts to adjudicate complex moderation decisions. On social media platforms, in particular, human moderators and automated systems appear to work hand-in-hand: automated systems surface potentially misinformative content that receives final verification from a human moderator. For IO detection, specialized knowledge (pertaining to specific geographies, languages, or political climates) is often invoked. Media fact-checking. Human fact-checkers and content moderators affiliated with news outlets, or who work as freelance fact-checkers. The International Fact-Checking Network (IFCN) is a professional network of media workers and fact-checkers; IFCN is also the de facto standards setting body for media fact-checking, and maintains a fact-checking code of ethics [ifcn_code]. In general, media fact-checking organizations with IFCN affiliations are established news organizations, non-profits, and watchdog organizations that employ human journalists and fact-checkers. Furthermore, (human) fact-checkers can receive IFCN compliance certificates after passing a qualifying exam. Brand safety and suitability companies. B2B companies that detect categories of potentially harmful speech on websites where ads might appear. Advertisers wishing to protect “brand safety” contract with these services to ensure that their ads do not appear alongside problematic content. The Global Alliance for Responsible Media (GARM) is the standards-setting body for brand safety and suitability companies [garm_standards]. - Zefr, a GARM member company, deploys AI to detect material that falls within predefined subcategories of problematic content (e.g., explicit content, misinformation, spam). In a press release for Zefr’s acquisition of an AI-driven content moderation company (AdVerif.ai) from 2022, the company disclosed that AdVerif.ai is “powered by fact-checking data from more than 50 IFCN-certified organizations around the globe” [Zefr_2022] —that is, AdVerif.ai trains its models on labeled datasets produced by (human) IFCN affiliates. - DoubleVerify, a GARM member company, “uses sophisticated approaches that rely on a combination of AI and comprehensive human review” [Pasquine_2022]. According to the company’s documentation, human assessors (a “semantic science team”) evaluate site infrastructure and contents; AI is used to scale their assessments. - Integral Ad Science (IAS), a GARM member company, deploys AI to detect low-quality sites via infrastructure features. The company’s data sources, and deployment methodology were not immediately evident upon web search; IAS recently announced a new partnership with Meta for ad placement management on Facebook [Integral_Ad_Science_2024]. Trust & safety operations. In-house content moderation teams at large social media platforms. - Twitter has deployed a crowd-sourced annotations platform called Community Notes (formerly Birdwatch) since 2021 [wojcik2022birdwatch]. - Until recently, Facebook partnered with IFCN affiliates to perform third-party manual checking of possibly misinformative content; first-line automated methods detect potentially harmful speech and surface near-duplicates of known problematic image (SimSearchNet++) and text content [facebook_files_2021, fb_third_party, AI_at_Meta_2020]. Meta announced in January 2025 that it was sunsetting its third-party checking program in favor of a Community Notes-like system [chan_meta_2025]. - TikTok employs thousands of content moderators across the globe who “work alongside automated moderation systems” [McIntyre_Bradbury_Perrigo_2023, Farah_2023]. Threat intelligence services. At-scale detection of advanced persistent threats, foreign influence operations, and other cyberattacks oftentimes perpetrated by nation state actors. - Mandiant strongly implies the use of hybrid detection methods, and disclaims that “defenders must constantly explore different techniques and leverage both subject matter expertise and technical capabilities to filter and uncover malicious activity”) [Kapellmann_Zafra_Serabian_Riddell_Brubaker_2022]. - Microsoft Threat Intelligence strongly implies the use of hybrid detection methods; in a report from September 2023, MTI cites the work of in-house “Microsoft Security teams” which are tracking an advanced social engineering attack [Microsoft_Incident_Response_2023]. Other details—including possible use of automated methods—are undisclosed. - Facebook Coordinated Inauthentic Behavior reports share quarterly updates about Meta’s takedown of coordinated activities across its platforms and others, including local news outlets. In a report from February 2023, Meta describes a CIB network in Serbia that used local news media to create the impression of grassroots support for the Serbian Progressive Party; while the nature of the detection methodology is unspecified, the complexity and geographic specificity of the CIB described suggest that specialists with country-level expertise were likely consulted [Nimmo_2023a]. Analytics firms. For- and non-profit organizations that offer checking services and research capacity to governments and businesses. - The Global Disinformation Index (GDI) “reviews news domains based on various metadata and computational signals.” Content, however, is manually reviewed by a “country expert,” who analyzes a random sample of 10 articles from a news site to determine veracity [Srinivasan_GDI]. - DFRLabs (Digital Forensic Research Lab) has disclosed that it employs human subject-area experts, and primarily addresses technology and policy issues pertaining to global and international affairs. In 2018, Facebook contracted its services to detect online trolls [Lapowsky_2018]. - Graphika Labs leverages network analysis to identify influence operations online. On its own website and in the popular press, Graphika has disclosed that it uses AI to map online networks and trace information flows [Schiffrin_2023, graphika_ai]. LLM-driven detection. A few LLM-powered detection methods have been discussed in the popular press, including those advertised by Google [Storey_2023] and OpenAI [fried_2023], but these deployments appear to be mostly experimental, or have required additional adjudication from human moderators. OpenAI in particular has advertised content moderation tools that address misinformation-adjacent tasks, such as toxic speech detection [perspective_openai]. Misinformation and toxic speech detection are not equivalent tasks, however, and the latter is narrowly defined in the Perspective training data documentation as a four-way classification task (the four class labels are “profanity/obscenity,” “identity-based negativity,” “insults,” and “threatening” language). | 1. Ajao et al. [ajao_sentiment_2019] | C A | Sentiment (A.i) | PHEME [pheme_2018] | LSTM, DT, RF, SVM | | | | | 0.86 (Acc.) | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | 2. Abulldah-All-Tanvir et al. [abdullah-all-tanvir_detecting_2019] | C A N | Content (C.i) | Twitter (API) | NB, RNN, LSTM, SVM, Logit | | | | | 0.89 (Acc.) | | 3. Bhutani et al. [bhutani_fake_2019] | C A | Content (C.i); sentiment (A.i) | Twitter (API), PolitiFact [politifact] | Naive Bayes, RF | | | | | 0.60 (AUC) | | 4. Bozarth et al. [bozarth-truth-2020] | C | Contents (C.i) | PolitiFact [politifact], Daily Dot, Zimdars, MBFC | LDA | | | | | n/a | | 5. Ciampaglia et al. [ciampaglia-wiki-2015] | C N | Shortest path search (C.ii) | DBpedia | kNN, RF | | | | | 0.97 (AUC) | | 6. Cui et al. [cui_same_2019] | C A U | Content (C.i); sentiment (A.i); user response (U.ii) | PolitiFact [politifact], GossipCop [gossipcop] | KNN, SVM, CSI [ruchansky_csi_2017], RMSprop | | | | | 0.82 (F1) | | 7. Debnath et al. [debnath_hierarchical_2020] | C | Content (C.i) | LIAR [wang_liar_2017] | CNN | | | | | 0.27 (Acc.) | | 8. Dey et al. [dey_fake_2018] | C A | Content (C.i); sentiment (A.i) | Twitter (API) | Clustering (kNN) | | | | | 0.67 (Acc.) | | 9. Galitsky et al. [galitsky_detecting_2015] | C | Content (C.i) | Amazon reviews | Parse thicket | | | | | 0.81 (Prec.) | | 10. Glockner et al. [glockner_missing_2022] | C | Content (C.i) | PolitiFact [politifact], Snopes [snopes], MultiFC | CNN, DNN | | | | | 0.58 (Acc.) | | 11. Gordon et al. [gordon_disagreement_2021] | C A | Content (C.i); source rep (U.i) | Credibility-Factors2020 | SVD | | | | | 0.63 (Acc.) | | 12. Gupta et al. [gupta_tweetcred_2015] | C A N | Content (C.i); stance (A.iii) | Twitter (API) | SVM | | | | | 0.60 (Agreement) | | 13. Hassan et al. [hassan_data_2014] | C | Content (C.i); checkability (C.iii) | NBA, weather datasets | Frequency | | | | | n/a | | 14. Jain et al. [jain_towards_2016] | C | Content (C.i); sentiment (A.i) | Twitter (API) | Gensim/TextBlob | | | | | 0.77 (Acc.) | | 15. Jiang et al. [jiang_linguistic_2018] | C | Content (C.i); ling./syntax (C.ii) | PolitiFact [politifact], Snopes [snopes] | SVM | | | | | 0.81 (Acc.) | | 16. Karimi et al. [karimi_multi-source_2018] | C | Content (C.i) | LIAR [wang_liar_2017] | LSTM, CNN | | | | | 0.39 (Acc.) | | 17. Kartal et al. [kartal_too_2020] | C | Content (C.i); checkability (C.iii) | Check That! dataset | Logit, SVM, RF | | | | | 0.26 (MAP) | | 18. Kou et al. [kou_hc-covid_2022] | C | Content (C.i) | CoAID, CONSTRAINT | Knowledge graph | | | | | 0.90 (Acc.) | | 19. Paudel et al. [paudel_lambretta_2023] | N A | Keyword detection (A.ii) | Abilov et al. dataset, Twitter (API) | AdaRank, ListNet, RF | | | | | 0.79 (MAP) | | 20. Popat et al. [popat_credibility_nodate] | C | Content (C.i) | PolitiFact [politifact], Snopes [snopes], NewsTrust | biLSTM, CNN | | | | | 0.88 (AUC) | | 21. Shiralkar et al. [shiralkar_finding_2017] | C N | KG search (C.ii) | DBpedia | Knowledge graph | | | | | 1.00 (AUC) | | 22. Shu et al. [shu_defend_2019] | C U | Word encoding (C.i); user response (U.ii) | GossipCop [gossipcop], PolitiFact [politifact] | RNN/RMSprop, CSI [ruchansky_csi_2017], LSTM, CNN | | | | | 0.93 (F1) | | 23. Tian et al. [tian_early_2020] | C U | Content (C.i); user response (U.ii) | Twitter15, Twitter16 | CNN-biLSTM | | | | | 0.82 (F1) | | 24. Zhang et al. [zhang_reply-aided_2019] | C U | Content (C.i); user response (U.ii) | RumourEval, PHEME [pheme_2018] | biLSTM, Multitask, SVM, CNN, | | | | | 0.89 (Acc.) | | 1. Afroz et al. [afroz_detecting_2012] | A | Content (C.i); syntax (A.i) | Brennan-Greenstadt | SVM, J48 Decision Trees | | | | | 0.97 (F1) | | 2. Ahmed et al. [ahmed_detecting_2018] | A | Syntax (A.i) | Twitter, Kaggle, Horne and Adali [horne_this_2017] | SVM | | | | | 0.92 (Acc.) | | 3. Bourgonje et al. [bourgonje_clickbait_2017] | A | Stance (A.ii) | Fake News Challenge Data | Logit | | | | | 0.90 (Acc.) | | 4. Brasoveanu et al. [brasoveanu_semantic_2019] | A C | Sentiment (A.i); keywords (A.ii) | LIAR [wang_liar_2017] | CNN, LSTM, CN | | | | | 0.64 (Acc.) | | 5. Della Vedova et al. [della_vedova_automatic_2018] | A N | Content (C.i); virality (N.iv) | FakeNewsNet, Buzzfeed | Logit | | | | | 0.82 (Acc.) | | 6. Horne et al. [horne_this_2017] | A | Syntax (A.i), headline (A.ii) | Buzzfeed, Burfoot & Baldwin | SVM | | | | | 0.78 (Acc.) | | 7. Jabiyev et al. [jabiyev_fade_2021] | A W | Topic detection (A.ii); site cred. (W.iv) | Snopes [snopes], FactCheck, PolitiFact [politifact] | SVM, DT, RF | | | | | 0.87 (Acc.) | | 8. Jadhav et al. [jadhav_fake_2019] | A | Content (C.i); syntax (A.i) | LIAR [wang_liar_2017] | DSSM/RNN | | | | | 0.99 (Acc.) | | 9. Jin et al. [jin_detection_2017] | A N U | (A.ii); (N.i); suspicious accounts (U.i) | Tweets; articles | n/a | | | | | 0.87 (Prec.) | | 10. Kapusta et al. [kapusta_fake_2020] | A | Sentiment & word freq. (A.i) | MBFC and custom | n/a | | | | | n/a | | 11. Kumar et al. [kumar_disinformation_2016] | A W U | (A.i); UI (W.iii); Author cred. (U.i) | 20K Wiki Hoaxes | Random forest | | | | | 0.87 (AUC) | | 12. Magdy et al. [magdy-stat-2010] | A | Content (C.i) | NYT Corpus [nyt_corpus], 100 Wikis | Pattern recog. | | | | | 0.99 (Recall) | | 13. Monti et al. [monti2019fake] | A N U | (A.ii); (N.i); suspicious accounts (U.i) | Tweets; articles | RNN/CNN | | | | | 0.927 (AUC) | | 14. Nasir et al. [nasir_fake_2021] | A | Syntax (A.i) | ISOT [isot]; FAKES [fa_kes] | RNN/CNN | | | | | 0.99 (Acc.) | | 15. Perez-Rosas et al. [perez-rosas_automatic_2017] | A | Syntax (A.i) | FakeNewsAMT; Celebrity | SVM | | | | | 0.74 (Acc.) | | 16. Potthast et al. [potthast-etal-2018-stylometric] | A | Syntax, sentiment, readability (A.i) | Buzzfeed-Webis | Bag-of-words | | | | | 0.46 (F1) | | 17. Reis et al. [reis_explainable_2019] | A | Syntax (A.i); contents (C.i); source rep. (U.i); timing (N.ii); URL (W.i) | BuzzFeed | GBM | | | | | 0.85 (AUC) | | 18. Rubin et al. [rubin_towards_nodate] | A | Syntax (A.i) | AMT | Clustering | | | | | 0.67 (Agreement) | | 19. Ruchansky et al. [ruchansky_csi_2017] | A U | (A.i); Responses (A.ii); acct metadata (U.i) | Twitter/Weibo posts | RNN/LSTM | | | | | 0.95 (Acc.) | | 20. Santos et al. [santos_readability_2020] | A | Readability (A.i) | Fake.Br corpus | SVM | | | | | 0.92 (Acc.) | | 21. Silva et al. [silva_embracing_2021] | A N | Topic detection (A.ii); propagation (N.i) | PolitiFact [politifact], GossipCop [gossipcop], CoAID | Clustering | | | | | 0.88 (Acc.) | | 21. Singh et al. [singh_automated_nodate] | A | Syntax (A.i) | Kaggle Fake News | SVM | | | | | 0.87 (Acc.) | | 22. Uppal et al. [uppal_fake_2020] | A | Discourse structure (A.i) | Buzzfeed, PolitiFact [politifact] | GRU, Dependency tree | | | | | 0.74 (Acc.) | | 1. Cao et al. [cao_aiding_2012] | U N | Acct. cred. (U.i); prop. (N.i) | Tuenti social network | Louvain clustering | | | | | 0.90+ (TP) | | 2. Danezis et al. [danezis2009sybilinfer] | U N | Acct. cred. (U.i); prop. (N.i) | LiveJournal data | Bayesian inf. | | | | | n/a* | | 3. Ezzeddine et al. [ezzeddine2023exposing] | U | Acct. behaviors (U.ii) | DATA | LSTM | | | | | 0.91 (AUC) | | 4. Hamdi et al. [hamdi_hybrid_2020] | U N | Account metadata (U.i); prop. (N.i) | CREDBANK | LDA, Bayes, Logit, SVM | | | | | 0.99 (AUC) | | 5. Helmstetter et al. [helmstetter-supervised-2018] | U N A | Acct metadata (U.i); post sharing data (U.ii) | Public site cred. lists | SVM, NB, DT, RF | | | | | 0.936 (F1) | | 6. Jain et al. [jain_towards_2016] | U A | Acct metadata (U.i); topic det. (A.ii) | Twitter (API) | Gensim, TextBlob | | | | | 0.77 (Acc.) | | 7. Leonardi et al. [leonardi_users_2021] | U N A | Acct metadata (U.i); prop. (N.i) | CoAID | RF | | | | | 0.81 (F1) | | 8. Saeed et al. [saeed2022trollmagnifier] | U | User behavior (U.ii) | Reddit Pushshift; Reddit IRA trolls list | RF | | | | | 0.98 (Acc.) | | 9. Sansonetti et al. [sansonetti_unreliable_2020] | U C | Acct metadata (U.i); acct activity (U.ii) | PolitiFact [politifact], Twitter (API) | LSTM-CNN, SVM, KNN | | | | | 0.92 (Acc.) | | 10. Santia et al. [santia_bots_2019] | U A | Source rep. (U.i); user response (U.ii); syntax (A.i) | BuzzFeed | SVM, RF, DT, NB | | | | | 0.77 (Prec.) | | 11. Shu et al. [shu_exploiting_2017] | U N A | User behavior (U.ii); prop. (N.i) syntax (A.i) | Buzzfeed, PolitiFact | Gibbs sampling | | | | | 0.85+ (Acc.) | | 12. Vargas et al. [vargas2020detection] | N A | Prop. (N.i); topic det. (A.ii) | Twitter (API) | RF | | | | | 0.98 (F1) | | 13. Wang et al. [wang_sybil_2013] | U N | User behavior (U.ii) | Renren data | SVM | | | | | 0.99 (Acc.) | | 14. Yu et al. [yu_sybillimit_2008] | U N | User behavior (U.i); prop. (N.i) | LiveJournal, Friendster, DBLP accounts | Random route | | | | | n/a* | | 15. Yuan et al. [yuan2019detecting] | U N | Acct metadata (U.i); timing (N.ii) | WeChat data | Clustering | | | | | 0.90+ (Prec.) | | 16. Zhang et al. [zhang_misinformation_2016] | U N | User behavior (U.ii); prop. (N.i) | Twitter, Slashdot, Epinion | Graph cut | | | | | n/a | | 17. Zhou et al. [zhou_network-based_2019] | U N | User suscept. (U.i); prop (N.i) | PolitiFact [politifact], BuzzFeed | SVM, KNN, NB, DT, RF | | | | | 0.93 (Acc.) | | 1. Alizadeh et al. [alizadeh2020content] | N A U | Propagation (N.i); syntax (A.i); acct metadata (U.i) | Twitter (API), Reddit IRA troll list | RF | | | | | 0.70+ (F1) | | 2. Antoniadis et al. [antoniadis_model_2015] | U A | Acct metadata (U.i); syntax (A.i) | Hurricane Sandy tweet dataset | J48, RF, KNN, Bayes | | | | | 0.79 (Avg. Prec.) | | 3. Assenmacher et al. [assenmacher2020two] | N A | Propagation (N.i); topic det. (A.ii) | Twitter (API) | Clustering | | | | | not reported | | 4. Buntain et al. [buntain_automatically_2017] | N U A | Time (N.ii); acct metadata (U.i); sentiment (A.i) | CREDBANK, Buzzfeed | RF | | | | | 0.65 (Acc.) | | 5. Castillo et al. [castillo_information_2011] | U A | Syntax (A.i); user behavior (U.ii) | Twitter Monitor events | SVM, DT | | | | | 0.874 (P) | | 6. Chen et al. [chen_rumor_2018] | N U A | Social graph (N.iii); syntax (A.i); user behavior (U.ii) | Weibo | RNN | | | | | 0.92 (Acc.) | | 7. Guo et al. [guo_rumor_2018] | N C | Prop. (N.i); semantics (C.i) | Twitter, Weibo | LSTM | | | | | 0.9 (Acc.) | | 8. Jin et al. [jin_news_2016] | N A | Propagation (N.i); stance (A.iii) | Sina Weibo posts | Clustering | | | | | 0.84 (Acc.) | | 9. Liu et al. [liu_early_nodate] | N | Propagation (N.i) | Weibo, Twitter15, Twitter16 | RNN, CNN | | | | | 0.897 (Acc.) | | 10. Ma et al. [ma_detect_2017] | N | Propagation (N.i) | Kochina, Ma, Shu Twitter datasets | RNN/BiLSTM | | | | | 0.75 (Acc.) | | 11. Magelinski et al. [magelinski_synch_2022] | N U | Prop. (N.i); timing (N.ii); user behavior (U.ii) | Twitter (API) | – | | | | | – n/a | | 12. Nguyen et al. [nguyen_fake_2019] | N A | Prop. (N.i); semantics (A.i) | Twitter, Weibo, PHEME [pheme_2018] | SVM, RNN, DT | | | | | 0.970 (Acc.) | | 13. Pacheco et al. [pacheco_uncovering_2020] | U N | Account metadata (U.i); timing (N.i) | Twitter (API) | Clustering | | | | | 0.8+ (Prec.) | | 14. Ratkiewicz et al. [ratkiewicz2011detecting] | N A U | Prop. (N.i); keywords (A.ii); user behavior (U.ii) | Twitter (API) | AdaBoost, SVM | | | | | 0.96 (Acc.) | | 15. Sharma et al. [sharma_identifying_2021] | N A U | Prop. (N.i); keywords (A.ii); user behavior (U.ii) | Twitter (API); Twitter IRA trolls list | GMM, Kmeans, NN | | | | | 0.94 (AUC) | | 16. Tschiatschek et al. [tschiatschek_fake_2018] | N U | Prop. (N.i); user rep. (U.i) | Facebook dataset | Bayes inf. | | | | | n/a | | 17. Zeng et al. [zeng_unconfirmed_2016] | N A | Prop. (N.i); stance (A.iii) | 4.3K tweets (from API) | Logit, NB, RF | | | | | 0.88 (Acc.) | | 1. Asr et al. [torabi_asr_data_2018] | W A | Source rep. (W.i); Syntax (A.i) | BuzzfeedUSE, Snopes, Rashkin, Rubin | CNN, SVM, NB | | | | | not reported | | 2. Baly et al. [baly_predicting_nodate] | W A N | Source rep. (W.i); Site infra. (W.ii); (A.i) | MediaBiasFactCheck [mediabiasfactheck] | SVM | | | | | 0.66 (Acc.) | | 3. Baly et al. [baly-etal-2020-written] | W A N | Source rep. (W.i); Site infra. (W.ii); (A.i) | MediaBiasFactCheck [mediabiasfactheck] | SVM | | | | | 0.7152 (Acc.) | | 4. Castelo et al. [castelo_agnostic_2019] | W A | Site infra. (W.ii); syntax (A.i) | Celebrity, US-Election2016 | SVM, kNN, RF | | | | | 0.86 (Acc.) | | 5. Chen et al. [chen_proactive_2020] | W A | Hosting infra. (URL) (W.ii); syntax (A.i) | PoliticalFakeNews | Clustering | | | | | 0.97 (AUC) | | 6. Hounsel et al. [hounsel-infra-2020] | W | Site infra. (W.i) | FactCheck, Snopes, PolitiFact, Buzzfeed | RF | | | | | 0.98 (AUC) | | 7. Ribeiro et al. [ribeiro_media_2018] | W U | Site infra. (W.i); User demog. (U.i) | Facebook API | Graph search | | | | | 0.97 (PCC) | Table 4: Focus corpus by scope and target. Coding of a focus set of 87 papers, sorted by information scope. Values in parentheses in “Target” field correspond to highlighted subcategories presented in Section 4 (e.g., “C.i” denotes target (i) in “Claims,” Section 4.1). If authors present evaluation results for multiple models, we underline the most performant model and record its corresponding performance score. $$ - + \tag{1066} $$ Table 5: Replication analysis of Baly et al.: Dropout(–) and feature importance(+) analyses of subsets of Baly et al.’s EMNLP18 dataset, stratified by political leaning and credibility. Most (secondmost) performant feature, as determined by its contribution to overall classifier accuracy on the full feature set, is highlighted in darker (lighter) hues. Fact and bias classification task performances are reported in the top and bottom halves of the table, respectively.