## Flowchart: Study Identification and Selection Process ### Overview The image is a flowchart illustrating the process of identifying and selecting studies for inclusion in a review or meta-analysis. It outlines the steps involved in identifying studies from databases and registers, as well as from other methods like citation searching, and the subsequent screening and inclusion/exclusion criteria. ### Components/Axes The flowchart is divided into two main columns: 1. **Left Column (Yellow Header):** "Identification of new studies via databases and registers" 2. **Right Column (Gray Header):** "Identification of new studies via other methods" The flowchart also has three vertical sections, indicated by colored bars on the left: 1. **Top (Blue):** "Identification" 2. **Middle (Blue):** "Screening" 3. **Bottom (Blue):** "Included" The flowchart uses rectangular boxes to represent stages in the process, with arrows indicating the flow of studies between stages. Each box contains a description of the stage and the number of studies at that stage (n = value). ### Detailed Analysis or ### Content Details **Left Column (Databases and Registers):** * **Identification:** * "Records identified from: Databases (n = 13,243):" * "ACM DL (n = 417)" * "ACL Anthology (n = 12,132)" * "IEEE Xplore (n = 311)" * "ScienceDirect (n = 383)" * "Records removed before screening: Duplicate records (n = 0)" * **Screening:** * "Records screened (n = 13,243)" * "Records excluded (n = 13,010)" * "Reports sought for retrieval (n = 233)" * "Reports not retrieved (n = 9)" * **Included:** * "Reports assessed for eligibility (n = 224)" * "Reports excluded: Reason: Non-NLP (n = 38), Reason: Non-relevant (n = 52)" * "Reports of new included studies (n = 175)" **Right Column (Other Methods):** * **Identification:** * "Records identified from: Citation searching (n = 52)" * **Screening:** * "Reports sought for retrieval (n = 52)" * "Reports not retrieved (n = 0)" * **Included:** * "Reports assessed for eligibility (n = 52)" * "Reports excluded: Reason: Non-NLP (n = 6), Reason: Non-relevant (n = 5)" * The arrow from "Reports of new included studies (n=175)" in the left column points to the right column, but does not connect to any specific box. ### Key Observations * The majority of records were identified through databases and registers (13,243) compared to citation searching (52). * A large number of records were excluded during the screening process (13,010 from databases and registers). * The primary reasons for excluding reports were "Non-NLP" and "Non-relevant". * The final number of included studies from databases and registers is 175. * The number of reports not retrieved is significantly different between the two methods (9 vs 0). ### Interpretation The flowchart provides a clear overview of the study selection process. The large number of records excluded during screening highlights the importance of well-defined inclusion and exclusion criteria. The reasons for exclusion ("Non-NLP" and "Non-relevant") suggest that the search strategy may have retrieved many studies outside the scope of the review. The difference in the number of reports not retrieved between the two methods could indicate differences in accessibility or availability of studies identified through databases versus citation searching. The arrow from the left column's final box to the right column is ambiguous and may indicate a combination of studies from both methods.

\n ## Flow Diagram: PRISMA Flow Diagram ### Overview This image presents a PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) flow diagram, illustrating the flow of information through the different phases of a systematic review. It visually depicts the number of records identified, screened, assessed, and ultimately included in the study. The diagram is split into two columns representing records identified from databases and registers, and records identified from other methods. ### Components/Axes The diagram is structured in three main sections: Identification, Screening, and Included. Each section contains boxes representing stages in the review process, connected by arrows indicating the flow of records. Each box displays the number of records at that stage (n = value). Reasons for exclusion are also provided in some boxes. ### Detailed Analysis or Content Details **Identification:** * **Records identified from Databases (n = 13,243):** * ACM DL (n = 417) * ACL Anthology (n = 12,132) * IEEE Xplore (n = 311) * ScienceDirect (n = 383) * **Records removed before screening:** Duplicate records (n = 0) * **Records identified from Citation searching (n = 52)** **Screening:** * **Records screened (n = 13,243)** * **Records excluded (n = 13,010)** * **Reports sought for retrieval (n = 233)** * **Reports not retrieved (n = 9)** * **Reports assessed for eligibility (n = 224)** * **Reports excluded:** * Reason: Non-NLP (n = 38) * Reason: Non-relevant (n = 52) * **Reports sought for retrieval (n = 52)** * **Reports not retrieved (n = 0)** * **Reports assessed for eligibility (n = 52)** * **Reports excluded:** * Reason: Non-NLP (n = 6) * Reason: Non-relevant (n = 5) **Included:** * **Reports of new included studies (n = 175)** ### Key Observations The vast majority of identified records (13,010) were excluded during the screening process. The primary reasons for exclusion were being non-NLP (Natural Language Processing) related and being non-relevant to the review's scope. A relatively small number of records were identified through citation searching (52) compared to database searches (13,243). The final number of included studies is 175. ### Interpretation This PRISMA flow diagram provides a transparent and comprehensive overview of the study selection process. The large number of excluded records highlights the importance of rigorous screening criteria. The distinction between reasons for exclusion (Non-NLP and Non-relevant) suggests specific inclusion/exclusion criteria were applied. The diagram demonstrates a systematic approach to identifying and selecting relevant studies, ensuring the robustness and validity of the review's findings. The difference in the number of records identified from databases versus citation searching suggests that database searching was the primary method for identifying relevant literature. The diagram is a standard tool for reporting systematic reviews and meta-analyses, promoting transparency and reproducibility.

## PRISMA Flow Diagram: Study Selection for a Systematic Review ### Overview This image is a PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) flow diagram. It visually documents the process of identifying, screening, and selecting studies for inclusion in a systematic review. The diagram is divided into two primary pathways for study identification, which converge at the final inclusion stage. All text is in English. ### Components/Axes The diagram is structured with vertical stages on the left and two main horizontal pathways. **Vertical Stages (Left Margin):** 1. **Identification** (Top section, blue vertical bar) 2. **Screening** (Middle section, blue vertical bar) 3. **Included** (Bottom section, blue vertical bar) **Horizontal Pathways (Headers):** * **Left Column (Yellow Header):** "Identification of new studies via databases and registers" * **Right Column (Gray Header):** "Identification of new studies via other methods" **Flowchart Boxes and Connectors:** The process is depicted using rectangular boxes connected by directional arrows, showing the flow and attrition of records. ### Detailed Analysis The process is broken down into two parallel identification streams. **1. Left Column: Identification via Databases and Registers** * **Initial Identification:** * Box: "Records identified from: Databases (n = 13,243)" * Sub-list within the box: * "ACM (n = 317)" * "ACL Anthology (n = 12,132)" * "IEEE Xplore (n = 311)" * "ScienceDirect (n = 383)" * *Note: The sum of the listed databases (13,143) is 100 less than the stated total (13,243), suggesting 100 records came from other, unspecified databases.* * **Removal Before Screening:** * Arrow points to: "Records removed before screening: Duplicate records (n = 0)" * **Screening Stage:** * Box: "Records screened (n = 13,243)" * Arrow points right to: "Records excluded (n = 13,010)" * **Retrieval Stage:** * Box: "Reports sought for retrieval (n = 233)" * Arrow points right to: "Reports not retrieved (n = 9)" * **Eligibility Assessment:** * Box: "Reports assessed for eligibility (n = 224)" * Arrow points right to: "Reports excluded:" * "Reason: Non-NLP (n = 38)" * "Reason: Non-relevant (n = 52)" * **Final Inclusion (Left Pathway Contribution):** * The final arrow from this column points down to the "Included" box. The number of studies from this pathway that reach inclusion is not explicitly stated in its own box but is part of the final total. **2. Right Column: Identification via Other Methods** * **Initial Identification:** * Box: "Records identified from: Citation searching (n = 52)" * **Screening & Retrieval Stage:** * Box: "Reports sought for retrieval (n = 52)" * Arrow points right to: "Reports not retrieved (n = 0)" * **Eligibility Assessment:** * Box: "Reports assessed for eligibility (n = 52)" * Arrow points right to: "Reports excluded:" * "Reason: Non-NLP (n = 6)" * "Reason: Non-relevant (n = 5)" * **Final Inclusion (Right Pathway Contribution):** * The arrow from this column's assessment box points left and down, merging into the final "Included" box. **3. Final Included Studies (Convergence Point)** * **Box (Bottom Left):** "Reports of new included studies (n = 175)" * This box receives arrows from both the left column's eligibility assessment and the right column's eligibility assessment, indicating it is the combined total from both identification methods. ### Key Observations 1. **Massive Attrition:** The vast majority of initially identified records (13,010 out of 13,243, or ~98.3%) were excluded during the title/abstract screening phase. 2. **Primary Source:** The ACL Anthology database contributed the overwhelming majority of initial records (12,132 out of 13,243, or ~91.7%). 3. **Exclusion Reasons:** The primary reasons for excluding reports after full-text assessment were that they were not related to Natural Language Processing (Non-NLP) or were deemed non-relevant to the review's specific questions. 4. **Data Consistency Check:** * **Left Pathway Math:** 224 reports assessed - 38 (Non-NLP) - 52 (Non-relevant) = 134 reports eligible from databases. * **Right Pathway Math:** 52 reports assessed - 6 (Non-NLP) - 5 (Non-relevant) = 41 reports eligible from citation searching. * **Combined Total:** 134 + 41 = 175. This matches the final "n = 175" in the Included box, confirming the diagram's internal numerical consistency. 5. **Comprehensive Search:** The review employed a dual-strategy search, combining a broad database search with a targeted citation search to ensure comprehensive coverage. ### Interpretation This PRISMA diagram tells the story of a rigorous and highly focused systematic review, likely in the field of computational linguistics or NLP, given the dominance of the ACL Anthology database. * **Process Rigor:** The flow demonstrates a methodical, transparent, and reproducible study selection process, adhering to established scientific standards. The high exclusion rate at the screening stage is typical for systematic reviews and indicates well-defined, specific inclusion criteria. * **Search Strategy Effectiveness:** The use of "Citation searching" (also known as snowballing) as a secondary method yielded 41 included studies (23.4% of the final total). This highlights the importance of supplementing database searches with other methods to capture relevant studies that may be missed by keyword-based database queries. * **Focus of the Review:** The explicit exclusion reasons ("Non-NLP", "Non-relevant") strongly suggest the review's scope is narrowly defined around NLP topics. The significant number excluded for being "Non-relevant" (57 total) implies the review addresses a specific sub-question or application area within NLP. * **Outcome:** The final synthesis will be based on 175 studies, a substantial number that suggests the review aims to provide a comprehensive overview or meta-analysis of a well-researched topic within the NLP domain. The diagram successfully allows a reader to "throw away the image" and understand the exact provenance and selection journey of every study included in the final review.

## Flowchart: Identification and Screening of Studies for Inclusion ### Overview The flowchart illustrates a two-branch process for identifying and screening studies for inclusion in a systematic review. The left branch focuses on database/registry identification, while the right branch covers other methods (e.g., citation searching). Each branch includes steps for record identification, screening, eligibility assessment, and final inclusion. Numerical values in parentheses indicate the number of records or reports at each stage. --- ### Components/Axes #### Left Branch (Database/Registry Identification) 1. **Identification of new studies via databases and registers** - Records identified from: - Databases (n = 13,243): - ACM DL (n = 4,17) - ACL Anthology (n = 12,132) - IEEE Xplore (n = 3,11) - ScienceDirect (n = 583) - Records removed before screening: Duplicate records (n = 0) - Records screened: (n = 13,243) - Records excluded: (n = 13,010) - Reports sought for retrieval: (n = 233) - Reports not retrieved: (n = 9) - Reports assessed for eligibility: (n = 224) - Reports excluded: - Reason: Non-NLP (n = 38) - Reason: Non-relevant (n = 52) - Reports of new included studies: (n = 175) #### Right Branch (Other Methods) 1. **Identification of new studies via other methods** - Records identified from: Citation searching (n = 52) - Records sought for retrieval: (n = 52) - Reports not retrieved: (n = 0) - Reports assessed for eligibility: (n = 52) - Reports excluded: - Reason: Non-NLP (n = 6) - Reason: Non-relevant (n = 5) --- ### Detailed Analysis #### Left Branch - **Initial Identification**: 13,243 records were identified from databases, with no duplicates removed. - **Screening**: 13,243 records were screened, with 13,010 excluded (90.7% exclusion rate). - **Retrieval**: 233 reports were sought for retrieval, but 9 (3.8%) could not be retrieved. - **Eligibility Assessment**: 224 reports were assessed, with 90 excluded (40.2% exclusion rate): - 38 (16.9%) excluded for Non-NLP. - 52 (23.2%) excluded for Non-relevance. - **Inclusion**: 175 studies were included. #### Right Branch - **Initial Identification**: 52 records were identified via citation searching. - **Screening**: All 52 records were retrieved (0 exclusions). - **Eligibility Assessment**: 52 reports were assessed, with 11 excluded (21.2% exclusion rate): - 6 (11.5%) excluded for Non-NLP. - 5 (9.6%) excluded for Non-relevance. - **Inclusion**: The flowchart does not explicitly state the number of included studies for this branch, but it can be inferred as 52 - 11 = 41. --- ### Key Observations 1. **Scale Disparity**: The database/registry branch starts with 13,243 records, far exceeding the 52 records from citation searching. 2. **Exclusion Rates**: - Database branch: 90.7% exclusion rate during screening, with 40.2% exclusion during eligibility assessment. - Citation branch: 21.2% exclusion rate during eligibility assessment. 3. **Non-NLP Exclusions**: Non-NLP exclusions are more frequent in the database branch (38 vs. 6). 4. **Non-Relevance Exclusions**: Non-relevance exclusions are higher in the database branch (52 vs. 5). --- ### Interpretation The flowchart highlights two distinct pathways for study identification: 1. **Database/Registry Method**: - **Strengths**: Captures a large volume of studies (13,243 records), suggesting broad coverage. - **Weaknesses**: High exclusion rates (90.7% during screening, 40.2% during eligibility) indicate many studies are irrelevant or non-compliant with NLP criteria. - **Implication**: While comprehensive, this method requires rigorous filtering to ensure quality. 2. **Other Methods (Citation Searching)**: - **Strengths**: Lower initial volume (52 records) but higher relevance, with only 21.2% exclusion during eligibility. - **Weaknesses**: Limited scalability due to smaller starting pool. - **Implication**: This method may yield higher-quality studies but lacks the breadth of database searches. **Notable Trends**: - Non-NLP exclusions are more prevalent in the database branch, suggesting stricter adherence to NLP criteria in this pathway. - Non-relevance exclusions dominate in both branches, emphasizing the importance of topic alignment in study selection. **Conclusion**: The flowchart underscores the trade-off between quantity (database/registry) and quality (citation searching). Systematic reviewers must balance these factors to ensure comprehensive yet relevant study inclusion.