2604.24665v1

# Benchmarking Source-Sensitive Reasoning in Turkish: Humans and LLMs under Evidential Trust Manipulation ## Abstract This paper investigates whether source trustworthiness shapes Turkish evidential morphology and whether large language models (LLMs) track this sensitivity. We study the past-domain contrast between -DI and -mIş in controlled cloze contexts where the information source is overtly external, while only its perceived reliability is manipulated (High-Trust vs. Low-Trust). In a human production experiment, native speakers of Turkish show a robust trust effect: High-Trust contexts yield relatively more -DI, whereas Low-Trust contexts yield relatively more -mIş, with the pattern remaining stable across sensitivity analyses. We then evaluate 10 LLMs in three prompting paradigms (open gap-fill, explicit past-tense gap-fill, and forced-choice A/B selection). LLM behavior is highly model- and prompt-dependent: some models show weak or local trust-consistent shifts, but effects are generally unstable, often reversed, and frequently overshadowed by output-compliance problems and strong base-rate suffix preferences. The results provide new evidence for a trust-/commitment-based account of Turkish evidentiality and reveal a clear human–LLM gap in source-sensitive evidential reasoning. Keywords: Turkish evidentiality, trustworthiness, LLMs, cloze task, human–LLM comparison, benchmarks Benchmarking Source-Sensitive Reasoning in Turkish: Humans and LLMs under Evidential Trust Manipulation | 1 University of Chicago, skarakas@uchicago.edu | | --- | | 2 Fırat University, ysimsek@firat.edu.tr | Abstract content ## 1. Introduction Evidentiality refers to the linguistic encoding of information source, that is, how a speaker indicates the basis on which a proposition is presented (e.g., direct perception, inference, or report) Willett (1988); Dendale and Tasmowski (2001); de Haan (2001); Plungian (2001); Aikhenvald (2004); Boye (2012); Ünal and Papafragou (2020). In many languages, evidential markers systematically signal whether the speaker witnessed an event, inferred it from available evidence, or learned it from someone else. Because evidentiality lies at the interface of semantics, pragmatics, and discourse, it has been central to research on speaker commitment, reliability, and perspective in natural language. A useful perspective on evidentiality comes from language acquisition and “thinking for speaking”: as children learn a language, they learn to attend to distinctions that the language regularly encodes Brown and Lenneberg (1954); Slobin (1996). In evidential languages, this can make knowledge source (e.g., direct experience, inference, report) especially salient early on, since speakers must repeatedly track and express it in discourse. For Turkish, this suggests that evidential morphology is not just grammatical, but part of a broader cognitive routine linking source monitoring, memory, and speaker stance Aksu-Koç and Slobin (1986). Turkish offers a particularly important case for evidentiality research because past-domain morphology is closely tied to distinctions in information source and epistemic stance Aksu-Koç and Slobin (1986); Underhill (1976); Johanson (2003); Göksel and Kerslake (2005); Kornfilt (1997). A well-known contrast is between the suffix -DI, traditionally associated with direct evidence or stronger speaker commitment, and the suffix -mIş, which is commonly associated with indirect evidence, such as inference or report Aksu-Koç and Slobin (1986); Underhill (1976); Johanson (2003); Gül (2009); Şener (2011); Göksel and Kerslake (2005). Evidentiality marking information source Direct firsthand access Indirect non-firsthand access Visual Non-visual sensory Inferential Reported / hearsay languages vary in how they grammaticalize these distinctions, and many systems do not map neatly onto a single taxonomy. Figure 1: Information-source marking, adapted from typological overviews of evidential systems Willett (1988); Aikhenvald (2004). A minimal illustration is given in 1 – 1: in 1, the use of -DI is classically associated with a context in which the speaker directly witnessed the event, whereas in 1, the use of -mIş suggests that the speaker did not directly witness the event but instead inferred it or learned it from another source. At the same time, a substantial body of work notes that the actual distribution and interpretation of these forms are sensitive to discourse context, information structure, and speaker stance, so the traditional labels are best understood as useful descriptive generalizations rather than rigid one-to-one mappings between form and meaning Gül (2009); Şener (2011). . ˙ Ahmet Ali’yi gör-dü. Ahmet Ali- acc see- pst.dir ‘Ahmet saw Ali (witnessed/direct evidence).’ . . ˙ Ahmet Ali’yi gör-müş. Ahmet Ali- acc see- pst.indir ‘Ahmet apparently/reportedly saw Ali (hearsay/inference).’ . The -DI / -mIş contrast offers a well-studied case for investigating how languages encode information source, speaker stance, and pragmatic inference Aksu-Koç and Slobin (1986); Johanson (2003); Göksel and Kerslake (2005); Gül (2009); Şener (2011); Kandemirci et al. (2023); Ataman-Devrim et al. (2025), and it is also central to developmental work on how children learn to track and express knowledge source in discourse Aksu-Koç and Slobin (1986); Slobin (1996). In addition, evidential distinctions are highly relevant for NLP tasks such as translation, summarization, and source-sensitive interpretation. ### 1.1. Why evidentiality matters for LLM research? Evidentiality is also increasingly relevant to LLM research, because many of the core reliability problems in modern text generation are, in effect, problems about source status: whether a claim is grounded in direct input evidence, retrieved documents, parametric memory, inference, or unsupported generation. Recent work on factuality and hallucination has shown that fluent model outputs may be unfaithful to available evidence or false in world knowledge terms, motivating finer-grained analyses of how models represent and communicate the basis of their claims (Maynez et al., 2020; Lin et al., 2022; Manakul et al., 2023; Min et al., 2023; Bang et al., 2025; Liu et al., 2026). In parallel, work on calibration and self-evaluation suggests that models can sometimes track uncertainty or answerability, but this ability is uneven and sensitive to task format and available context (Kadavath et al., 2022). At the same time, frontier and open models are improving rapidly in reasoning, long-context processing, multilinguality, and agentic performance, as reflected in recent reports (Singh et al., 2025; Yang et al., 2025). These gains make source-sensitive evaluation more, not less, important: as models become stronger and more fluent, the central question is increasingly not only whether they can produce plausible language, but whether they can reliably represent the evidential basis of what they say. From this perspective, evidentiality offers a linguistically grounded framework for studying how systems encode (or fail to encode) distinctions among witnessed, inferred, and reported information. This is especially important in settings that explicitly require provenance, such as retrieval-augmented generation and citation-based QA, where models must connect claims to supporting evidence and where attribution quality itself has become a major evaluation target (Lewis et al., 2020; Nakano et al., 2022; Gao et al., 2023; Li et al., 2024). More broadly, evidential categories provide a principled way to ask whether LLMs merely optimize surface plausibility or whether they track source-sensitive epistemic distinctions that are central to human communication. Besides, this line of work is also important because direct human–LLM comparisons for Turkish remain relatively scarce, especially for source-sensitive semantic/pragmatic phenomena such as evidentiality. Much of the recent Turkish LLM literature has focused on model development, benchmarking, and engineering issues (Bayram et al., 2025a, b; Er et al., 2025; Isbarov et al., 2025; Umutlu et al., 2025; Wicaksono et al., 2025; Bayram et al., 2026; Karakaş and Şimşek, 2026; Toraman et al., 2026; Uğur et al., 2026), rather than tightly controlled comparisons between human judgments and model preferences on linguistically targeted contrasts. Emerging Turkish-focused human–LLM studies suggest that such comparisons can reveal substantial mismatches between surface-form preferences and human-like contextual reasoning in several linguistic domains (Keleş and Dinçtopal Deniz, 2024, 2025; Karakaş, 2026a, b). ## 2. Related Work #### Turkish evidentiality: traditional descriptive and grammatical accounts. Turkish has long occupied a central place in the evidentiality literature because its past-domain morphology is closely tied to distinctions in information source and speaker stance (Underhill, 1976; Aksu-Koç and Slobin, 1986; Johanson, 2003; Kornfilt, 1997; Göksel and Kerslake, 2005). A traditional descriptive contrast distinguishes -DI and -mIş: -DI is often introduced as the form associated with direct evidence (or stronger speaker commitment), while -mIş is associated with indirect evidence, especially inferential and reportative uses (Underhill, 1976; Johanson, 2003; Göksel and Kerslake, 2005; Aksu-Koç and Slobin, 1986). This contrast has also been influential in acquisition and discourse-based work, where Turkish evidential morphology is treated as a system that helps speakers track and communicate how information was obtained (Aksu-Koç and Slobin, 1986). At the same time, the Turkish literature shows that the traditional mapping is not a simple one-to-one encoding. A central debate is whether -DI is a true direct evidential, or primarily a past tense form whose “directness” and stronger commitment effects arise through discourse-pragmatic inference in canonical contexts (Gül, 2009; Şener, 2011). Likewise, analyses of -mIş emphasize sensitivity to evidential source, discourse configuration, and speaker perspective, rather than a single fixed meaning (Gül, 2009; Johanson, 2003). For this reason, recent work often treats direct / indirect as useful descriptive labels while allowing context-sensitive meaning and pragmatic strengthening. For the present study, we adopt Giannakidou and Mari’s framework on veridicality, nonveridicality, commitment, and evidential bias (Giannakidou, 1998, 2006; Giannakidou and Mari, 2016, 2018, 2021). This framework is useful because it separates notions often conflated in evidentiality research: (i) information source, (ii) speaker commitment, and (iii) the structure of the speaker’s information state. It also motivates treating source trustworthiness as a factor shaping evidential interpretation and commitment (Boscaro et al., 2025). Let $M=M_i(w,t)$ be speaker $i$ ’s information state at world $w$ and time $t$ . A compact way to encode the relevant distinction is to contrast veridical and nonveridical states: . $\textsc{Veridical}_M(p)$ iff all worlds in $M$ are $p$ -worlds: $∀ w∈ M [p(w)]$ . . $\textsc{Nonveridical}_M(p)$ iff $M$ contains both a $p$ -world and a $¬ p$ -world: $∃ w∈ M [p(w)]$ and $∃ v∈ M [¬ p(v)]$ . The formula in 2 states that the speaker’s information state supports only $p$ -worlds (full support for $p$ ), while the formula in 2 states that the information state remains compatible with both $p$ and $¬ p$ possibilities. In this perspective, evidential and modal expressions can encode an evidence-based bias toward $p$ without requiring full veridical commitment. This distinction is useful for Turkish evidentiality because it allows us to ask not only which source type is invoked, but also whether that source is treated as strong enough to support a more veridical commitment profile. In our trust-based design, source trustworthiness is therefore modeled as a factor that can shift speakers between stronger commitment and weaker, evidence-biased interpretations, rather than as a purely descriptive source label. ## 3. Methods To examine how Turkish evidential morphology responds to contextual trustworthiness, we used two complementary experimental paradigms with two corresponding datasets. First, we designed a controlled cloze-style task that can be administered to both human participants and LLMs. Second, we constructed a larger dataset for LLM-only evaluation in order to test generalization over a broader set of contexts. In all tasks, the target is a cloze (fill-in-the-blank) completion: participants or models infer and produce the missing final word of a sentence from the preceding context. This shared format provides a direct measure of context-sensitive morphological preferences in production and enables human–model comparison within a single task setup. ### 3.1. Dataset 1 for Human and LLM Comparison Dataset 1 consists of 60 original items, partitioned into three categories: high-trust, low-trust, and filler sentences. All items were manually authored by the researchers. High-trust sources correspond to announcement channels that are generally considered reliable in everyday life, whereas low-trust sources typically consist of channels that are difficult to trust. These sources are provided in Appendix A. During construction, we aimed to make the target completion as uniquely and unambiguously recoverable as possible, thereby minimizing cases with multiple equally plausible completions, while also balancing potential confounds such as lexical frequency and sentence length across conditions. In the cloze task, participants were asked to complete the final blank with the most natural and contextually appropriate continuation. . Ḃelediyenin SMS uyarısına göre sular ……. ‘According to the municipality’s SMS alert, the water ……..’ . . Ẏan binadaki teyzenin dediğine göre sular ……. ‘According to the lady next door, the water …….’ . Across conditions, the source of information is kept overtly external through explicit attribution frames (e.g., X’e göre ‘according to X’; cf. 3.1 – 3.1). The manipulation targets the perceived reliability of that external source: High-Trust items present the proposition as coming from institutionally authoritative channels (e.g., an official municipal SMS notification), whereas Low-Trust items present it as coming from informal or weakly accountable channels (e.g., a statement by a neighbor). To verify that this manipulation isolates trust rather than idiosyncratic item properties, we ran a separate norming study with a broad participant pool. Participants consistently rated High-Trust sources as more credible than Low-Trust sources, and items that failed to show the intended separation were revised or removed. ### 3.2. Dataset 2 for LLM-Only Evaluation Dataset 2 is an expanded item set containing all experimental items from Dataset 1 together with additional items, and was designed exclusively for large language model evaluation. It preserves the same structural properties and manipulation logic as Dataset 1, while increasing coverage and statistical stability by substantially expanding the number of scenarios and lexical realizations. The expanded set contains 200 items in total. Dataset 2 was not administered to human participants; instead, it was designed to test model behavior on a broader item base without increasing participant burden, and to assess whether patterns observed in the human-scale dataset replicate and generalize when the trust manipulation is instantiated across a wider range of contexts. ## 4. Experiments ### 4.1. Human Experiment The human experiment received prior approval from the University of Chicago Institutional Review Board (IRB26-0198). All participants provided informed consent before participation. We recruited 75 unique participants who self-identified as native speakers of Turkish, were at least 18 years old, and were residing in Turkey at the time of participation. The experiment was implemented and administered online using the PCIbex / PennController platform (Zehr and Schwarz, 2018). The main study produced 4,500 trials in total: 1,500 High-Trust trials, 1,500 Low-Trust trials, and 1,500 filler trials. For each trial, we recorded the participant’s completion response. Our primary analyses target the critical High-Trust vs. Low-Trust contrast, so we analyze 3,000 trials after excluding fillers. All responses were analyzed in de-identified form. This balanced design allows a controlled test of how the trust manipulation affects production preferences. ### 4.2. LLM Experiments In this study, LLMs were tested using two different datasets and ten models (Team et al., 2025; neuralwork, n.d.; NovusResearch, n.d.; mradermacher, n.d.; Yang et al., 2025; Trendyol, n.d.; TURKCELL, n.d.; Bezir et al., 2024; ocaklisemih, n.d.; cenkersisman, n.d.). and three prompts.The models evaluated in this study consist of Turkish-focused and multilingual large language models. gpt2-turkish-10M, WiroAI-turkish-LLM-8B, Turkcell-LLM-7B, Trendyol-LLM-7B, Gemma-2-9B-IT-TR, and Novus-7B-TR are models that have been either directly trained on Turkish or fine-tuned with Turkish data. In contrast, Gemma-3 and Qwen3-32B are trained on large-scale multilingual data and are capable of processing Turkish through their multilingual capacities. Some models, on the other hand, are built on a multilingual foundation and later fine-tuned with Turkish instruction data. This diversity allows us to compare the effects of Turkish-focused and multilingual training approaches on sensitivity to evidentiality morphology. During the study, the models were asked to fill in the blank format that was created 3.1 – 3.1. Three prompts were used to perform this task; these prompts are given in Appendix B. While the models were expected to generate the word in the space indicated by the space in the first two prompts, in the third prompt two different words were given as options, one of which is the word conjugated with "-DI" 4.2, and the other is the word conjugated with "-mIş" 4.2. In the third prompt, the models were expected to choose the most appropriate word between these two options according to the flow of the sentence. While generating the model outputs, the temperature was set to 0.1 and the top-p to 1; an additional max token setting of 8 was added to the third prompt. . ˙ Getir-il-di bring- pass - pst.dir ‘was brought (-DI form)’ . . ˙ Getir-il-miş bring- pass - pst.indir ‘apparently/ reportedly was brought (-mIş form)’ . ## 5. Human experiment results ### 5.1. Production: trust robustly shifts -DI vs. -mIş Table 1 shows strict-coding counts for critical trials (High+Low; fillers excluded). Descriptively, High-trust contexts produce more DI completions, whereas Low-trust contexts produce more mIş completions. The Other category is similar across conditions (High: 28.4%; Low: 29.5%), suggesting that the trust manipulation primarily redistributes responses within the evidential morphology space rather than increasing off-target productions. | hightrustrow High trust-lowtrustrow Low trust | dihi 727 (48.5%) 548 (36.5%) | 347 (23.1%)-mislo 509 (33.9%) | 426 (28.4%) 443 (29.5%) | | --- | --- | --- | --- | Table 1: Strict coding counts by condition (critical trials only; $n=1500$ per condition). Percentages are within condition. Figure 2 plots the within-condition proportions of DI vs. mIş after excluding Other. In the High-trust condition, $\textsc{DI}=727/(727+347)=0.677$ and $\textsc{mI\c{s}}=0.323$ . In the Low-trust condition, $\textsc{DI}=548/(548+509)=0.518$ and $\textsc{mI\c{s}}=0.482$ . DI mIş $0$ $0.2$ $0.4$ $0.6$ $0.8$ $1$ Proportion High trust DI mIş $0$ $0.2$ $0.4$ $0.6$ $0.8$ $1$ Proportion Low trust Figure 2: Human production (strict coding): within-condition proportions of -DI vs. -mIş among DI/mIş responses only. Two robustness checks yield the same qualitative pattern. (i) Under lenient last-token coding (allowing two-word VP completions), the association remains strong ( $χ^2(1)=52.685$ , $p=3.92× 10^-13$ ; OR $=1.859$ , 95% CI $[1.571, 2.199]$ ). (ii) A content-controlled stratified analysis with matched content keys gives a pooled OR $=2.029$ , 95% CI $[1.654, 2.490]$ (CMH $χ^2(1)=46.789$ , $p=7.91× 10^-12$ ), with no strong evidence of heterogeneity across strata ( $p=0.137$ ). | Gemma-3-27B-IT | 64.5 | 195 | 3 | 168 | 21 | 98.5 | 88.9 | +9.6 | $7.44× 10^-5$ | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | Gemma-2-9B-IT-TR | 47.2 | 158 | 2 | 113 | 10 | 98.8 | 91.9 | +6.9 | $0.006$ | | Orbita-v0.1 | 40.3 | 83 | 40 | 97 | 22 | 67.5 | 81.5 | -14.0 | $0.018$ | | Qwen3-32B | 24.7 | 54 | 3 | 81 | 10 | 94.7 | 89.0 | +5.7 | $0.371$ | | Novus-7B-TR | 21.3 | 79 | 1 | 46 | 2 | 98.8 | 95.8 | +2.9 | $0.556$ | | WiroAI-TR-8B | 16.8 | 54 | 5 | 40 | 2 | 91.5 | 95.2 | -3.7 | $0.696$ | | GPT-OSS-20B-Astro | 11.3 | 29 | 11 | 26 | 2 | 72.5 | 92.9 | -20.4 | $0.058$ | | GPT2-TR-10M | 2.8 | 10 | 0 | 7 | 0 | 100.0 | 100.0 | +0.0 | – | | Turkcell-LLM-7B | 1.2 | 5 | 0 | 2 | 0 | 100.0 | 100.0 | +0.0 | – | | Trendyol-LLM-7B | 0.0 | 0 | 0 | 0 | 0 | – | – | – | – | Table 2: Experiment I (prompting-based gap-fill) LLM results. Usable = percentage of all generations classifiable as DI or mIş (i.e., DI/MIŞ%). DI% is computed as $DI/(DI+MIS)$ within each trust condition. $Δ$ DI is High minus Low (percentage points). $p$ values are Fisher exact tests on {High, Low} $×$ {DI, mIş}. | Gemma-3-27B-IT | 96.0 | 93 | 2 | 82 | 15 | 97.9 | 84.5 | +13.4 | $1.51× 10^-3$ | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | Gemma-2-9B-IT-TR | 91.0 | 92 | 0 | 88 | 2 | 100.0 | 97.8 | +2.2 | $0.243$ | | Orbita-v0.1 | 22.0 | 19 | 6 | 11 | 8 | 76.0 | 57.9 | +18.1 | $0.327$ | | Qwen3-32B | 32.5 | 24 | 3 | 35 | 3 | 88.9 | 92.1 | -3.2 | $0.686$ | | Novus-7B-TR | 28.0 | 27 | 6 | 21 | 2 | 81.8 | 91.3 | -9.5 | $0.449$ | | WiroAI-TR-8B | 24.5 | 24 | 3 | 20 | 2 | 88.9 | 90.9 | -2.0 | $1.000$ | | GPT-OSS-20B-Astro | 27.5 | 17 | 11 | 19 | 8 | 60.7 | 70.4 | -9.7 | $0.573$ | | GPT2-TR-10M | 4.5 | 3 | 0 | 6 | 0 | 100.0 | 100.0 | +0.0 | – | | Turkcell-LLM-7B | 2.0 | 3 | 0 | 1 | 0 | 100.0 | 100.0 | +0.0 | – | | Trendyol-LLM-7B | 0.0 | 0 | 0 | 0 | 0 | – | – | – | – | Table 3: Experiment II (explicit past-tense generation) LLM results. Each item was generated three times (tur1--tur3), and the final item-level label was determined by majority vote among outputs classifiable as DI or mIş (ties excluded). Usable = percentage of all items (out of 200) receiving a majority DI or mIş label. DI% is computed as $DI/(DI+MIS)$ within each trust condition. $Δ$ DI is High minus Low (percentage points). $p$ values are Fisher exact tests on {High, Low} $×$ {DI, mIş} using item-level majority labels. ## 6. The First Experiment with LLMs Experiment I evaluates LLM behavior in a prompting-based gap-fill task using 200 trust-manipulated Turkish cloze items. Each item was presented to the model three times, and the final label was determined by majority vote across the three outputs. Table 2 summarizes the model-level results on the prompted task, reporting (i) the percentage of usable outputs (i.e., generations classifiable as DI or mIş), (ii) run-level DI / mIş counts in High- and Low-Trust conditions, (iii) the within-condition DI share, and (iv) Fisher exact tests for the High-vs.-Low trust contrast. A first important result is that the amount of usable evidential data varies substantially across models, ranging from 64.5% (Gemma-3-27B-IT) to 0.0% (Trendyol-LLM-7B). This means that, in the prompting setup, a large portion of outputs may be off-target and therefore cannot be interpreted as an evidential choice. Among models with substantial usable DI / mIş outputs, the trust effect is not uniform. Gemma-3-27B-IT shows a clear and statistically reliable shift in the expected direction: the DI share decreases from 98.5% in High-Trust contexts to 88.9% in Low-Trust contexts ( $Δ$ DI $=+9.6$ pp, $p=7.44× 10^-5$ ), indicating relatively more mIş in Low-Trust items. Gemma-2-9B-IT-TR shows a similar but smaller significant pattern (98.8% vs. 91.9%; $Δ$ DI $=+6.9$ pp, $p=0.006$ ). In contrast, Orbita-v0.1 shows a significant effect in the opposite direction, with a higher DI share in Low-Trust than in High-Trust contexts (81.5% vs. 67.5%; $Δ$ DI $=-14.0$ pp, $p=0.018$ ). The remaining models mostly show null, unstable, or weak effects. Qwen3-32B, Novus-7B-TR, and WiroAI-TR-8B exhibit small and non-significant differences ( $p=.371$ , $.556$ , and $.696$ , respectively). GPT-OSS-20B-Astro shows a larger numerical reverse shift ( $Δ$ DI $=-20.4$ pp) but does not reach conventional significance ( $p=.058$ ). For GPT2-TR-10M and Turkcell-LLM-7B, all usable outputs are DI, so no informative DI / mIş comparison can be made; Trendyol-LLM-7B yields no usable outputs. Overall, Experiment I suggests that in an open-generation prompting setup, trust-sensitive evidential behavior is highly model-dependent and often overshadowed by prompting/compliance limitations and a strong general bias toward DI. ## 7. The Second Experiment with LLMs Experiment II evaluates LLM behavior in an explicit past-tense generation setup using the same 200 trust-manipulated Turkish cloze items. Unlike Experiment I, the prompt here explicitly instructed models to produce a past-tense completion. As in Experiment I, each item was sampled three times. To obtain a single interpretable evidential label per item, we assigned a final item-level label by majority vote across the three outputs (tur1--tur3) among outputs classifiable as DI or mIş (ties excluded). Table 3 summarizes the model-level results for the explicit-past prompting condition, reporting (i) the percentage of usable items (i.e., items receiving a majority label classifiable as DI or mIş), (ii) item-level DI / mIş counts in High- and Low-Trust conditions, (iii) the within-condition DI share, and (iv) Fisher exact tests for the High-vs.-Low trust contrast. A first key result is that usability remains strongly model-dependent even with explicit past-tense instructions, ranging from 96.0% (Gemma-3-27B-IT) and 91.0% (Gemma-2-9B-IT-TR) to 0.0% (Trendyol-LLM-7B). Thus, explicit past-tense prompting substantially improves the proportion of usable evidential outputs for some models, but not for all. Among models with substantial usable DI / mIş outputs, Gemma-3-27B-IT shows the clearest and statistically reliable trust-sensitive pattern in the predicted direction. Its DI share decreases from 97.9% in High-Trust contexts to 84.5% in Low-Trust contexts ( $Δ$ DI $=+13.4$ pp, $p=1.51× 10^-3$ ), while the number of mIş labels increases from 2 items (High) to 15 items (Low). This indicates a relative increase in mIş under Low-Trust contexts. Gemma-2-9B-IT-TR also shows high usability and a small numerical shift in the expected direction (100.0% vs. 97.8%; $Δ$ DI $=+2.2$ pp), but the difference is not statistically significant ( $p=.243$ ). Orbita-v0.1 shows a larger numerical shift in the expected direction ( $Δ$ DI $=+18.1$ pp), but its low usability (22.0%) and non-significant test ( $p=.327$ ) limit the strength of this result. The remaining models mostly show null, unstable, or reverse-direction patterns. Qwen3-32B, Novus-7B-TR, WiroAI-TR-8B, and GPT-OSS-20B-Astro all show non-significant High-vs.-Low differences ( $p=.686$ , $.449$ , $1.000$ , and $.573$ , respectively), and several exhibit a numerical reverse shift (i.e., a higher DI share in Low-Trust than in High-Trust contexts). For GPT2-TR-10M and Turkcell-LLM-7B, all usable majority labels are DI, so no informative DI / mIş comparison can be made. Trendyol-LLM-7B produces no usable items. Overall, Experiment II shows that explicit past-tense prompting can improve evidential compliance and reveal a trust-sensitive DI / mIş pattern in some models (most clearly Gemma-3-27B-IT), but the effect remains highly model-dependent and is weak, unstable, or absent in most others. ### 7.1. Exploratory Head-Level Modulation of Trust-Cue Attention As an exploratory mechanistic analysis, we examined attention-weight redistribution in Gemma-2-9B-IT-TR under the explicit past-tense prompting condition. We focused on the pre-generation decision state (q = last_prompt) and quantified attention directed to the trust-cue span, operationalized as the token sequence immediately preceding göre. Across heads, the strongest condition sensitivity was confined to a small subset of mid-to-late layers, with the single best layer at $L=25$ and the highest-scoring layers concentrated in the range $L=22$ – $29$ . However, the aggregate pairwise contrast was weak and directionally unstable ( $Δ_pair$ mean $=-0.0179$ , 95% bootstrap CI $[-0.0513, 0.0170]$ ; $P(Δ_pair>0)=46\$ ). Under this operationalization, the results do not provide strong evidence that trust is robustly encoded via systematic reallocation of attention mass to the cue span in Gemma-2-9B-IT-TR; accordingly, we treat the attention maps as qualitative diagnostic evidence rather than as a stable mechanistic signature. The corresponding visualizations are provided in Appendix C. | Gemma-2-9B-IT-TR Gemma-3-27B-IT GPT-OSS-20B-Astro | 60.0 78.9 94.9 | 44.0 66.3 84.0 | 16.0 12.6 10.9 | 58.0 56.0 55.3 | 0.0 3.5 0.5 | 0.033 0.054 0.019 | 0.268 0.381 0.173 | 97.5 97.5 92.0 | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | Orbita-v0.1 | 91.0 | 84.0 | 7.0 | 53.5 | 0.0 | 0.199 | 0.994 | 96.0 | | Turkcell-LLM-7B | 97.0 | 90.0 | 7.0 | 53.3 | 0.5 | 0.082 | 0.491 | 81.0 | | WiroAI-TR-8B | 15.2 | 11.0 | 4.2 | 52.3 | 0.5 | 0.408 | 1.000 | 91.0 | | Qwen3-32B | 81.0 | 77.0 | 4.0 | 52.0 | 0.0 | 0.603 | 1.000 | 95.0 | | Novus-7B-TR | 99.0 | 98.0 | 1.0 | 50.5 | 0.0 | 1.000 | 1.000 | 99.0 | | Trendyol-LLM-7B | 0.0 | 0.0 | 0.0 | 50.0 | 0.0 | 1.000 | 1.000 | 100.0 | | GPT2-TR-10M | 0.0 | – | – | 0.0 | 99.5 | – | – | 99.5 | Table 4: Experiment III (A/B forced-choice prompting) combined model results using item-level majority vote over three runs. DI% High/Low are computed after excluding abstentions (invalid outputs and tie/no-majority items). $Δ$ pp = DI% High minus DI% Low. Acc% uses the target mapping High $→$ -DI, Low $→$ -mIş. Fisher exact tests are computed on High/Low $×$ {DI, mIş} after excluding abstentions; Holm $p$ values correct for multiple testing across models. 3-run same% is the proportion of items with identical outputs across all three replicates. Applying the same analysis to Gemma-3-27B-IT yielded a larger and more internally consistent pair-level contrast. Condition-sensitive modulation was again localized to a restricted subset of mid-to-late-layer heads, but with a clearer concentration in higher layers (best layer $L=31$ ; top layers $L=31,33,36,37$ ). The aggregate pairwise effect was reliably non-zero under the present metric definition ( $Δ_pair$ mean $=-0.0983$ , 95% bootstrap CI $[-0.1151, -0.0817]$ ; $P(Δ_pair>0)=9\$ ), indicating systematic condition-dependent redistribution of attention toward the trust-cue span. We interpret this as evidence that Gemma-3-27B-IT exhibits a more coherent internal sensitivity to trust-related contextual cues than Gemma-2-9B-IT-TR in this prompting setup. At the same time, this interpretation should remain qualified, since raw span-level attention mass can be affected by surface differences such as cue-length asymmetries across conditions. The relevant maps are shown in Appendix C. ## 8. The Third Experiment with LLMs Experiment III evaluates LLM behavior in a forced-choice prompting setup, in which models are explicitly asked to choose between two candidate completions: a -DI form (option A) and a -mIş form (option B). This design is more constrained than Experiments I–II because it removes open-ended lexical generation and directly targets the evidential choice itself. The same 200 trust-manipulated Turkish items are used (100 High-Trust, 100 Low-Trust), and each item is queried three times (tur1--tur3). Final item-level labels are determined by majority vote across the three runs. Outputs other than A / B, as well as ties/no-majority cases, are treated as abstentions and excluded from evidential-rate and Fisher-test calculations. Table 4 reports per-model condition-wise DI rates (DI % High and DI % Low), the High-minus-Low difference ( $Δ$ pp), target-mapping accuracy (High $→$ -DI, Low $→$ -mIş), and abstention rate over all 200 items. Table 4 reports per-model Fisher exact tests (High/Low $×$ DI / MIS), Holm-corrected $p$ values across models, and 3-run agreement (same %), which measures response stability across the three replicates. A first important result is that most usable models show a directionally trust-consistent trend: DI % is higher in High-Trust than in Low-Trust contexts. The largest shift is observed for neuralwork/gemma-2-9b-it-tr ( $Δ$ pp $=16.0$ ), followed by google/gemma-3-27b-it ( $12.6$ pp) and ocaklisemih/gpt-oss-20b-turkish-astrology-gguf ( $10.9$ pp). Several other models also show smaller positive shifts (e.g., mradermacher/Orbita-v0.1-i1-GGUF, Turkcell-LLM-7b-v1, WiroAI/wiroai-turkish-llm-8b, Qwen/Qwen3-32B, NovusResearch/Novus-7b-tr_v1). Trendyol-LLM-7b-base-v1.0 shows no shift (0.0 vs. 0.0). This pattern suggests that, under direct A/B comparison, many models weakly track the trust manipulation in the predicted direction. At the same time, the results also show that base-rate preferences remain strong and often dominate behavior. Several models overwhelmingly prefer one option across both conditions (e.g., very high DI rates for Novus, Turkcell, Orbita, and GPT-OSS; near-zero DI rates for Trendyol and GPT2-TR-10M), which compresses the effect of trust and limits target-mapping accuracy. Consistent with this, Acc% values remain close to chance for most models (roughly 50–58%), even when the direction of the trust effect is correct. In other words, models can show a small High $>$ Low DI shift without demonstrating robust context-sensitive evidential selection at the item level. Statistical testing reinforces this conclusion. Some models yield nominally small Fisher exact $p$ values (e.g., $p=.019$ for GPT-OSS and $p=.033$ for Gemma-2-9B-IT-TR; Table 4), but none remains significant after Holm correction across models. Thus, Experiment III provides evidence for a weak, directionally consistent trend in several systems, but not for a robust per-model trust effect after correction for multiple comparisons. Finally, response stability across the three replicates is generally high (3-run same% often above 90%), indicating that model choices in this A/B paradigm are relatively deterministic once the prompt format is fixed. This is methodologically useful: compared with open-ended prompting, the forced-choice setup reduces output-format variability and improves comparability across models. However, the combination of high replicate agreement and near-chance Acc% for many systems suggests that the models are often making stable choices driven by global option preferences rather than reliably applying the trust cue in a human-like way. ## 9. Discussion This study makes theoretical and methodological contributions to Turkish evidentiality and its evaluation in LLMs. Human results show a clear trust effect: more credible external sources favor -DI, while less credible sources favor -mIş (Section 5; Table 1; Figure 2). In contrast, current LLMs do not reliably reproduce this pattern, and their apparent evidential behavior varies with prompt format, output compliance, and response constraints (Tables 2, 3, 4). The paper’s central theoretical contribution is a direct experimental test of a trustworthiness -based perspective on Turkish evidentiality, inspired by the Giannakidou–Mari framework. To our knowledge, this is a novel application of that framework to Turkish -DI / -mIş. Rather than treating evidential morphology only as a categorical source-type marker, we test whether the credibility of an explicitly external source shifts evidential choice; the human data show that it does. This matters because the source is explicitly external in both conditions ((e.g., X’e göre ‘according to X’)); the manipulation changes only the source’s trustworthiness. The observed shift therefore supports an analysis in which Turkish evidential morphology is sensitive not just to source type, but also to source quality and the speaker’s resulting commitment profile. In Giannakidou–Mari terms, the pattern is consistent with differences in support strength for the prejacent: more trustworthy sources favor stronger commitment (and relatively more -DI), whereas less trustworthy sources favor weaker, evidence-weighted commitment (and relatively more -mIş) (Giannakidou and Mari, 2016, 2018, 2021; Boscaro et al., 2025). Our findings do not by themselves settle the long-standing question of whether -DI is lexically a direct evidential or a past tense form whose “directness” effects arise pragmatically. However, they do provide an important constraint on the debate. Because the manipulation keeps source attribution explicit and varies trustworthiness instead, the human pattern is difficult to reduce to a simple source-present/source-absent contrast. The results are more naturally captured by accounts that allow context-sensitive interactions among evidential morphology, speaker commitment, and discourse-pragmatic reasoning. In this sense, the present data strengthen the case for treating Turkish evidential choice as part of a broader inferential and commitment-sensitive system, rather than a purely rigid source-labeling mechanism. #### What the LLM results may suggest. Across the three LLM experiments, the main result is a dissociation between surface task performance under prompting constraints and human-like trust-sensitive evidential reasoning. In Experiment I (open gap-fill prompting), many models produce a high proportion of unusable outputs, and even among usable outputs the trust effect is highly model-dependent. In Experiment II (explicit past-tense prompting), usability improves substantially for some models (especially Gemma variants), and Gemma-3-27B-IT shows the clearest trust-consistent shift, but the pattern remains weak, absent, or unstable in most models. In Experiment III (A/B forced-choice), response stability becomes high and several models show directionally trust-consistent trends, yet effects are generally modest, target-mapping accuracy remains near chance for many systems, and no per-model effect survives Holm correction. Thus, these results suggest that many LLMs can exhibit format-dependent traces of the expected pattern, but they do not reliably reproduce the human sensitivity to source trustworthiness as a stable, cross-paradigm property. This is precisely the kind of mismatch that a linguistically targeted, human–LLM comparison can reveal: a model may appear “reasonable” in one prompt format while still failing to encode the underlying contextual factor in a robust way. A related question is why explicit past-tense prompting changes model performance. We suggest that the main reason is task specification: in the open gap-fill setting, models must jointly infer a plausible lexical completion, tense, and evidential morphology, which leaves the task relatively underspecified. By explicitly requiring a past-tense completion in Experiment II, the prompt narrows the response space and makes the -DI / -mIş contrast more directly relevant to the decision. In this sense, the improvement for some models is consistent with recent work showing that more specific instructions can reduce prompt sensitivity relative to underspecified prompts, even if they do not solve the deeper representational problem (Pecher et al., 2026). At the same time, our results also align with recent findings that morphological generalization remains difficult for LLMs in agglutinative languages such as Turkish, especially when models must produce the appropriate inflection rather than merely recognize it (Ismayilzada et al., 2025). The paper also contributes a reusable evaluation paradigm for Turkish source-sensitive semantics/pragmatics by combining: (i) a human cloze dataset, (ii) an expanded LLM-only dataset, and (iii) three prompting regimes that vary constraint level (open generation, explicit tense generation, forced choice). This multi-paradigm design makes it possible to separate at least three sources of variation that are often conflated in LLM evaluations: compliance/formatting behavior, base-rate morphological preference, and context sensitivity to the trust manipulation. In particular, the contrast between Experiments I–III shows that forcing the response space can improve comparability and stability, but it can also expose strong default biases that limit genuine contextual adaptation. More broadly, the study argues for evaluating Turkish LLMs on controlled linguistic contrasts with human baselines, rather than relying only on broad benchmarks. A useful point of comparison is TurBLiMP, the first Turkish benchmark of linguistic minimal pairs, which evaluates LMs on 16 grammatical phenomena and has significantly expanded linguistically informed evaluation for Turkish (Başar et al., 2025). Our study is complementary to that line of work. Whereas TurBLiMP focuses on minimal-pair judgments over a broad range of grammatical phenomena, it does not target evidentiality. This matters because evidentiality is not only a morphosyntactic contrast in Turkish, but also a source-sensitive semantic–pragmatic phenomenon tied to speaker commitment, information source, and discourse reasoning. Our results therefore extend Turkish LLM evaluation into a domain that is central to Turkish grammar but not captured by existing minimal-pair benchmarks. For evidentiality and related phenomena, this is especially important because the relevant behavior is not merely lexical accuracy but context-dependent mapping between linguistic form and epistemic/discourse structure. ## 10. Conclusion This paper investigated how source trustworthiness shapes Turkish evidential choice in the -DI / -mIş contrast, and whether current LLMs track this sensitivity. In a controlled human cloze experiment, we found a robust and replicable trust effect: participants produced relatively more -DI in High-Trust contexts and relatively more -mIş in Low-Trust contexts. This provides new empirical support for a trust-/commitment-based perspective on Turkish evidentiality and shows that the Giannakidou–Mari framework offers a productive theoretical lens for Turkish data. Across three LLM experiments (open gap-fill prompting, explicit past-tense prompting, and forced-choice A/B prompting), model behavior was substantially more variable and strongly dependent on prompt format, output compliance, and base-rate suffix preferences. While some models showed weak or local trust-consistent trends, the effect was not robust across models or paradigms. Overall, the results reveal a clear human–LLM gap in source-sensitive evidential reasoning and motivate future work on more faithful evaluation and modeling of evidential and commitment-related meaning in Turkish. ## Ethics Statement This work investigated the interaction between contextual source trustworthiness and Turkish evidential morphology (-DI vs. -mIş) using both human cloze responses and LLM-based preference scoring. Human data were collected under informed-consent procedures and analyzed only at the aggregate level; no personally identifying information is reported, and all examples are presented in anonymized form. The human-subjects component of this study received Institutional Review Board (IRB) approval. For the computational component, model outputs are treated as task-conditioned behavioral responses rather than as evidence of human-like semantic or cognitive representations. Because LLM preferences can be sensitive to evaluation protocol design (including prompting format and tokenization effects), our findings should not be overgeneralized to broad claims about model competence or human evidential reasoning. Finally, by focusing on Turkish, this study helps address the English-centric bias of much computational psycholinguistics and LLM evaluation. ## Limitations This study has several limitations. First, the strongest behavioral evidence comes from the human experiment on Dataset 1, whereas the larger Dataset 2 was used only for LLM evaluation; future work should test whether the same human trust effect scales to the expanded item set. Second, LLM results are partly constrained by prompt compliance and output-format errors, especially in open-generation settings, which can reduce the amount of usable evidential data and make model comparisons harder to interpret. Third, although the forced-choice setup improves control and stability, it may introduce response biases (e.g., option or format preferences) that are not specific to evidential reasoning. The exploratory attention analysis is also limited in scope since it was conducted only for two relatively successful models from Experiment II and is intended as a preliminary qualitative diagnostic rather than a definitive mechanistic account. ## Acknowledgements We thank Anastasia Giannakidou for valuable input on the semantic and pragmatic theory and the interpretation of the results, Ming Xiang for suggestions on the human experimental design, and Christopher Potts and Craig Thorburn for their comments on future directions. We are also grateful to the anonymous reviewers for their helpful feedback and suggestions. Any remaining errors are our own. ## 11. Bibliographical References - A. Y. Aikhenvald (2004) Evidentiality. Oxford University Press, Oxford. External Links: Link Cited by: Figure 1, §1. - A. Aksu-Koç and D. I. Slobin (1986) A psychological account of the development and use of evidentials in Turkish. In Evidentiality: The Linguistic Coding of Epistemology, W. Chafe and J. Nichols (Eds.), pp. 159–167. External Links: Link Cited by: §1, §1, §2. - M. Ataman-Devrim, G. Soley, and A. Aksu-Koç (2025) Children’s understanding of source reliability and knowledge generalizability from grammatical cues: evidence from Turkish. Journal of Child Language, pp. 1–23. Note: First published online 11 December 2025 External Links: Document, Link Cited by: §1. - Y. Bang, Z. Ji, A. Schelten, A. Hartshorn, T. Fowler, C. Zhang, N. Cancedda, and P. Fung (2025) HalluLens: LLM hallucination benchmark. arXiv preprint arXiv:2504.17550. External Links: 2504.17550, Link Cited by: §1.1. - E. Başar, F. Padovani, J. Jumelet, and A. Bisazza (2025) TurBLiMP: a Turkish benchmark of linguistic minimal pairs. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, Suzhou, China, pp. 16495–16510. External Links: Document, Link Cited by: §9. - M. A. Bayram, A. A. Fincan, A. S. Gümüş, S. Karakaş, B. Diri, S. Yıldırım, and D. Çelik (2026) Tokens with meaning: a hybrid tokenization approach for Turkish. External Links: 2508.14292, Link Cited by: §1.1. - M. A. Bayram, A. A. Fincan, A. S. Gümüş, S. Karakaş, B. Diri, and S. Yıldırım (2025a) Tokenization standards and evaluation in natural language processing: a comparative analysis of large language models on turkish. In 2025 33rd Signal Processing and Communications Applications Conference (SIU), pp. 1–4. External Links: Document Cited by: §1.1. - M. A. Bayram, A. A. Fincan, A. S. Gümüş, S. Karakaş, B. Diri, and S. Yıldırım (2025b) Tokenization standards for linguistic integrity: turkish as a benchmark. arXiv preprint arXiv:2502.07057. External Links: Link, Document Cited by: §1.1. - A. Bezir, F. B. Türkay, and C. Asmazoğlu (2024) WiroAI turkish llm 8b. Note: Hugging Face model card External Links: Link Cited by: §4.2. - M. Boscaro, A. Giannakidou, and A. Mari (2025) Evidence type and trustworthiness: the view from social media. In The Grammar of Interaction, pp. 98–131. External Links: Document, Link Cited by: §2, §9. - K. Boye (2012) Epistemic meaning: a crosslinguistic and functional-cognitive study. De Gruyter Mouton, Berlin. External Links: Link Cited by: §1. - R. W. Brown and E. H. Lenneberg (1954) A study in language and cognition. Journal of Abnormal and Social Psychology 49 (3), pp. 454–462. External Links: Document Cited by: §1. - cenkersisman (n.d.) gpt2-turkish-10m. Note: Hugging Face model cardAccessed: 2026-01-20 External Links: Link Cited by: §4.2. - F. de Haan (2001) The place of inference within the evidential system. International Journal of American Linguistics 67 (2), pp. 193–219. External Links: Link Cited by: §1. - P. Dendale and L. Tasmowski (2001) Introduction: evidentiality and related notions. Journal of Pragmatics 33 (3), pp. 339–348. External Links: Document, Link Cited by: §1. - Y. A. Er, I. Kesen, G. Gül Şahin, and A. Erdem (2025) Cetvel: a unified benchmark for evaluating language understanding, generation and cultural capacity of LLMs for Turkish. arXiv preprint arXiv:2508.16431. External Links: Document, Link Cited by: §1.1. - T. Gao, H. Yen, J. Yu, and D. Chen (2023) Enabling large language models to generate text with citations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore, pp. 6465–6488. External Links: Link, Document Cited by: §1.1. - A. Giannakidou and A. Mari (2016) Epistemic future and epistemic must: nonveridicality, evidence, and partial knowledge. In Mood, Aspect, Modality Revisited: New Answers to Old Questions, J. Błaszczak, A. Giannakidou, D. Klimek-Jankowska, and K. Migdalski (Eds.), pp. 75–124. External Links: Link Cited by: §2, §9. - A. Giannakidou and A. Mari (2018) A unified analysis of the future as epistemic modality: the view from greek and italian. Natural Language & Linguistic Theory 36 (1), pp. 85–129. External Links: Document Cited by: §2, §9. - A. Giannakidou and A. Mari (2021) Truth and veridicality in grammar and thought. University of Chicago Press, Chicago. External Links: Document, ISBN 9780226763200, 9780226763347, 9780226763484, Link Cited by: §2, §9. - A. Giannakidou (1998) Polarity sensitivity as (non)veridical dependency. John Benjamins, Amsterdam/Philadelphia. External Links: Link Cited by: §2. - A. Giannakidou (2006) Only, emotive factive verbs, and the dual nature of polarity dependency. Language 82 (3), pp. 575–603. External Links: Document, Link Cited by: §2. - A. Göksel and C. Kerslake (2005) Turkish: a comprehensive grammar. Routledge, London and New York. External Links: ISBN 0-415-21761-X, Link Cited by: §1, §1, §2. - D. Gül (2009) Semantics of Turkish evidential -(i)miş. In Essays on Turkish Linguistics: Proceedings of the 14th International Conference on Turkish Linguistics, August 6–8, 2008, S. Ay, Ö. Aydın, İ. Ergenç, S. Gökmen, S. İşsever, and D. Peçenek (Eds.), pp. 177–186. External Links: Link Cited by: §1, §1, §1, §2. - J. Isbarov, A. Akhundjanova, M. Hajili, K. Huseynova, D. Gaynullin, A. Rzayev, O. Tursun, A. Turdubaeva, I. Saetov, R. Kharisov, S. Belginova, A. Kenbayeva, A. Alisheva, A. Köksal, S. Rustamov, and D. Ataman (2025) TUMLU: a unified and native language understanding benchmark for Turkic languages. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vienna, Austria, pp. 22816–22838. External Links: Link, Document Cited by: §1.1. - M. Ismayilzada, D. Circi, J. Sälevä, H. Sirin, A. Köksal, B. Dhingra, A. Bosselut, D. Ataman, and L. V. D. Plas (2025) Evaluating morphological compositional generalization in large language models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), Albuquerque, New Mexico, pp. 1270–1305. External Links: Document, Link Cited by: §9. - L. Johanson (2003) Evidentiality in Turkic. In Studies in Evidentiality, A. Y. Aikhenvald and R. M. W. Dixon (Eds.), pp. 273–290. External Links: Link Cited by: §1, §1, §2, §2. - S. Kadavath, T. Conerly, A. Askell, T. Henighan, D. Drain, E. Perez, N. Schiefer, Z. Hatfield-Dodds, N. DasSarma, E. Tran-Johnson, S. Johnston, S. El-Showk, A. Jones, N. Elhage, T. Hume, A. Chen, Y. Bai, S. Bowman, S. Fort, D. Ganguli, D. Hernandez, J. Jacobson, J. Kernion, S. Kravec, L. Lovitt, K. Ndousse, C. Olsson, S. Ringer, D. Amodei, T. Brown, J. Clark, N. Joseph, B. Mann, S. McCandlish, C. Olah, and J. Kaplan (2022) Language models (mostly) know what they know. External Links: 2207.05221, Link Cited by: §1.1. - B. Kandemirci, A. Theakston, D. B. Thomsen, and S. Brandt (2023) Does evidentiality support source monitoring and false belief understanding? a cross-linguistic study with Turkish- and english-speaking children. Child Development 94 (4), pp. 889–904. External Links: Document, Link Cited by: §1. - S. Karakaş and Y. Şimşek (2026) From lemmas to dependencies: what signals drive light verbs classification?. In Proceedings of the Second Workshop Natural Language Processing for Turkic Languages (SIGTURK 2026), K. Oflazer, A. Köksal, and O. Varol (Eds.), Rabat, Morocco, pp. 220–227. External Links: Link, Document, ISBN 979-8-89176-370-8 Cited by: §1.1. - S. Karakaş (2026a) Clause-internal or clause-external? Testing Turkish reflexive binding in adapted versus chain of thought large language models. arXiv preprint arXiv:2602.00380. External Links: Document, Link Cited by: §1.1. - S. Karakaş (2026b) Plausibility as commonsense reasoning: humans succeed, large language models do not. External Links: 2604.04825, Link Cited by: §1.1. - O. Keleş and N. Dinçtopal Deniz (2024) A comparative study with human data: do LLMs have superficial language processing?. In 2024 32nd IEEE Conference on Signal Processing and Communications Applications (SIU), External Links: Document, Link Cited by: §1.1. - O. Keleş and N. Dinçtopal Deniz (2025) When men bite dogs: testing good-enough parsing in Turkish with humans and large language models. In Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, Albuquerque, New Mexico, USA, pp. 219–231. External Links: Document, Link Cited by: §1.1. - J. Kornfilt (1997) Turkish. Routledge, London. External Links: Link Cited by: §1, §2. - P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W. Yih, T. Rocktäschel, S. Riedel, and D. Kiela (2020) Retrieval-augmented generation for knowledge-intensive NLP tasks. arXiv preprint arXiv:2005.11401. External Links: Document, Link Cited by: §1.1. - Y. Li, X. Yue, Z. Liao, and H. Sun (2024) AttributionBench: how hard is automatic attribution evaluation?. In Findings of the Association for Computational Linguistics: ACL 2024, Bangkok, Thailand, pp. 14919–14935. External Links: Link, Document Cited by: §1.1. - S. Lin, J. Hilton, and O. Evans (2022) TruthfulQA: measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Dublin, Ireland, pp. 3214–3252. External Links: Link, Document Cited by: §1.1. - X. Liu, X. Yang, Z. Li, P. Li, and R. He (2026) AgentHallu: benchmarking automated hallucination attribution of LLM-based agents. arXiv preprint arXiv:2601.06818. External Links: 2601.06818, Link Cited by: §1.1. - P. Manakul, A. Liusie, and M. Gales (2023) SelfCheckGPT: zero-resource black-box hallucination detection for generative large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore, pp. 9004–9017. External Links: Link, Document Cited by: §1.1. - J. Maynez, S. Narayan, B. Bohnet, and R. McDonald (2020) On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 1906–1919. External Links: Link, Document Cited by: §1.1. - S. Min, K. Krishna, X. Lyu, M. Lewis, W. Yih, P. Koh, M. Iyyer, L. Zettlemoyer, and H. Hajishirzi (2023) FActScore: fine-grained atomic evaluation of factual precision in long form text generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore, pp. 12076–12100. External Links: Link, Document Cited by: §1.1. - mradermacher (n.d.) Orbita-v0.1-i1-GGUF. Note: Hugging Face model cardAccessed: 2026-01-20 External Links: Link Cited by: §4.2. - R. Nakano, J. Hilton, S. Balaji, J. Wu, L. Ouyang, C. Kim, C. Hesse, S. Jain, V. Kosaraju, W. Saunders, X. Jiang, K. Cobbe, T. Eloundou, G. Krueger, K. Button, M. Knight, B. Chess, and J. Schulman (2022) WebGPT: browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332. External Links: Document, Link Cited by: §1.1. - neuralwork (n.d.) gemma-2-9b-it-tr. Note: Hugging Face model cardAccessed: 2026-01-20 External Links: Link Cited by: §4.2. - NovusResearch (n.d.) Novus-7b-tr_v1. Note: Hugging Face model cardAccessed: 2026-01-20 External Links: Link Cited by: §4.2. - ocaklisemih (n.d.) gpt-oss-20b-Turkish-astrology-gguf. Note: Hugging Face model cardAccessed: 2026-01-20 External Links: Link Cited by: §4.2. - B. Pecher, M. Spiegel, R. Belanec, and J. Cegin (2026) Revisiting prompt sensitivity in large language models for text classification: the role of prompt underspecification. arXiv preprint arXiv:2602.04297. External Links: 2602.04297, Document, Link Cited by: §9. - V. A. Plungian (2001) The place of evidentiality within the universal grammatical space. Journal of Pragmatics 33 (3), pp. 349–357. External Links: Link Cited by: §1. - N. Şener (2011) Semantics and pragmatics of evidentials in turkish. Ph.D. Thesis, University of Connecticut, Storrs, CT. External Links: Link Cited by: §1, §1, §1, §2. - A. Singh, A. Fry, A. Perelman, A. Tart, A. Ganesh, A. El-Kishky, A. McLaughlin, A. Low, A. Ostrow, A. Ananthram, A. Nathan, A. Luo, A. Helyar, A. Madry, A. Efremov, A. Spyra, A. Baker-Whitcomb, A. Beutel, A. Karpenko, A. Makelov, A. Neitz, A. Wei, A. Barr, A. Kirchmeyer, A. Ivanov, A. Christakis, A. Gillespie, A. Tam, A. Bennett, A. Wan, A. Huang, A. M. Sandjideh, A. Yang, A. Kumar, A. Saraiva, A. Vallone, A. Gheorghe, A. G. Garcia, A. Braunstein, A. Liu, A. Schmidt, A. Mereskin, A. Mishchenko, A. Applebaum, A. Rogerson, A. Rajan, A. Wei, A. Kotha, A. Srivastava, A. Agrawal, A. Vijayvergiya, A. Tyra, A. Nair, A. Nayak, B. Eggers, B. Ji, B. Hoover, B. Chen, B. Chen, B. Barak, B. Minaiev, B. Hao, B. Baker, B. Lightcap, B. McKinzie, B. Wang, B. Quinn, B. Fioca, B. Hsu, B. Yang, B. Yu, B. Zhang, B. Brenner, C. R. Zetino, C. Raymond, C. Lugaresi, C. Paz, C. Hudson, C. Whitney, C. Li, C. Chen, C. Cole, C. Voss, C. Ding, C. Shen, C. Huang, C. Colby, C. Hallacy, C. Koch, C. Lu, C. Kaplan, C. Kim, C. Minott-Henriques, C. Frey, C. Yu, C. Czarnecki, C. Reid, C. Wei, C. Decareaux, C. Scheau, C. Zhang, C. Forbes, D. Tang, D. Goldberg, D. Roberts, D. Palmie, D. Kappler, D. Levine, D. Wright, D. Leo, D. Lin, D. Robinson, D. Grabb, D. Chen, D. Lim, D. Salama, D. Bhattacharjee, D. Tsipras, D. Li, D. Yu, D. Strouse, D. Williams, D. Hunn, E. Bayes, E. Arbus, E. Akyurek, E. Y. Le, E. Widmann, E. Yani, E. Proehl, E. Sert, E. Cheung, E. Schwartz, E. Han, E. Jiang, E. Mitchell, E. Sigler, E. Wallace, E. Ritter, E. Kavanaugh, E. Mays, E. Nikishin, F. Li, F. P. Such, F. de Avila Belbute Peres, F. Raso, F. Bekerman, F. Tsimpourlas, F. Chantzis, F. Song, F. Zhang, G. Raila, G. McGrath, G. Briggs, G. Yang, G. Parascandolo, G. Chabot, G. Kim, G. Zhao, G. Valiant, G. Leclerc, H. Salman, H. Wang, H. Sheng, H. Jiang, H. Wang, H. Jin, H. Sikchi, H. Schmidt, H. Aspegren, H. Chen, H. Qiu, H. Lightman, I. Covert, I. Kivlichan, I. Silber, I. Sohl, I. Hammoud, I. Clavera, I. Lan, I. Akkaya, I. Kostrikov, I. Kofman, I. Etinger, I. Singal, J. Hehir, J. Huh, J. Pan, J. Wilczynski, J. Pachocki, J. Lee, J. Quinn, J. Kiros, J. Kalra, J. Samaroo, J. Wang, J. Wolfe, J. Chen, J. Wang, J. Harb, J. Han, J. Wang, J. Zhao, J. Chen, J. Yang, J. Tworek, J. Chand, J. Landon, J. Liang, J. Lin, J. Liu, J. Wang, J. Tang, J. Yin, J. Jang, J. Morris, J. Flynn, J. Ferstad, J. Heidecke, J. Fishbein, J. Hallman, J. Grant, J. Chien, J. Gordon, J. Park, J. Liss, J. Kraaijeveld, J. Guay, J. Mo, J. Lawson, J. McGrath, J. Vendrow, J. Jiao, J. Lee, J. Steele, J. Wang, J. Mao, K. Chen, K. Hayashi, K. Xiao, K. Salahi, K. Wu, K. Sekhri, K. Sharma, K. Singhal, K. Li, K. Nguyen, K. Gu-Lemberg, K. King, K. Liu, K. Stone, K. Yu, K. Ying, K. Georgiev, K. Lim, K. Tirumala, K. Miller, L. Ahmad, L. Lv, L. Clare, L. Fauconnet, L. Itow, L. Yang, L. Romaniuk, L. Anise, L. Byron, L. Pathak, L. Maksin, L. Lo, L. Ho, L. Jing, L. Wu, L. Xiong, L. Mamitsuka, L. Yang, L. McCallum, L. Held, L. Bourgeois, L. Engstrom, L. Kuhn, L. Feuvrier, L. Zhang, L. Switzer, L. Kondraciuk, L. Kaiser, M. Joglekar, M. Singh, M. Shah, M. Stratta, M. Williams, M. Chen, M. Sun, M. Cayton, M. Li, M. Zhang, M. Aljubeh, M. Nichols, M. Haines, M. Schwarzer, M. Gupta, M. Shah, M. Huang, M. Dong, M. Wang, M. Glaese, M. Carroll, M. Lampe, M. Malek, M. Sharman, M. Zhang, M. Wang, M. Pokrass, M. Florian, M. Pavlov, M. Wang, M. Chen, M. Wang, M. Feng, M. Bavarian, M. Lin, M. Abdool, M. Rohaninejad, N. Soto, N. Staudacher, N. LaFontaine, N. Marwell, N. Liu, N. Preston, N. Turley, N. Ansman, N. Blades, N. Pancha, N. Mikhaylin, N. Felix, N. Handa, N. Rai, N. Keskar, N. Brown, O. Nachum, O. Boiko, O. Murk, O. Watkins, O. Gleeson, P. Mishkin, P. Lesiewicz, P. Baltescu, P. Belov, P. Zhokhov, P. Pronin, P. Guo, P. Thacker, Q. Liu, Q. Yuan, Q. Liu, R. Dias, R. Puckett, R. Arora, R. T. Mullapudi, R. Gaon, R. Miyara, R. Song, R. Aggarwal, R. Marsan, R. Yemiru, R. Xiong, R. Kshirsagar, R. Nuttall, R. Tsiupa, R. Eldan, R. Wang, R. James, R. Ziv, R. Shu, R. Nigmatullin, S. Jain, S. Talaie, S. Altman, S. Arnesen, S. Toizer, S. Toyer, S. Miserendino, S. Agarwal, S. Yoo, S. Heon, S. Ethersmith, S. Grove, S. Taylor, S. Bubeck, S. Banesiu, S. Amdo, S. Zhao, S. Wu, S. Santurkar, S. Zhao, S. R. Chaudhuri, S. Krishnaswamy, Shuaiqi, Xia, S. Cheng, S. Anadkat, S. P. Fishman, S. Tobin, S. Fu, S. Jain, S. Mei, S. Egoian, S. Kim, S. Golden, S. Mah, S. Lin, S. Imm, S. Sharpe, S. Yadlowsky, S. Choudhry, S. Eum, S. Sanjeev, T. Khan, T. Stramer, T. Wang, T. Xin, T. Gogineni, T. Christianson, T. Sanders, T. Patwardhan, T. Degry, T. Shadwell, T. Fu, T. Gao, T. Garipov, T. Sriskandarajah, T. Sherbakov, T. Kaftan, T. Hiratsuka, T. Wang, T. Song, T. Zhao, T. Peterson, V. Kharitonov, V. Chernova, V. Kosaraju, V. Kuo, V. Pong, V. Verma, V. Petrov, W. Jiang, W. Zhang, W. Zhou, W. Xie, W. Zhan, W. McCabe, W. DePue, W. Ellsworth, W. Bain, W. Thompson, X. Chen, X. Qi, X. Xiang, X. Shi, Y. Dubois, Y. Yu, Y. Khakbaz, Y. Wu, Y. Qian, Y. T. Lee, Y. Chen, Y. Zhang, Y. Xiong, Y. Tian, Y. Cha, Y. Bai, Y. Yang, Y. Yuan, Y. Li, Y. Zhang, Y. Yang, Y. Jin, Y. Jiang, Y. Wang, Y. Wang, Y. Liu, Z. Stubenvoll, Z. Dou, Z. Wu, and Z. Wang (2025) OpenAI gpt-5 system card. External Links: 2601.03267, Link Cited by: §1.1. - D. I. Slobin (1996) From “thought and language” to “thinking for speaking”. In Rethinking Linguistic Relativity, J. J. Gumperz and S. C. Levinson (Eds.), pp. 70–96. External Links: Link Cited by: §1, §1. - G. Team, A. Kamath, J. Ferret, S. Pathak, N. Vieillard, R. Merhej, S. Perrin, T. Matejovicova, A. Ramé, M. Rivière, L. Rouillard, T. Mesnard, G. Cideron, J. Grill, S. Ramos, E. Yvinec, M. Casbon, E. Pot, I. Penchev, G. Liu, F. Visin, K. Kenealy, L. Beyer, X. Zhai, A. Tsitsulin, R. Busa-Fekete, A. Feng, N. Sachdeva, B. Coleman, Y. Gao, B. Mustafa, I. Barr, E. Parisotto, D. Tian, M. Eyal, C. Cherry, J. Peter, D. Sinopalnikov, S. Bhupatiraju, R. Agarwal, M. Kazemi, D. Malkin, R. Kumar, D. Vilar, I. Brusilovsky, J. Luo, A. Steiner, A. Friesen, A. Sharma, A. Sharma, A. M. Gilady, A. Goedeckemeyer, A. Saade, A. Feng, A. Kolesnikov, A. Bendebury, A. Abdagic, A. Vadi, A. György, A. S. Pinto, A. Das, A. Bapna, A. Miech, A. Yang, A. Paterson, A. Shenoy, A. Chakrabarti, B. Piot, B. Wu, B. Shahriari, B. Petrini, C. Chen, C. L. Lan, C. A. Choquette-Choo, C. Carey, C. Brick, D. Deutsch, D. Eisenbud, D. Cattle, D. Cheng, D. Paparas, D. S. Sreepathihalli, D. Reid, D. Tran, D. Zelle, E. Noland, E. Huizenga, E. Kharitonov, F. Liu, G. Amirkhanyan, G. Cameron, H. Hashemi, H. Klimczak-Plucińska, H. Singh, H. Mehta, H. T. Lehri, H. Hazimeh, I. Ballantyne, I. Szpektor, I. Nardini, J. Pouget-Abadie, J. Chan, J. Stanton, J. Wieting, J. Lai, J. Orbay, J. Fernandez, J. Newlan, J. Ji, J. Singh, K. Black, K. Yu, K. Hui, K. Vodrahalli, K. Greff, L. Qiu, M. Valentine, M. Coelho, M. Ritter, M. Hoffman, M. Watson, M. Chaturvedi, M. Moynihan, M. Ma, N. Babar, N. Noy, N. Byrd, N. Roy, N. Momchev, N. Chauhan, N. Sachdeva, O. Bunyan, P. Botarda, P. Caron, P. K. Rubenstein, P. Culliton, P. Schmid, P. G. Sessa, P. Xu, P. Stanczyk, P. Tafti, R. Shivanna, R. Wu, R. Pan, R. Rokni, R. Willoughby, R. Vallu, R. Mullins, S. Jerome, S. Smoot, S. Girgin, S. Iqbal, S. Reddy, S. Sheth, S. Põder, S. Bhatnagar, S. R. Panyam, S. Eiger, S. Zhang, T. Liu, T. Yacovone, T. Liechty, U. Kalra, U. Evci, V. Misra, V. Roseberry, V. Feinberg, V. Kolesnikov, W. Han, W. Kwon, X. Chen, Y. Chow, Y. Zhu, Z. Wei, Z. Egyed, V. Cotruta, M. Giang, P. Kirk, A. Rao, K. Black, N. Babar, J. Lo, E. Moreira, L. G. Martins, O. Sanseviero, L. Gonzalez, Z. Gleicher, T. Warkentin, V. Mirrokni, E. Senter, E. Collins, J. Barral, Z. Ghahramani, R. Hadsell, Y. Matias, D. Sculley, S. Petrov, N. Fiedel, N. Shazeer, O. Vinyals, J. Dean, D. Hassabis, K. Kavukcuoglu, C. Farabet, E. Buchatskaya, J. Alayrac, R. Anil, Dmitry, Lepikhin, S. Borgeaud, O. Bachem, A. Joulin, A. Andreev, C. Hardin, R. Dadashi, and L. Hussenot (2025) Gemma 3 technical report. External Links: 2503.19786, Link Cited by: §4.2. - C. Toraman, A. K. Sever, A. A. Cengiz, E. E. Arslan, G. Sevinç, S. Kantar, M. M. Birdal, Y. F. Güldemir, A. B. Kanburoğlu, S. Felekoğlu, B. Ş. Kütük, B. Tufan, E. Genç, S. Coşkun, G. E. Demir, M. E. Arayıcı, O. Dursun, O. Gungor, S. Üsküdarlı, A. Topraksoy, and E. Darıcı (2026) TurkBench: a benchmark for evaluating Turkish large language models. In Proceedings of the Second Workshop Natural Language Processing for Turkic Languages (SIGTURK 2026), K. Oflazer, A. Köksal, and O. Varol (Eds.), Rabat, Morocco, pp. 126–154. External Links: Link, Document, ISBN 979-8-89176-370-8 Cited by: §1.1. - Trendyol (n.d.) Trendyol-LLM-7b-base-v1.0. Note: Hugging Face model cardAccessed: 2026-01-20 External Links: Link Cited by: §4.2. - TURKCELL (n.d.) Turkcell-LLM-7b-v1. Note: Hugging Face model cardAccessed: 2026-01-20 External Links: Link Cited by: §4.2. - Ö. Uğur, M. Göksu, M. Çimen, M. Yılmaz, E. Şavirdi, A. T. Demir, R. Güllüce, İ. Çetin, and Ö. C. Sağbaş (2026) Mecellem models: turkish models trained from scratch and continually pre-trained for the legal domain. arXiv preprint arXiv:2601.16018. External Links: Link, Document Cited by: §1.1. - E. E. Umutlu, A. A. Cengiz, A. K. Sever, S. Erdem, B. Aytan, B. Tufan, A. Topraksoy, E. Darıcı, and C. Toraman (2025) Evaluating the quality of benchmark datasets for low-resource languages: a case study on Turkish. In Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²), Vienna, Austria and virtual meeting, pp. 471–487. External Links: Link Cited by: §1.1. - E. Ünal and A. Papafragou (2020) Relations between language and cognition: evidentiality and sources of knowledge. Topics in Cognitive Science 12 (1), pp. 115–135. External Links: Document, Link Cited by: §1. - R. Underhill (1976) Turkish grammar. MIT Press, Cambridge, MA. External Links: Link Cited by: §1, §2. - D. Wicaksono, H. A. A. Rozaq, and N. Boz (2025) Emotion recognition for low-resource Turkish: fine-tuning BERTurk on TREMO and testing on xenophobic political discourse. arXiv preprint arXiv:2505.12160. External Links: Document, Link Cited by: §1.1. - T. Willett (1988) A cross-linguistic survey of the grammaticization of evidentiality. Studies in Language 12 (1), pp. 51–97. External Links: Link Cited by: Figure 1, §1. - A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, C. Zheng, D. Liu, F. Zhou, F. Huang, F. Hu, H. Ge, H. Wei, H. Lin, J. Tang, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Zhou, J. Lin, K. Dang, K. Bao, K. Yang, L. Yu, L. Deng, M. Li, M. Xue, M. Li, P. Zhang, P. Wang, Q. Zhu, R. Men, R. Gao, S. Liu, S. Luo, T. Li, T. Tang, W. Yin, X. Ren, X. Wang, X. Zhang, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Zhang, Y. Wan, Y. Liu, Z. Wang, Z. Cui, Z. Zhang, Z. Zhou, and Z. Qiu (2025) Qwen3 technical report. External Links: 2505.09388, Link Cited by: §1.1, §4.2. - J. Zehr and F. Schwarz (2018) PennController for internet based experiments (ibex). OSF. External Links: Document Cited by: §4.1. ## Appendix A Source Categories | High | Resmi Gazete | Official Gazette | | --- | --- | --- | | High | Kurumun resmi sayfası | Official website of the institution | | High | Hastanenin astığı duyuru | Hospital notice board announcement | | High | Belediyenin SMS uyarısı | Municipality SMS alert | | High | Resmi mail | Official email | | Low | Anonim telegram grubu | Anonymous Telegram group | | Low | Yan apartmanda oturan komşu | Neighbor living in the adjacent building | | Low | Diğer şirketteki çalışan | Employee at another company | | Low | Yanlış haberler paylaşan Instagram grubu | Instagram group sharing false news | | Low | Sokakta geçen biri | A random passerby on the street | Table 5: Examples of high- and low-trust source categories used in the experiment. ## Appendix B Prompt Templates ### B.1. Turkish Prompts 1. Prompt 1 (TR) Aşağıdaki cümlede “__________.” ile belirtilmiş bir boşluk var. Bu boşluğu cümlenin yapısının akışına en uygun fiille tamamla. Cevap verirken kurallar: - Sadece boşluğa gelecek kısmıyaz. - Cümleyi tekrar etme. - Açıklama yapma. Tek kelime cevap ver ve fiil kullanarak tamamla - Cevap vermeden geçme. Cümle: {prefix} __________. Cevap: 1. Prompt 2 (TR) Aşağıdaki cümlede “__________.” ile belirtilmiş bir boşluk var. Bu boşluğu cümlenin yapısının akışına ve geçmiş zamana göre tamamlanabilecek en uygun kelimeyle tamamla. Cevap verirken kurallar: - Sadece boşluğa gelecek kısmıyaz. - Cümleyi tekrar etme. - Açıklama yapma. - Cevap vermeden geçme. Cümle: {prefix} __________. Cevap: 1. Prompt 3 (TR) Cümledeki boşluğu doldurmak için doğru seçeneği seç. Cümle: {sentence} Seçenekler: A) {di} B) {mis} Kurallar: - Sadece A ya da B yaz. - Açıklama yapma. Cevap: ### B.2. English Prompts 1. Prompt 1 (EN) There is a blank indicated by “__________.” in the sentence below. Fill this blank with the verb that best fits the flow and structure of the sentence. Rules for your answer: - Write only what should go in the blank. - Do not repeat the sentence. - Do not provide an explanation. Answer with a single word and complete it using a verb. - Do not skip answering. Sentence: {prefix} __________. Answer: 1. Prompt 2 (EN) There is a blank indicated by “__________.” in the sentence below. Fill this blank with the most appropriate word that can complete the sentence according to the flow/structure and past tense. Rules for your answer: - Write only what should go in the blank. - Do not repeat the sentence. - Do not provide an explanation. - Do not skip answering. Sentence: {prefix} __________. Answer: 1. Prompt 3 (EN) Choose the correct option to fill the blank in the sentence. Sentence: {sentence} Options: A) {di} B) {mis} Rules: - Write only A or B. - Do not provide an explanation. Answer: ## Appendix C Exploratory Attention Maps <details> <summary>2604.24665v1/figures/gemma2_attn_highlow.png Details</summary>  ### Visual Description ## Heatmap: High-Low difference (trust-span attention mass) | q=last_prompt_token ### Overview The image is a heatmap visualizing the "High-Low difference" metric across neural network layers and attention heads. The color intensity represents the magnitude of differences, with a colorbar indicating values from -0.075 (purple) to +0.100 (yellow). The x-axis represents attention heads (0–14), and the y-axis represents layers (0–40). ### Components/Axes - **X-axis (Head)**: Labeled "Head" with integer markers from 0 to 14. - **Y-axis (Layer)**: Labeled "Layer" with integer markers from 0 to 40. - **Colorbar**: Positioned on the right, ranging from -0.075 (purple) to +0.100 (yellow). Intermediate values include: - -0.050 (dark blue) - -0.025 (medium blue) - 0.000 (cyan) - 0.025 (light green) - 0.050 (bright green) - 0.075 (yellow-green) - 0.100 (yellow) ### Detailed Analysis - **Highest Positive Values (+0.075 to +0.100)**: - Layer 25, Heads 1 and 9 (bright yellow). - Layer 30, Head 9 (yellow-green). - Layer 35, Head 9 (yellow-green). - **Most Negative Values (-0.075)**: - Layer 0, Head 0 (dark purple). - Layer 35, Head 14 (dark purple). - **Intermediate Values**: - Most regions show cyan to medium blue shades (near 0.000 to -0.025). - Layer 20, Head 6 (bright yellow-green, ~+0.075). - Layer 25, Head 6 (bright yellow, ~+0.100). ### Key Observations 1. **Concentration of High Values**: Layers 25–35 and Heads 1, 6, 9 exhibit the strongest positive differences. 2. **Negative Outliers**: Layer 0 (Head 0) and Layer 35 (Head 14) show the most negative differences. 3. **Spatial Patterns**: Positive values cluster in mid-layers (20–35), while negative values dominate early (Layer 0) and late (Layer 35) layers. ### Interpretation The heatmap suggests that attention mechanisms in mid-layers (20–35) exhibit significant variability in "High-Low difference" metrics, particularly in Heads 1, 6, and 9. The extreme negative values in Layer 0 (Head 0) and Layer 35 (Head 14) may indicate baseline or residual attention patterns. The clustering of high values in specific heads implies these components play a disproportionate role in modulating attention dynamics. The absence of values beyond ±0.100 suggests bounded differences, possibly due to normalization or architectural constraints. </details> Figure 3: Gemma-2-9B-IT-TR (Experiment II): high–low difference in trust-span attention mass (query token: last_prompt_token). The pair-level effect is weak and inconsistent ( $Δ_pair$ mean $=-0.0179$ , 95% bootstrap CI $[-0.0513, 0.0170]$ ; $P(Δ_pair>0)=46\$ ), so we treat this as a qualitative diagnostic rather than strong evidence of robust trust encoding. <details> <summary>2604.24665v1/figures/gemma3_attn_highlow.png Details</summary>  ### Visual Description ## Heatmap: High-Low difference (trust-span attention mass) | q=last_prompt_token ### Overview This heatmap visualizes the "High-Low difference" metric across neural network layers and attention heads, with values ranging from -0.15 (purple) to +0.10 (yellow). The metric likely quantifies variability in attention mechanisms, with "trust-span" suggesting a focus on reliable or consistent attention patterns. The majority of values cluster near zero (teal), with scattered extremes in yellow and purple. ### Components/Axes - **X-axis (Head)**: 0 to 30 (31 categories, integer steps) - **Y-axis (Layer)**: 0 to 60 (61 categories, integer steps) - **Color Scale**: - Purple (-0.15) to Yellow (+0.10) - Neutral teal (~0.00) dominates - **Legend**: Positioned right-aligned, with gradient from purple (low) to yellow (high) ### Detailed Analysis - **High Values (+0.10, Yellow)**: - Layer 25, Head 10 - Layer 30, Head 25 - Layer 40, Head 15 - Layer 50, Head 20 - **Low Values (-0.15, Purple)**: - Layer 10, Head 5 - **Neutral Values (~0.00, Teal)**: - Most regions (e.g., Layer 0-10, Heads 0-30; Layer 30-50, Heads 0-30) ### Key Observations 1. **Sparse Extremes**: Only 5 distinct extreme values (4 high, 1 low) across 61x31 grid. 2. **Vertical Clustering**: High values cluster in mid-to-lower layers (25-50). 3. **Horizontal Distribution**: High values appear in heads 10, 15, 20, 25—no clear horizontal pattern. 4. **Low Value Anomaly**: Single purple cell at Layer 10, Head 5 stands out as the most negative value. ### Interpretation The heatmap reveals that trust-span attention mass differences are **highly sparse**, with most attention heads showing minimal variability (teal). The isolated yellow cells in mid-lower layers (25-50) suggest specific heads in these layers exhibit significantly higher attention mass variability, potentially indicating: - **Layer-Specific Sensitivity**: Later layers (25-50) may process more complex features requiring adaptive attention. - **Head Specialization**: Heads 10, 15, 20, 25 might specialize in tasks requiring greater attention flexibility. - **Outlier Behavior**: The purple cell at Layer 10, Head 5 could represent a head with unusually consistent (or rigid) attention patterns. The data implies that attention mechanisms in neural networks exhibit **layer-dependent variability**, with later layers showing more pronounced differences in attention mass. This could reflect architectural design choices or emergent properties of attention dynamics in deep networks. </details> Figure 4: Gemma-3-27B-IT (Experiment II): high–low difference in trust-span attention mass (query token: last_prompt_token). The map shows localized modulation concentrated in a small cluster of mid-to-late layers and heads, consistent with the exploratory analysis in Section 7.