2406.13803

# Semantic Structure-Mapping in LLM and Human Analogical Reasoning **Authors**: Sam Musker, Alex Duchnowski, Raphaël Millière, Ellie Pavlick Abstract Analogical reasoning is considered core to human learning and cognition. Recent studies have compared the analogical reasoning abilities of human subjects and Large Language Models (LLMs) on abstract symbol manipulation tasks, such as letter string analogies. However, these studies largely neglect analogical reasoning over semantically meaningful symbols, such as natural language words. This ability to draw analogies that link language to non-linguistic domains, which we term semantic structure-mapping, is thought to play a crucial role in language acquisition and broader cognitive development. We test human subjects and LLMs on analogical reasoning tasks that require the transfer of semantic structure and content from one domain to another. Advanced LLMs match human performance across many task variations. However, humans and LLMs respond differently to certain task variations and semantic distractors. Overall, our data suggest that LLMs are approaching human-level performance on these important cognitive tasks, but are not yet entirely human like. keywords: language models , analogies , structure-mapping journal: review. Brown University Macquarie University 1 Introduction The recent advances of large language models (LLMs) have raised the question of whether LLMs can serve as useful cognitive models in the study of various aspects of human learning, cognition, and behavior [1, 2, 3]. One such recent debate has focused on whether LLMs acquire the ability to perform analogical reasoning as a by-product of their self-supervised learning objective [4, 5, 6, 7]. Analogical reasoning—the ability to align abstract structures between a source and target domain—is posited to play a central role in human learning and generalization, for example, our ability to reason efficiently in unfamiliar domains [8, 9]. Thus, the question of whether LLMs can reason analogically in a human-like way directly bears on their ability to serve as computational models of human behavior beyond just next-word prediction. Recent work has focused on the ability of advanced LLMs to match human analogical reasoning performance on tasks that involve recognition of spatial and logical transformations in matrices [4] or detecting patterns in strings of letters or numbers [7]. For example, Mitchell [7] uses analogy tasks such as abcd:abce::ijkl:?? in order to test the extent to which LLMs and humans can recognize and generalize abstract structures and operations (in this example, ordered sequences and successor functions). Such studies have produced mixed results, with evidence suggesting that advanced LLMs achieve the same performance and even produce similar error patterns to those observed in humans [4, 6], but with doubts remaining about the robustness of LLMs’ abilities, particularly with respect to increasingly abstract and challenging domains [10]. Previous work has focused almost exclusively on analogies using abstract and arbitrary symbols, where structures are derived from symbols’ spatial positions in the text prompt, but the symbols themselves are unimportant. This leaves out questions about reasoning analogically over semantically meaningful symbols, such as words in natural language. This type of analogical reasoning, which we call semantic structure-mapping, requires mapping between semantic structure in one domain (e.g., the relationship between a dog and a puppy, or that a dog has four legs) and non-semantic (arbitrary) structure in the other domain (e.g., spatial position in the text prompt). This type of mapping is thought to play a crucial role in human cognition and development, such as in the language-analogical reasoning feedback loop proposed by Structure-Mapping Theory (SMT) [11]. Moreover, if LLMs are to provide insight into how humans perform certain cognitive functions, it will likely involve the role of distributional semantic learning [12, 13, 14] in the acquisition or representation of those functions. Therefore, we focus on investigating how humans and LLMs compare in tasks requiring semantic structure-mapping and assessing whether patterns differ from those observed on tasks involving only arbitrary symbols. We design two experiments, focused respectively on the mapping of semantic structure (i.e., semantic relationships between symbols, such as relating the symbol dog to the symbol puppy) and semantic content (i.e., information attached to a symbol such as the knowledge that a dog has four legs). In each experiment, the subject (human or LLM) is presented with a set of left-hand terms (the source domain) and a corresponding set of right-hand terms (the target domain), with the final right-hand term omitted. The subject is asked to fill in this blank. An exact copy of our prompt and an example question is shown in Figure 1. We design multiple variants of such questions designed to probe structure-mapping that involves semantic structure and semantic content, respectively. We additionally design a series of control and distractor conditions—e.g., interleaving informative mappings (square => C C C) with uninformative ones (lime => X X X) in order to expose differences in the underlying mechanism. Overall, the most advanced LLMs we tested match human performance across our primary conditions, even producing human-like error patterns. However, significant differences emerge in several control settings. Even the most advanced LLMs show more sensitivity than humans to information presentation order and struggle to ignore irrelevant semantic information that humans readily dismiss. Thus, our results contribute to the ongoing debate about analogical reasoning, corroborating both work arguing for impressive LLM performance [4, 15] and work highlighting important mechanistic differences between humans and LLMs [10, 16]. Code and data are available at https://github.com/AnonymousReview123/Semantic_Structure_Mapping_Anon. By presenting data on the unique role of semantic structure and content in analogical reasoning, we suggest differences remain in how LLMs and humans represent and map semantic structure, although this gap may be closing as models increase in size and incorporate more diverse training signals. We argue that this has important implications for studying cognitive development and the role of LLMs in this research going forward. 2 Methods 2.1 Experiment Details 2.1.1 Semantic Structure Each subject was presented with a quiz, which is a sequence of four such questions generated using four sets of base domains and four sets of target domains selected such that a participant sees each base and target domain exactly once. Eight variants of the task were devised to investigate the influence of task variations as described above. Questions are introduced with the prompt “We are conducting an experiment on general reasoning abilities. Below we will show you various words and drawings of each, after which you will need to complete the last drawing. Respond as concisely as possible with only the last drawing.” We use the term “drawings” to describe the elements in the target domain because it loosely encapsulates the idea of mapping between the source and target domains. In a similar way to how drawings serve as partial structurally isomorphic representations that depict a subject with varying degrees of abstraction [17], the elements in our target domains establish a space of relations that are isomorphic to those in the source domain. In some cases the term “drawing” is straightforwardly applicable, as when the capitalization of characters corresponds to the term for a mature animal. In other cases the use is strained, as when capitalization instead corresponds to a shape being symmetrical. The transparently liberal use of the term “drawing” is used to prime subjects to reason creatively while attending to the correspondence between source and target domains. The prompt’s lack of reference to analogical reasoning accesses pre-theoretic responses to the extent possible. For the same purpose the experiment is introduced to human subjects and LLMs as studying “general reasoning abilities.” 2.1.2 Semantic Content Each condition (described in Table 4) contains two quizzes, with four questions per quiz. Unless otherwise stated, methodological details of the Semantic Content experiment match those of the Semantic Structure experiment. The four conditions are divided into those that require numeric reasoning and those that do not. Within the numeric and non-numeric conditions respectively, one condition utilizes only one dimension of variation (referred to as “single-attribute”) whereas another adds a second dimension of variation (“multi-attribute”). This allows for comparing the relative performance of human subjects and models when the task is made to require compositional reasoning over layered transformations. Questions were formatted like the following example: | horse => * * * * | | --- | | cat => * * * * | | ant => ! ! ! ! ! ! | | bee => ! ! ! ! ! ! | | chicken => ! ! | | spider => ! ! ! ! ! ! ! ! | | dog => * * * * | | human => | In this example, the number of symbols corresponds to a number-of-legs feature, and the usage of exclamation marks and asterisks corresponds to an egg-laying feature (or, alternatively, a mammal feature). The right-hand sequences of characters thereby encode properties of the entities denoted by the left-hand words. Given that humans are two-legged mammals, the correct answer here would be * *. In order to solve this task, the participant must understand both aspects of the information encoded in the right-hand terms and then construct the answer by generalizing to a new example. 2.2 Participants 2.2.1 LLMs We run our experiments on the following LLMs: GPT-3 [18], GPT-4 [19], Pythia-12B [20], Claude 2 [21], Claude 3 Opus [22], and Falcon-40B [23]. All of the above are transformer-based LLMs trained primarily on a next word prediction objective. GPT-3 consists of a 175B parameter model trained on text completion and finetuned to produce more coherent answers. The details of GPT-4 are not publicly known, but it is considered by some sources to be a mixture-of-experts (MoE) model consisting of numerous GPT-3-scale language models [24]. GPT-4, unlike GPT-3, supplements text-completion pretraining and finetuning with reinforcement learning from human feedback (RLHF) in order to better align model outputs with the expectations of a human user. The training of Claude 2 also includes RLHF, but its performance falls short of GPT-4. The more recent Claude 3 (in our case, the most advanced Opus version) is considered to approximately match GPT-4 performance in general. GPT-3 and -4 are developed by OpenAI, whereas Claude 2 and 3 are developed by Anthropic. Pythia-12B and Falcon-40B are open-weights LLMs trained on a text-completion objective and consist of 12B and 40B parameters respectively. Neither undergoes RLHF. Pythia-12B is developed by EleutherAI, and Falcon-40B is developed by the Technology Innovation Institute. 2.2.2 Human Subjects We also test human participants on our experiments. Reported in the main text are results obtained from 194 (mostly undergraduate) University-Name University students (132 in the Semantic Structure experiment, and 62 in the Semantic Content experiment). The split of participants between experiments approximately matches the 9:4 ratio of experiment conditions. The number of participants by condition are as follows: Defaults 18, Distracted 18, Only RHS 18, Permuted Pairs 17, Permuted Questions 17, Random Finals 15, Random Permuted Pairs 6, Randoms 8, Relational 15, Categorial 16, Multi Attribute 16, Numeric 16, Numeric Multi Attribute 14. The Relational, Categorial, Multi Attribute, Numeric, and Numeric Multi Attribute conditions each have two quizzes while the remaining conditions each have four quizzes per condition. Subjects were assigned randomly to a single quiz from one condition without the re-use of subjects. Roughly the same number of participants were assigned to each condition, with the exception of the Random and Random Permuted Pairs conditions. These were together assigned roughly the expected number of subjects for a single condition due to their similarity. The subjects were recruited through email advertisements and offered $10 in compensation. Earlier results obtained for the Semantic Structure experiment from an online sample of participants recruited through Prolific are reported in Figure 11 of the Appendix. We ensure that humans and LLMs are given comparable information in our prompting design. A given human participant sees one quiz with four questions, with questions revealed one at a time with the answer shown following each response. LLMs are prompted with the first question of a quiz, then the second question with the first question and its (correct) answer accumulating in the prompt, and so forth for the four questions in a quiz. This prompt accumulation mimics the availability in the memory of human subjects of previous answers within a quiz. 2.3 Statistical testing In each experiment, we are interested in the relative performance of human subjects and the best-performing models and how this depends on the particular experiment conditions. Differences between most models and human subjects are large and do not require statistical analysis, and so we focus our statistical analysis on the performance of GPT-4 relative to human subjects and Claude 3 relative to human subjects. For each experiment and pair of subjects (human subjects and GPT-4, or human subjects and Claude 3) we fit a logistic model to the data with and without interactions between the subject type and the experiment condition. In all cases, the outcome variable is the un-aggregated per-question score achieved by a subject (either a 0 or 1), and the predictor variables are experiment condition (e.g. “Defaults” or “Permuted Pairs”) and subject type (e.g. “human subjects” or “GPT-4”). We use four likelihood ratio tests to assess whether the interaction between subject type and experiment condition is significant for a given pair of subjects within a particular experiment, as motivated by Glover [25]. In all four cases the interaction is significant, and so we use simple effects analysis to investigate the direction and significance of the effect of subject type within particular conditions. For the semantic content experiment, we additionally perform a logistic simple effects analysis comparing the performance of a single subject type (human, GPT-4, or Claude 3) in compositional versus non-compositional conditions for the numeric and non-numeric cases respectively with the non-compositional condition as reference. For example, we assess the effect of the condition being Multi Attribute with Categorial as the reference condition for only the subject type Claude 3 (and likewise for the other two examined subject types). Further details are provided in Sections A.1 and A.2 of the Appendix. 3 Results 3.1 Mapping Semantic Structure We first design a set of experiments investigating the ability of LLMs and human subjects to map semantic structure in the source domain onto arbitrary, non-semantic structure in the target domain. In this set of experiments, our source domain (left-hand side) is a set of words which are assumed to possess some relational structure, and our target domain (right-hand side) is a set of strings related via non-linguistic string operations. | We are conducting an experiment on general reasoning abilities. Below we will show you various words and drawings of each, after which you will need to complete the last drawing. Respond as concisely as possible with only the last drawing. | | --- | | Question 1: | | square => C C C | | rectangle => c c c | | circle => C C | | oval => | square rectangle circle oval C C C c c c C C c c Figure 1: An example question (from the Defaults condition of the Semantic Structure experiment) with a representation of the structure-mapping solution below. The source domain is in blue and the target domain is in orange (for the provided elements) and yellow (for the inferred element). 3.1.1 Overall Performance Human subjects perform well overall, obtaining accuracy between 0.4 and 0.9 across the various conditions. The most advanced LLMs that we test attain accuracies in the range 0.1-0.95 across conditions. This performance range is comparable to prior work on analogical reasoning over arbitrary symbols. For example, the results of human subjects on the “zero-generalization setting” studied by both Webb et al. [4] and Mitchell et al. [10] range from 0.2-0.8 in the former study and from 0.5-1.0 in the latter study. Similarly, results for LLMs (GPT-3, GPT-3.5, and GPT-4) across those conditions range from 0.1-1.0 in the two studies. Thus, our data suggest that analogies involving semantic structure-mapping are not inherently easier or harder than those which make use of arbitrary symbols. Our Defaults condition consists of lexical items as a source domain and one of several string operation relations as a target domain. To investigate the robustness of performance metrics, we introduce three control conditions: (1) Permuted Questions, in which we present unaltered versions of the core task with varied question ordering; (2) Permuted Pairs, in which we alter the order in which the lines of the analogy are presented; and (3) Distracted, in which we interleave unrelated mappings between the lines of the target analogy. These conditions are shown in Table 1. We do not expect Permuted Questions to materially alter the task, but might see some effect of the Permuted Pairs and Distracted conditions, as they could make the relevant relations less transparent: see, for example, work on the blocking advantage in humans [26] and in LLMs [27]. | Defaults | Basic test of semantic structure-mapping | square => C C C rectangle => c c c circle => C C oval => | | --- | --- | --- | | Permuted Pairs | Like Defaults, but with row order permuted | rectangle => c c c circle => C C square => C C C oval => | | Distracted | Like Defaults, but with a distractor row added | square => C C C rectangle => c c c pillow => A P circle => C C oval => | Table 1: Defaults and control conditions used to measure ability of humans and LLMs to perform analogical reasoning tasks that involve semantic structure-mapping. The Permuted Questions condition (not shown) is identical to Defaults, but with question order permuted. Figure 2 shows the performance of humans and LLMs in the Defaults condition as a function of their performance on MMLU MMLU scores are few-shot for GPT-4 and 5-shot for other models. The reported human baseline is the estimate for human experts given by Hendrycks et al. [28]. The score for Pythia 12B could not be found and so we use the reported value for Pythia 6.9B Tulu., a widely-used language competency benchmark. Increasing MMLU score is associated with higher accuracy on the Defaults condition. Smaller models do not perform competitively (Pythia-12B obtains an accuracy of 0.0, Falcon 40B 0.1, GPT-3 0.5, and Claude 2 0.6). This steadily increasing performance is presumed to correlate with the scale of model parameters and training data [29]. We focus our remaining analysis on comparing human subjects to GPT-4 and Claude 3. In the Defaults condition, neither GPT-4 (coef=-0.7696, z=-1.659, p=0.097) nor Claude 3 (coef=-0.6131, z=-1.299, p=0.194) performs significantly worse than human subjects. <details> <summary>extracted/5679376/Images/Comparison_Default_MMLU.png Details</summary>  ### Visual Description ## Chart: Model Accuracy vs. MMLU Score ### Overview The image is a scatter plot comparing the accuracy of several language models against their MMLU (Massive Multitask Language Understanding) score. The plot displays data points for different models, with error bars indicating the uncertainty in their accuracy. The models include Human, Claude 3 Opus, GPT-4, Claude 2, GPT-3, Falcon-40B, and Pythia-12B-Deduped. ### Components/Axes * **X-axis:** MMLU score, ranging from 30 to 100. * **Y-axis:** Accuracy, ranging from 0.0 to 1.0. * **Data Points:** Each data point represents a language model. * Human*: Black data point with error bars. Located at approximately (92, 0.9). * Claude 3 Opus: Red data point with error bars. Located at approximately (87, 0.8). * GPT-4: Red data point with error bars. Located at approximately (88, 0.75). * Claude 2: Red data point with error bars. Located at approximately (80, 0.58). * GPT-3: Red data point with error bars. Located at approximately (60, 0.5). * Falcon-40B: Blue data point with error bars. Located at approximately (65, 0.1). * Pythia-12B-Deduped*: Cyan data point with error bars. Located at approximately (35, 0.0). * **Error Bars:** Vertical lines extending above and below each data point, representing the uncertainty in the accuracy measurement. * **Labels:** Text labels identifying each data point (language model). Gray lines connect the data points to their labels. ### Detailed Analysis * **Human*:** The "Human*" data point is located at the top-right of the plot, indicating high accuracy and a high MMLU score. The approximate coordinates are (92, 0.9). * **Claude 3 Opus:** Located near the top, with an MMLU score around 87 and an accuracy of approximately 0.8. * **GPT-4:** Positioned slightly below Claude 3 Opus, with an MMLU score around 88 and an accuracy of approximately 0.75. * **Claude 2:** Located in the middle of the plot, with an MMLU score around 80 and an accuracy of approximately 0.58. * **GPT-3:** Positioned to the left of Claude 2, with an MMLU score around 60 and an accuracy of approximately 0.5. * **Falcon-40B:** Located near the bottom, with an MMLU score around 65 and an accuracy of approximately 0.1. * **Pythia-12B-Deduped*:** Located at the bottom-left of the plot, indicating low accuracy and a low MMLU score. The approximate coordinates are (35, 0.0). ### Key Observations * There is a general trend of increasing accuracy with increasing MMLU score. * The "Human*" data point represents a performance ceiling, with the highest accuracy and MMLU score. * The models Claude 3 Opus and GPT-4 are close in performance, both having high accuracy and MMLU scores. * Falcon-40B and Pythia-12B-Deduped* have significantly lower accuracy and MMLU scores compared to the other models. ### Interpretation The plot illustrates the relationship between a language model's ability to understand and perform multiple tasks (MMLU score) and its overall accuracy. The data suggests that models with higher MMLU scores tend to have higher accuracy. The "Human*" data point serves as a benchmark, indicating the potential upper limit of performance. The relative positions of the other models provide a comparison of their capabilities, with Claude 3 Opus and GPT-4 demonstrating strong performance, while Falcon-40B and Pythia-12B-Deduped* show comparatively weaker performance. The error bars indicate the variability in the accuracy measurements, which should be considered when comparing the models. </details> Figure 2: Human and LLM accuracy in the Defaults condition, relative to performance on the MMLU benchmark. Models in blue are not instruction-tuned while models in orange are. Error bars show standard errors. Figure 3 compares humans to high-performing LLMs in the Defaults and Permuted Pairs conditions. LLM performance drops in the Permuted Pairs condition, while humans seem equally able to infer the mapping regardless of word presentation order. This effect is significant for both Claude 3 (coef = -1.7802, z = -4.217, p < 0.001) and GPT-4 (coef = -1.6796, z = -3.975, p < 0.001). This suggests that, while the overall performance is comparable, there are likely meaningful mechanistic differences in how the analogy is processed in humans versus LLMs. The remaining control conditions and data for all tested models are shown in Figure 15 of the Appendix . In these conditions, we find that humans and models are roughly equally affected. For example, accuracy in the Distracted condition drops by approximately 0.25 for all three subject types. <details> <summary>extracted/5679376/Images/fig3_new.png Details</summary>  ### Visual Description ## Bar Chart: Accuracy Comparison ### Overview The image is a bar chart comparing the accuracy of "Defaults" and "Permuted pairs" across three categories: Human, GPT-4, and Claude 3. The chart displays accuracy on the y-axis, ranging from 0.0 to 1.0. Each category (Human, GPT-4, Claude 3) has two bars stacked on top of each other, representing "Permuted pairs" (bottom, light blue) and "Defaults" (top, white with black outline). Error bars are present on each stacked bar. ### Components/Axes * **Y-axis:** "Accuracy", ranging from 0.0 to 1.0 in increments of 0.2. * **X-axis:** Categorical labels: "Human", "GPT-4", and "Claude 3". * **Legend (Top-Right):** * "Defaults" (white bar with black outline) * "Permuted pairs" (light blue bar) ### Detailed Analysis * **Human:** * "Permuted pairs" accuracy: approximately 0.84 with an error bar extending roughly +/- 0.04. * "Defaults" accuracy: approximately 0.05 (the remaining portion of the bar to reach the top) with an error bar extending roughly +/- 0.04. * **GPT-4:** * "Permuted pairs" accuracy: approximately 0.55 with an error bar extending roughly +/- 0.04. * "Defaults" accuracy: approximately 0.24 (the remaining portion of the bar to reach the top) with an error bar extending roughly +/- 0.04. * **Claude 3:** * "Permuted pairs" accuracy: approximately 0.53 with an error bar extending roughly +/- 0.04. * "Defaults" accuracy: approximately 0.29 (the remaining portion of the bar to reach the top) with an error bar extending roughly +/- 0.04. ### Key Observations * Human performance with "Permuted pairs" is significantly higher than GPT-4 and Claude 3. * The "Defaults" accuracy is higher for Claude 3 than GPT-4 and Human. * The error bars appear to be of similar magnitude across all categories and conditions. ### Interpretation The chart suggests that humans are better at handling "Permuted pairs" compared to the AI models GPT-4 and Claude 3. The "Defaults" accuracy indicates the baseline performance without permutation. The difference in performance between "Defaults" and "Permuted pairs" highlights the impact of permutation on each category. The error bars provide a visual representation of the uncertainty or variability in the accuracy measurements. </details> Figure 3: Human and LLM accuracy in the Defaults and Permuted Pairs conditions. Error bars show standard errors. 3.1.2 Effect of Semantic Structure on Reasoning We next investigate more directly the extent to which humans and LLMs leverage semantic structure in order to complete our analogy tasks. To do this, we design three variants of our Defaults analogy task (see Table 2). First, the Only RHS condition removes the source domain entirely. High performance in this condition thus indicates that a subject is able to complete the questions based only on the evident pattern in the target domain. We then introduce two variants which make the semantic structure in the source domain less coherent: the Randoms condition uses unrelated words, while the Random Finals condition uses of three related words followed by one random word. We thus take the performance difference between the RHS Only condition and either the Random or Random Final condition to be a measure of the subject’s bias toward using the semantic structure of the source domain. That is, if the subject is capable of solving the task by simply ignoring the left hand side (the Only RHS condition), then poor performance in the other conditions indicates that the subject was misled by the presence of the altered left hand side. | Only RHS | Test of how well the answer can be inferred without using any structure-mapping | C C C c c c C C | | --- | --- | --- | | Randoms | Variant of Defaults in which there is no semantic structure relating the words on the left hand side | banana => C C C fireplace => c c c bean => C C plug => | | Random Last | Variant of Defaults in which the final term is not semantically related to the preceding terms | square => C C C rectangle => c c c circle => C C lime => | Table 2: Conditions involving alteration or omission of the source domain. The Random Permuted Pairs condition (not shown) is identical to Randoms, but with the order of elements within questions permuted. Both humans and models competently complete the Only RHS condition (see Figure 4). Accuracy is approximately 0.8 for Claude 3 with human subjects and GPT-4 slightly higher at 0.9. GPT-4 is not significantly different from humans in this condition (coef = 0.1178, z = 0.223, p = 0.824), and Claude 3 is worse than humans by a barely significant margin (coef = -0.9130, z = -1.994, p = 0.046). Thus, both humans and LLMs are able to complete the task without the guidance of the left hand side. Considering this, we look at the performance degradation associated with encountering incoherent semantic structure on the left hand side. Humans exhibit a modest decrease in accuracy of about 0.15 in the Random and Random Permuted Pairs conditions relative to defaults. Claude-3 and GPT-4, however, exhibit much larger drops: Claude 3 decreases by approximately 0.5 relative to Defaults, while GPT-4 decreases by 0.6 and 0.4 in the Random and Random Permuted Pairs conditions. Across these two conditions, both GPT-4 (coef = -2.1972, z = -5.211, p < 0.001) and Claude 3 (coef = -2.0680, z = -4.960, p < 0.001) perform significantly worse than humans. <details> <summary>extracted/5679376/Images/fig4_new.png Details</summary>  ### Visual Description ## Bar Chart: Accuracy Comparison ### Overview The image is a bar chart comparing the accuracy of "Human", "GPT-4", and "Claude 3" across three categories: "Only RHS", "Randoms", and "Random finals". The y-axis represents accuracy, ranging from 0.0 to 1.0. Error bars are present on each bar, indicating the uncertainty in the accuracy measurements. ### Components/Axes * **X-axis:** Categorical axis with three categories: "Human", "GPT-4", and "Claude 3". * **Y-axis:** Numerical axis labeled "Accuracy", ranging from 0.0 to 1.0 in increments of 0.2. * **Legend:** Located in the top-right corner, defining the colors for each category: * "Only RHS": White bar outline * "Randoms": Light red * "Random finals": Light blue ### Detailed Analysis * **Human:** * "Random finals": Approximately 0.48 with error bars extending from approximately 0.43 to 0.53. * "Randoms": Approximately 0.31, resulting in a combined height of approximately 0.79. * "Only RHS": The remaining height up to approximately 0.89. * **GPT-4:** * "Random finals": Approximately 0.07 with error bars extending from approximately 0.05 to 0.09. * "Randoms": Approximately 0.11, resulting in a combined height of approximately 0.18. * "Only RHS": The remaining height up to approximately 0.90 with error bars extending from approximately 0.88 to 0.92. * **Claude 3:** * "Random finals": Approximately 0.28 with error bars extending from approximately 0.26 to 0.30. * "Randoms": Approximately 0.07, resulting in a combined height of approximately 0.35. * "Only RHS": The remaining height up to approximately 0.77 with error bars extending from approximately 0.75 to 0.79. ### Key Observations * GPT-4 has the highest "Only RHS" accuracy, followed by Human and then Claude 3. * Human has the highest "Random finals" accuracy, followed by Claude 3 and then GPT-4. * The error bars suggest that the differences between the models may not be statistically significant in all cases. ### Interpretation The bar chart compares the accuracy of Human, GPT-4, and Claude 3 across three different categories. The "Only RHS" category represents the accuracy when only the right-hand side is considered. The "Randoms" category represents the accuracy when random samples are used. The "Random finals" category represents the accuracy when random samples are used for the final stage. GPT-4 excels in the "Only RHS" category, suggesting it performs well when focusing on the right-hand side. Human performs best in the "Random finals" category, indicating its strength in handling random samples in the final stage. Claude 3's performance is relatively lower compared to the other two models in all categories. The error bars indicate the uncertainty in the accuracy measurements. Overlapping error bars suggest that the differences between the models may not be statistically significant in some cases. Further statistical analysis would be needed to confirm the significance of the observed differences. </details> Figure 4: Human and LLM accuracy in Only RHS, Randoms, and Random finals conditions. Data from the Random Permuted Pairs condition is shown in Figure 15 of the Appendix . Error bars show standard errors. From this we conclude that human subjects are able to easily identify when the left hand side contains no useful semantic structure to leverage. When there is none, they are able to employ a strategy that only relies on the right hand side. By contrast, models do not seem capable of easily identifying the lack of informativeness of the left hand side in these conditions, as they do not use the strategy of only attending to the right hand side, even though they show their capability of using this strategy when no left hand side is present. This suggests mechanistic differences between how human subjects and models process this task. Although the performance of human subjects does not drop notably in the Random condition compared to the Only RHS condition, it does drop by a wide margin in the Random Finals condition. In this condition, accuracy is approximately 0.5 lower than in the Only RHS condition. This further suggests that the semantic relatedness of the left hand side affects the strategy of human subjects: when the left hand side is clearly unrelated, the information it provides is discarded, but when much of the left hand side appears related, the information is not discarded and the random final word of the source domain prompts an incorrect answer from human subjects. Models also show a large drop in performance in the Random Finals condition relative to Only RHS, with Claude 3 dropping by 0.5 and GPT-4 dropping by 0.8. Simple effects analysis shows that both Claude 3 (coef = -1.0464, z = -2.799, p = 0.005) and GPT-4 (coef = -2.7850, z = -5.168, p < 0.001) are significantly worse than humans in the Random Finals condition. However, we see this difference as less informative than that both models drop in performance across all the random conditions relative to their own performance in the Only RHS condition. 3.1.3 Other Observations We additionally analyze the extent to which human subjects and models improve by question (Figure 14 of the Appendix), and the extent to which the errors made by humans and models follow the same distribution across questions grouped by target domain and across qualitative error types (Figure 13 and Table 5 of the Appendix). We find that humans and models alike improve over subsequent questions, adding to a body of evidence about in-context learning [30, 31, 32]. Humans and models show similar error distributions by target domain, but qualitative error types reveal a closer correspondence between human and GPT-4 errors than Claude 3. 3.1.4 Diagnosing the Use of an RHS-Only Heuristic To clarify whether subjects actually make use of left-right relations or only complete right-side patterns in the Semantic Structure experiment, we design the Relational condition, a $2× n$ variant of the Defaults condition which cannot be solved (consistently) using only the right-hand terms (see the example in Table 3). | pants => H # H | | --- | | glove => X # X | | torso => V | | foot => Z | | head => M | | shirt => V # V | | hat => | Table 3: An example from the Relational variant of the Defaults task, used to diagnose subjects’ tendency to rely on RHS-only heuristics to solve the task. Results are shown in Figure 5. Human subjects and Claude 3 exhibit similar performance, with accuracies of approximately 0.7. GPT-4, however, attains much lower accuracy of approximately 0.35. Simple effects analysis shows that GPT-4 obtains significantly worse accuracy than human subjects (coef = -1.3669, z = -3.065, p = 0.002), while the accuracy of Claude 3 does not differ significantly from human subjects (coef = 0.2111, z = 0.467, p = 0.640). <details> <summary>extracted/5679376/Images/fig5_new.png Details</summary>  ### Visual Description ## Bar Chart: Accuracy Comparison ### Overview The image is a bar chart comparing the accuracy of "Defaults" and "Relational" approaches across three categories: Human, GPT-4, and Claude 3. The chart displays accuracy on the y-axis, ranging from 0.0 to 1.0. Each category (Human, GPT-4, Claude 3) has two bars representing the "Defaults" and "Relational" approaches. Error bars are included on each bar. ### Components/Axes * **Y-axis:** "Accuracy", ranging from 0.0 to 1.0 in increments of 0.2. * **X-axis:** Categories: Human, GPT-4, Claude 3. * **Legend (Top-Right):** * "Defaults": Represented by a white bar with a black outline. * "Relational": Represented by a light blue bar. ### Detailed Analysis Here's a breakdown of the accuracy values for each category and approach: * **Human:** * Relational (Light Blue): Accuracy is approximately 0.65, with an error bar extending from approximately 0.55 to 0.75. * Defaults (White): The total height of the bar is approximately 0.89, so the Defaults accuracy is approximately 0.89 - 0.65 = 0.24. The error bar extends from approximately 0.8 to 0.95. * **GPT-4:** * Relational (Light Blue): Accuracy is approximately 0.32, with an error bar extending from approximately 0.25 to 0.40. * Defaults (White): The total height of the bar is approximately 0.79, so the Defaults accuracy is approximately 0.79 - 0.32 = 0.47. The error bar extends from approximately 0.75 to 0.82. * **Claude 3:** * Relational (Light Blue): Accuracy is approximately 0.70, with an error bar extending from approximately 0.65 to 0.78. * Defaults (White): The total height of the bar is approximately 0.82, so the Defaults accuracy is approximately 0.82 - 0.70 = 0.12. The error bar extends from approximately 0.78 to 0.85. ### Key Observations * For Human and Claude 3, the "Relational" approach has a higher accuracy than for GPT-4. * For GPT-4, the "Relational" approach has a lower accuracy than for Human and Claude 3. * The "Defaults" approach has a higher accuracy for GPT-4 than for Human and Claude 3. ### Interpretation The bar chart compares the accuracy of "Defaults" and "Relational" approaches across Human, GPT-4, and Claude 3. The data suggests that the "Relational" approach is more effective for Human and Claude 3, while the "Defaults" approach is more effective for GPT-4. This could indicate that GPT-4 benefits more from the default settings, while Human and Claude 3 benefit from a relational approach. The error bars provide a measure of the variability in the data, which should be considered when interpreting the results. </details> Figure 5: Human and LLM accuracy in the Relational condition followup, with Defaults condition performance for reference. Error bars show standard errors. 3.1.5 Takeaways Despite weak performance from many models on our analogical reasoning tasks, GPT-4 and Claude 3 perform well, showing similar patterns to humans in leveraging semantic structure of corresponding domains to solve analogies. However, differences do remain in how they handle semantic structure in the source domain. Humans prefer leveraging semantic structure when a clear pattern exists (evidenced by the Defaults and Random Finals conditions) but can ignore words when structure is lacking (Randoms condition). Models show the former bias but not the latter ability, appearing distracted by random lexical items. Nevertheless, model results increasingly resemble human subjects, suggesting larger models may close this gap. Furthermore, qualitative differences exist even between the best models. GPT-4 and Claude 3 match human performance in the Defaults condition, but when the structure is generalized from $2× 2$ to $2× n$ in the Relational followup, making a right-hand-only strategy unworkable, Claude 3 maintains human-level performance while GPT-4 drops significantly. Despite limited public information, it’s notable that models produced using presumably similar approaches can exhibit meaningfully different behavioral patterns. 3.2 Mapping Semantic Content The Semantic Structure experiment, which presented subjects with source and target domains with corresponding semantic structure (i.e., with corresponding relations between terms), provides insight into the relative bias of human subjects and models to transfer this structure across domains. The Semantic Content experiment modifies the tasks to investigate the extent to which human subjects and models can transfer elements of the linguistic meaning of terms from one domain to another. To achieve this, we ensure that elements of the target domain directly depend on properties of corresponding source domain elements, requiring knowledge of the source domain terms’ meaning for perfect performance. As in the Semantic Structure experiment, source and target domains are paired such that patterns in the target domain mirror those in the source domain. Together, these experiments compare the subject’s ability and tendency to use a structure-mapping approach. Four tasks are generated, encoding either one or two dimensions of variation and either involving or not involving numeric reasoning (see Table 4). | Categorial: Right-hand terms are single characters corresponding to a Categorial property of the left-hand terms. | chicken => ! spider => ! cat => * horse => * ant => ! dog => * bee => ! human => | | --- | --- | | Multi-Attribute: Right-hand terms are a sequence of several characters that vary according to two properties of the left-hand terms. | grandfather => ! grandmother => * mother => * * father => ! ! brother => ! ! ! sister => | | Numeric: Right-hand terms are a sequence of a single repeated character, with the number of repetitions corresponding to a numeric property of the left-hand terms. | chicken => * * human => * * dog => * * * * spider => * * * * * * * * cat => * * * * horse => * * * * bee => | | Numeric Multi-Attribute: Right-hand terms are a sequence of a repeated character, with the number of repetitions corresponding to a numeric property of the left-hand terms and the character corresponding to a Categorial property. | horse => * * * * cat => * * * * ant => ! ! ! ! ! ! bee => ! ! ! ! ! ! chicken => ! ! spider => ! ! ! ! ! ! ! ! dog => * * * * human => | Table 4: The conditions of the Semantic Content experiment. <details> <summary>extracted/5679376/Images/exp2_new.png Details</summary>  ### Visual Description ## Bar Chart: Accuracy Comparison ### Overview The image is a bar chart comparing the accuracy of humans, GPT-4, and Claude 3 across three attribute types: single attribute, numeric, and categorical. The chart displays accuracy on the y-axis, ranging from 0.0 to 1.0, and the three entities (Human, GPT-4, Claude 3) on the x-axis. Error bars are included on each bar. ### Components/Axes * **Y-axis:** "Accuracy", ranging from 0.0 to 1.0 in increments of 0.2. * **X-axis:** Categorical labels: "Human", "GPT-4", "Claude 3". * **Legend:** Located in the top-right corner, it identifies the bar colors: * White with black outline: "Single attribute" * Light Red: "Numeric" * Light Blue: "Categorial" ### Detailed Analysis Here's a breakdown of the accuracy values for each entity and attribute type, including trend descriptions: * **Human:** * Single attribute: Accuracy is approximately 0.73, with an error bar extending from approximately 0.68 to 0.80. * Numeric: Accuracy is approximately 0.50, with an error bar extending from approximately 0.45 to 0.58. * Categorial: Accuracy is approximately 0.45, with an error bar extending from approximately 0.35 to 0.53. * **GPT-4:** * Single attribute: Accuracy is approximately 0.70, with an error bar extending from approximately 0.68 to 0.73. * Numeric: Accuracy is approximately 0.25, with an error bar extending from approximately 0.23 to 0.28. * Categorial: Accuracy is approximately 0.65, with an error bar extending from approximately 0.63 to 0.68. * **Claude 3:** * Single attribute: Accuracy is approximately 0.56, with an error bar extending from approximately 0.54 to 0.58. * Numeric: Accuracy is approximately 0.45, with an error bar extending from approximately 0.43 to 0.48. * Categorial: Accuracy is approximately 0.58, with an error bar extending from approximately 0.55 to 0.61. ### Key Observations * For humans, single attribute accuracy is the highest, followed by numeric, and then categorical. * GPT-4 shows the highest accuracy for single attribute and categorical, but significantly lower accuracy for numeric attributes. * Claude 3 has relatively similar accuracy for all three attribute types, with categorical being slightly higher. * GPT-4 has the largest difference in accuracy between attribute types. ### Interpretation The bar chart illustrates the performance of humans, GPT-4, and Claude 3 on different types of attributes. Humans excel at single attribute tasks, while GPT-4 struggles with numeric attributes. Claude 3 demonstrates more consistent performance across all attribute types. The error bars provide an indication of the variability in the accuracy measurements. The data suggests that the models have different strengths and weaknesses depending on the nature of the task. </details> Figure 6: Human and model accuracy by condition in the Semantic Content experiment. Error bars show standard errors. Results for human subjects, GPT-4, and Claude 3 are shown in Figure 6 (other tested models attain much lower accuracy as before). 3.2.1 Human Performance Continues to be Robust Human subjects perform robustly and consistently, as in the previous experiment. Human accuracy ranges from 0.4 to 0.8 across conditions, comparable to the earlier Semantic Structure experiment. As expected, subjects generally describe their strategy as relating properties of the left-hand terms to their representations on the right-hand side. 3.2.2 Claude 3 Matches Human Performance Stably Across Conditions Claude 3 matches human performance stably across the different conditions of the Semantic Content experiment with its accuracy falling into a comparable range of 0.4 to 0.7. The model exhibits marginally better performance in the Multi-Attribute condition and marginally worse performance in the remaining three. These differences are insignificant across all conditions, which covers the Categorial (coef = -0.8109, z = -1.879, p = 0.060), Multi-Attribute (coef = 0.6206, z = 1.478, p = 0.140), Numeric (coef = -0.1788, z = -0.439, p = 0.661), and Numeric Multi-Attribute (coef = -0.2009, z = -0.484, p = 0.629) conditions. Therefore, Claude 3 performs as well as human subjects across all conditions of this experiment. 3.2.3 GPT-4 Lags Human Subjects on Numeric Reasoning GPT-4 achieves good results in the Categorial and Multi-Attribute conditions, with mean accuracies of approximately 0.7 in both (compared to 0.7 and 0.4 respectively for human subjects). GPT-4 is not significantly worse than humans in the Categorial condition (coef = -0.3927, z = -0.889, p = 0.374). GPT-4 significantly outperforms human subjects in the Multi-Attribute condition (coef = 1.0624, z = 2.429, p = 0.015). However, its accuracy drops to 0.2-0.3 in the remaining conditions and we find that GPT-4 is significantly worse than humans in both the Numeric (coef = -1.4781, z = -3.321, p = 0.001) and Numeric Multi-Attribute conditions (coef = -0.9694, z = -2.185, p = 0.029). In these conditions, GPT-4 fails to correctly relate the number of characters in a response to the numeric property of the object (see Table 7 for an illustrative example). GPT-4’s failure to reason about the number of characters in the expected way is further observed in the sanity check shown in Table 8 of the Appendix, even when the model is not required to relate a property of a word to its representation. 3.2.4 Human Performance Drops in Compositional Conditions, But Models Remain Constant When comparing the performance of a subject in a non-compositional (single-attribute) condition to the corresponding compositional (multi-attribute) version, we observe some decrease in performance for human subjects but not for models (note that this surprising result is subject to alternative explanations, addressed in the discussion below). The accuracy of human subjects drops from approximately 0.7 to approximately 0.4 when comparing the Categorial condition to the corresponding compositional version (the Multi-Attribute condition). A simple effects analysis confirms that this decline is significant (coef = -1.2267, z = -3.091, p = 0.002). We see a non-significant decrease in accuracy for human subjects when comparing the Numeric condition to its compositional counterpart, with performance dropping from approximately 0.6 to approximately 0.5 (coef = -0.3795, z = -1.028, p = 0.304). By contrast, we do not find either model to be significantly worse in compositional conditions than non-compositional ones. In fact, GPT-4 exhibits a slight improvement in the compositional conditions, though this change is statistically insignificant for both the Multi-Attribute condition relative to the Categorial condition (coef = 0.2281, z = 0.477, p = 0.634) and for the Numeric Multi-Attribute condition relative to the Numeric condition (coef = 0.1292, z = 0.254, p = 0.799). For Claude 3 we similarly find the differences to be insignificant for the Multi-Attribute condition relative to the Categorial condition (coef = 0.2049, z = 0.452, p = 0.651) and for the Numeric Multi-Attribute condition relative to the Numeric condition (coef = -0.4013, z = -0.893, p = 0.372). 3.2.5 Takeaways The Semantic Content experiment confirms that human subjects perform robustly and flexibly across diverse task variations. Claude 3 matches human performance in all conditions, indicating it shares humans’ tendency to use the source domain’s semantic content when completing target domains. While GPT-4’s poor performance in numeric conditions is notable, it reflects a failure in numeric reasoning rather than a difference in analogical reasoning. We find evidence of decreased human performance, but not model performance, in compositional conditions, contrasting with some existing research [33]. However, other factors may be at play. Models’ negative compositionality effect may be masked by a positive effect, such as increased available information: when the target domain represents two source domain properties, models may more easily recognize the encoding of source domain properties. Human subjects may benefit less from this competing effect if they do not struggle to observe this information encoding. 4 Discussion Our results show that the best-performing LLMs are able to successfully complete many analogical reasoning tasks with human-level accuracy using novel stimuli not present in their training data. They also show that there remain meaningful differences in how such analogies are processed, evidenced by differences in how humans and models respond to distracting or misleading information. However, we observe a clear trend: more recent models come increasingly close to matching human performance across our tasks. In particular, Claude 3, the most recently-released model we test, exhibits impressively robust performance across most task variations, even closing the gap with humans in some test conditions in which its predecessor (GPT-4) exhibited limitations (such as the Relational task version in which mapping from the source domain must be used for success). Together, these results raise questions about the ability of LLMs and similar models to serve as candidate cognitive models, which we discuss briefly below. 4.1 Evaluating the Competence of LLMs The breadth of Claude 3’s success in our tasks is noteworthy. It suggests that state-of-the-art LLMs can broadly match human performance not only in formal analogical reasoning tasks, as suggested by Webb et al. [4], but also in tasks that require mapping semantic information across linguistic and non-linguistic domains. As such, our results weigh against a long-standing view in cognitive science, according to which connectionist models without a built-in symbolic component are constitutively limited in their ability to robustly handle analogical reasoning tasks [7]. They also inform discussions of whether LLMs possess “functional” linguistic competence, in addition to “formal” linguistic competence [3]. Further work is needed to characterize the precise mechanism that LLMs are using to solve these tasks; it is possible–though increasingly unlikely given the robustness of the behavioral results–that success is due to a myriad of heuristics rather than a systematic analogical reasoning process. Even so, evidence of LLMs completing analogical reasoning tasks in domains designed to involve linguistic structure-mapping, in addition to tasks over abstract symbols, runs counter to the claim that LLMs are capable of formal but not functional linguistic competence. There remain examples of LLMs performing much worse than humans on analogical reasoning tasks [10], which must be reconciled with our results. Here the competence-performance distinction, originally introduced by Noam Chomsky [34], can be usefully applied to the evaluation of LLMs [35, 2, 36]. This distinction allows researchers to theorize about the abstract computational principles governing cognition separately from the “noise” introduced by performance factors. In humans, it is generally assumed that there is a double dissociation between performance and competence: neither success nor failure on a task designed to measure a particular capacity can always be taken as conclusive evidence that subjects have or lack that capacity, due to auxiliary factors affecting task performance. When it comes to LLMs, by contrast, the distinction is typically applied in a single direction: human-like performance on benchmarks is often explained away by reliance on shallow heuristics [37] and/or lack of construct validity [38], while sub-human performance is often taken as reliable evidence of lack of competence. However, LLM performance can also be negatively affected by strong auxiliary task demands [39] and mismatched conditions in comparisons with human subjects [40]. These are compelling reasons to apply the dissociation in both directions to LLMs as well. From this perspective, our results offer evidence to support both sides of the present debate about whether LLMs possess human-level analogical reasoning (see Webb et al. [15], Mitchell et al. [10], and Hodel et al. [16]). Supporting the argument of Webb et al. [15] that deficiencies in capabilities other than analogical reasoning can explain poor model performance in some tasks, we find that GPT-4’s failure in the numeric conditions of our Semantic Content experiment may be due to a deficiency in counting ability. However, contrary to Webb et al. [4], who report impressive analogical reasoning in both GPT-3 and GPT-4, we do find a qualitative difference in the performance of these two models, with GPT-3 performing quite poorly on our tasks. Among the models tested, only GPT-4 and Claude 3 produce results that merit detailed comparison with human subjects. This suggests that claims of human-level performance of LLMs on analogical reasoning tasks may have been premature and might have relied on insufficiently challenging tasks. However, other differences we observe between human subjects and LLMs across task variations are not subject to an auxiliary task demand explanation and suggest that the underlying mechanisms of analogical reasoning in these systems may differ from that in humans. Importantly, these differences persist even in our best performing model, Claude 3. For instance, Claude 3 responds differently than human subjects when some or all words in the target domain are replaced with random words, indicating that they may use distinct strategies for identifying and leveraging relational similarities between source and target domains. Furthermore, Claude 3 remains more sensitive than human subjects to the ordering of elements within domains, which is difficult to explain if LLMs are using a generalizable symbolic working memory approach. Collectively, these patterns bear on the larger question of how we should arbitrate disputes about competence in machine-human comparisons. On the one hand, it seems reasonable to assume that any system that can reliably achieve success at or above human level on experiments like ours–without relying on memorization and other confounds–should be considered competent at analogical reasoning through structure-mapping. On the other hand, we should be open to the possibility that such competence may be implemented differently in LLMs and humans. The question of whether we require human-likeness of the mechanism to declare human-level “competence” is ultimately not empirical, but rather demands philosophical consensus among the scientific community around our ultimate goals and metrics for achieving them. 4.2 Analogy in Human(-like) Learning and Bootstrapping Unlike previous research comparing analogical reasoning in human subjects and LLMs, our tasks involve transferring semantic structure and content from source to target domains, rather than reasoning over abstract symbols. Our experiments thus investigate whether LLMs’ analogical reasoning resembles that of human subjects in a manner pertinent to its purportedly central role in broader cognition. Following Gentner [11], emphasis has been placed on relational similarity, rather than just feature similarity, in mapping from a familiar source to a foreign target domain during analogical reasoning to allow for the flexible transfer of knowledge [41, 42, 43]. This conception allows analogical reasoning to play a fundamental role in human cognition, supporting the emergence of diverse cognitive abilities via “bootstrapping” [44, 45, 46]. In bootstrapping, two cognitive processes mutually support each other’s development. In Gentner’s Structure-Mapping Theory (SMT), language development and structure-mapping-based analogical reasoning are hypothesized to co-develop, with structure-mapping developing the necessary relational reasoning to model language-world relations, and language acquisition in turn developing symbolic reasoning capacities that amplify structure-mapping abilities. Consequently, analogical reasoning is seen as a central cognitive phenomenon of interest. The success of some LLMs in many of our tasks suggests that the most advanced models may be capable of employing a structure-mapping based approach to analogical reasoning, in which relations in the source domain are used to constrain and guide reasoning about relations in the target domain. This raises the possibility that a bootstrapping cycle between language development and analogical reasoning in humans, as proposed by Gentner [44], may be paralleled in language models. The emergence of such competence from training primarily on text prediction would yield new hypotheses about the emergence of analogical reasoning as a central cognitive faculty from generic learning mechanisms (possibly combined with the unique pressures of language acquisition). However, the mixed success of LLMs and the significant differences from humans in certain conditions underscore the need for continued research to test the robustness of any conclusion that analogical reasoning in LLMs closely matches that of human subjects. As LLM outputs continue to converge toward human responses–an expected product of the language modelling objective–it is crucial to develop novel tasks that examine analogical reasoning ability and are not attested in the training data. While our task allows for clear discrimination between human performance and that of most models prior to Claude 3, further differences in analogical reasoning patterns between humans and Claude 3 likely exist beyond those revealed by our tests. More granular testing would help clarify the extent of the remaining discrepancies between humans and the most advanced LLMs, and much further work is required to verify the hypothesis that language models parallel the bootstrapping cycle between language development and analogical reasoning in humans. The proprietary nature of leading LLMs like Claude 3 unfortunately limits our ability to directly investigate the features that may explain the emergence of a response pattern largely mirroring that of human subjects. However, increasingly sophisticated open-weights models are being released, which may allow for interpretability work to analyze the internal mechanisms of a model and shed light on the underlying mechanisms that enable advanced LLMs to exhibit impressive analogical reasoning abilities in many tasks. 5 Acknowledgments This work was supported in part by NIH NIGMS COBRE grant #5P20GM10364510. References - [1] A. Srivastava, A. Rastogi, A. Rao, A. A. M. S. E. al., Beyond the imitation game: Quantifying and extrapolating the capabilities of language models (2023). arXiv:2206.04615. - [2] E. Pavlick, Symbols and grounding in large language models, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 381 (2251) (Jun. 2023). doi:10.1098/rsta.2022.0041. URL http://dx.doi.org/10.1098/rsta.2022.0041 - [3] K. Mahowald, A. A. Ivanova, I. A. Blank, N. Kanwisher, J. B. Tenenbaum, E. Fedorenko, Dissociating language and thought in large language models (2023). arXiv:2301.06627. - [4] T. Webb, K. J. Holyoak, H. Lu, Emergent analogical reasoning in large language models, Nature Human Behaviour 7 (9) (2023) 1526––1541. - [5] S. J. Han, K. Ransom, A. Perfors, C. Kemp, Inductive reasoning in humans and large language models (2023). arXiv:2306.06548. - [6] X. Hu, S. Storks, R. L. Lewis, J. Chai, In-context analogical reasoning with pre-trained language models (2023). arXiv:2305.17626. - [7] M. Mitchell, Abstraction and analogy-making in artificial intelligence, Annals of the New York Academy of Sciences 1505 (1) (2021) 79–101. arXiv:https://nyaspubs.onlinelibrary.wiley.com/doi/pdf/10.1111/nyas.14619, doi:https://doi.org/10.1111/nyas.14619. URL https://nyaspubs.onlinelibrary.wiley.com/doi/abs/10.1111/nyas.14619 - [8] K. J. Holyoak, D. Gentner, B. N. Kokinov, Introduction: The Place of Analogy in Cognition, in: The Analogical Mind: Perspectives from Cognitive Science, The MIT Press, 2001. arXiv:https://direct.mit.edu/book/chapter-pdf/2323335/9780262316057\_caa.pdf, doi:10.7551/mitpress/1251.003.0003. URL https://doi.org/10.7551/mitpress/1251.003.0003 - [9] D. R. Hofstadter, Epilogue: Analogy as the Core of Cognition, in: The Analogical Mind: Perspectives from Cognitive Science, The MIT Press, 2001. arXiv:https://direct.mit.edu/book/chapter-pdf/2323391/9780262316057\_cao.pdf, doi:10.7551/mitpress/1251.003.0020. URL https://doi.org/10.7551/mitpress/1251.003.0020 - [10] M. Lewis, M. Mitchell, Using counterfactual tasks to evaluate the generality of analogical reasoning in large language models (2024). arXiv:2402.08955. - [11] D. Gentner, Structure-mapping: A theoretical framework for analogy*, Cognitive Science 7 (2) (1983) 155–170. arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1207/s15516709cog0702_3, doi:https://doi.org/10.1207/s15516709cog0702\_3. URL https://onlinelibrary.wiley.com/doi/abs/10.1207/s15516709cog0702_3 - [12] K. Erk, Towards a semantics for distributional representations, in: A. Koller, K. Erk (Eds.), Proceedings of the 10th International Conference on Computational Semantics (IWCS 2013) – Long Papers, Association for Computational Linguistics, Potsdam, Germany, 2013, pp. 95–106. URL https://aclanthology.org/W13-0109 - [13] G. Boleda, Distributional semantics and linguistic theory, CoRR abs/1905.01896 (2019). arXiv:1905.01896. URL http://arxiv.org/abs/1905.01896 - [14] L. Gleitman, C. Fisher, 6 universal aspects of word learning, in: J. A. McGilvray (Ed.), The Cambridge Companion to Chomsky, Cambridge University Press, 2005, p. 123. - [15] T. Webb, K. J. Holyoak, H. Lu, Evidence from counterfactual tasks supports emergent analogical reasoning in large language models (2024). arXiv:2404.13070. - [16] D. Hodel, J. West, Response: Emergent analogical reasoning in large language models (2024). arXiv:2308.16118. - [17] S. French, A model-theoretic account of representation (or, i don’t know much about art…but i know it involves isomorphism), Philosophy of Science 70 (5) (2003) 1472–1483. doi:10.1086/377423. - [18] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, D. Amodei, Language models are few-shot learners (2020). arXiv:2005.14165. - [19] OpenAI, J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, R. Avila, I. Babuschkin, S. Balaji, V. Balcom, P. Baltescu, H. Bao, M. Bavarian, J. Belgum, I. Bello, J. Berdine, G. Bernadett-Shapiro, C. Berner, L. Bogdonoff, O. Boiko, M. Boyd, A.-L. Brakman, G. Brockman, T. Brooks, M. Brundage, K. Button, T. Cai, R. Campbell, A. Cann, B. Carey, C. Carlson, R. Carmichael, B. Chan, C. Chang, F. Chantzis, D. Chen, S. Chen, R. Chen, J. Chen, M. Chen, B. Chess, C. Cho, C. Chu, H. W. Chung, D. Cummings, J. Currier, Y. Dai, C. Decareaux, T. Degry, N. Deutsch, D. Deville, A. Dhar, D. Dohan, S. Dowling, S. Dunning, A. Ecoffet, A. Eleti, T. Eloundou, D. Farhi, L. Fedus, N. Felix, S. P. Fishman, J. Forte, I. Fulford, L. Gao, E. Georges, C. Gibson, V. Goel, T. Gogineni, G. Goh, R. Gontijo-Lopes, J. Gordon, M. Grafstein, S. Gray, R. Greene, J. Gross, S. S. Gu, Y. Guo, C. Hallacy, J. Han, J. Harris, Y. He, M. Heaton, J. Heidecke, C. Hesse, A. Hickey, W. Hickey, P. Hoeschele, B. Houghton, K. Hsu, S. Hu, X. Hu, J. Huizinga, S. Jain, S. Jain, J. Jang, A. Jiang, R. Jiang, H. Jin, D. Jin, S. Jomoto, B. Jonn, H. Jun, T. Kaftan, Łukasz Kaiser, A. Kamali, I. Kanitscheider, N. S. Keskar, T. Khan, L. Kilpatrick, J. W. Kim, C. Kim, Y. Kim, J. H. Kirchner, J. Kiros, M. Knight, D. Kokotajlo, Łukasz Kondraciuk, A. Kondrich, A. Konstantinidis, K. Kosic, G. Krueger, V. Kuo, M. Lampe, I. Lan, T. Lee, J. Leike, J. Leung, D. Levy, C. M. Li, R. Lim, M. Lin, S. Lin, M. Litwin, T. Lopez, R. Lowe, P. Lue, A. Makanju, K. Malfacini, S. Manning, T. Markov, Y. Markovski, B. Martin, K. Mayer, A. Mayne, B. McGrew, S. M. McKinney, C. McLeavey, P. McMillan, J. McNeil, D. Medina, A. Mehta, J. Menick, L. Metz, A. Mishchenko, P. Mishkin, V. Monaco, E. Morikawa, D. Mossing, T. Mu, M. Murati, O. Murk, D. Mély, A. Nair, R. Nakano, R. Nayak, A. Neelakantan, R. Ngo, H. Noh, L. Ouyang, C. O’Keefe, J. Pachocki, A. Paino, J. Palermo, A. Pantuliano, G. Parascandolo, J. Parish, E. Parparita, A. Passos, M. Pavlov, A. Peng, A. Perelman, F. de Avila Belbute Peres, M. Petrov, H. P. de Oliveira Pinto, Michael, Pokorny, M. Pokrass, V. H. Pong, T. Powell, A. Power, B. Power, E. Proehl, R. Puri, A. Radford, J. Rae, A. Ramesh, C. Raymond, F. Real, K. Rimbach, C. Ross, B. Rotsted, H. Roussez, N. Ryder, M. Saltarelli, T. Sanders, S. Santurkar, G. Sastry, H. Schmidt, D. Schnurr, J. Schulman, D. Selsam, K. Sheppard, T. Sherbakov, J. Shieh, S. Shoker, P. Shyam, S. Sidor, E. Sigler, M. Simens, J. Sitkin, K. Slama, I. Sohl, B. Sokolowsky, Y. Song, N. Staudacher, F. P. Such, N. Summers, I. Sutskever, J. Tang, N. Tezak, M. B. Thompson, P. Tillet, A. Tootoonchian, E. Tseng, P. Tuggle, N. Turley, J. Tworek, J. F. C. Uribe, A. Vallone, A. Vijayvergiya, C. Voss, C. Wainwright, J. J. Wang, A. Wang, B. Wang, J. Ward, J. Wei, C. Weinmann, A. Welihinda, P. Welinder, J. Weng, L. Weng, M. Wiethoff, D. Willner, C. Winter, S. Wolrich, H. Wong, L. Workman, S. Wu, J. Wu, M. Wu, K. Xiao, T. Xu, S. Yoo, K. Yu, Q. Yuan, W. Zaremba, R. Zellers, C. Zhang, M. Zhang, S. Zhao, T. Zheng, J. Zhuang, W. Zhuk, B. Zoph, Gpt-4 technical report (2024). arXiv:2303.08774. - [20] S. Biderman, H. Schoelkopf, Q. Anthony, H. Bradley, K. O’Brien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff, A. Skowron, L. Sutawika, O. van der Wal, Pythia: A suite for analyzing large language models across training and scaling (2023). arXiv:2304.01373. - [21] Anthropic, Claude-2 language model, https://www.anthropic.com/index/claude-2, accessed: 2023-11-06 (2023). - [22] Anthropic, The Claude 3 Model Family: Opus, Sonnet, Haiku, https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3.pdf, accessed: 2024-05-01 (2024). - [23] Technology Innovation Institute, Falcon language model, https://falconllm.tii.ae/falcon.html, accessed: 2023-10-31 (2023). - [24] B. Liu, L. Ding, L. Shen, K. Peng, Y. Cao, D. Cheng, D. Tao, Diversifying the mixture-of-experts representation for language models with orthogonal optimizer (2023). arXiv:2310.09762. - [25] S. Glover, P. Dixon, Likelihood ratios: A simple and flexible statistic for empirical psychologists, Psychonomic Bulletin & Review 11 (5) (2004) 791–806. doi:10.3758/BF03196706. URL https://doi.org/10.3758/BF03196706 - [26] P. Carvalho, R. Goldstone, Category structure modulates interleaving and blocking advantage in inductive category acquisition, in: Proceedings of the 34th Annual Conference of the Cognitive Science Society, 2012, pp. 186–191. - [27] J. Russin, E. Pavlick, M. J. Frank, Human curriculum effects emerge with in-context learning in neural networks (2024). arXiv:2402.08674. - [28] D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, J. Steinhardt, Measuring massive multitask language understanding (2021). arXiv:2009.03300. - [29] J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, D. Amodei, Scaling laws for neural language models (2020). arXiv:2001.08361. - [30] S. M. Xie, A. Raghunathan, P. Liang, T. Ma, An explanation of in-context learning as implicit bayesian inference (2022). arXiv:2111.02080. - [31] Y. Zhang, F. Zhang, Z. Yang, Z. Wang, What and how does in-context learning learn? bayesian model averaging, parameterization, and generalization (2023). arXiv:2305.19420. - [32] A. Raventós, M. Paul, F. Chen, S. Ganguli, Pretraining task diversity and the emergence of non-bayesian in-context learning for regression, in: A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, S. Levine (Eds.), Advances in Neural Information Processing Systems, Vol. 36, Curran Associates, Inc., 2023, pp. 14228–14246. URL https://proceedings.neurips.cc/paper_files/paper/2023/file/2e10b2c2e1aa4f8083c37dfe269873f8-Paper-Conference.pdf - [33] O. Press, M. Zhang, S. Min, L. Schmidt, N. A. Smith, M. Lewis, Measuring and narrowing the compositionality gap in language models (2023). arXiv:2210.03350. - [34] N. Chomsky, Aspects of the Theory of Syntax, 50th Edition, The MIT Press, 1965. URL http://www.jstor.org/stable/j.ctt17kk81z - [35] C. Firestone, Performance vs. competence in human-machine comparisons, Proc Natl Acad Sci U S A 117 (43) (2020) 26562–26571. - [36] E. Pavlick, Semantic structure in deep learning (january 2022), Annual Review of Linguistics 8 (2022) 447–471. URL http://dx.doi.org/10.1146/annurev-linguistics-031120-122924 - [37] R. T. McCoy, S. Yao, D. Friedman, M. Hardy, T. L. Griffiths, Embers of autoregression: Understanding large language models through the problem they are trained to solve (2023). arXiv:2309.13638. - [38] T. Ullman, Large language models fail on trivial alterations to theory-of-mind tasks (2023). arXiv:2302.08399. - [39] J. Hu, M. C. Frank, Auxiliary task demands mask the capabilities of smaller language models (2024). arXiv:2404.02418. - [40] A. K. Lampinen, Can language models handle recursively nested grammatical structures? a case study on comparing models and humans (2023). arXiv:2210.15303. - [41] H. Gust, U. Krumnack, K.-U. Kühnberger, A. Schwering, Analogical reasoning: A core of cognition., KI 22 (2008) 8–12. - [42] G. S. Halford, W. H. Wilson, S. Phillips, Relational knowledge: the foundation of higher cognition, Trends in Cognitive Sciences 14 (11) (2010) 497–505. doi:https://doi.org/10.1016/j.tics.2010.08.005. URL https://www.sciencedirect.com/science/article/pii/S1364661310002020 - [43] K. J. Holyoak, 234 Analogy and Relational Reasoning, in: The Oxford Handbook of Thinking and Reasoning, Oxford University Press, 2012. arXiv:https://academic.oup.com/book/0/chapter/293248246/chapter-ag-pdf/44513038/book\_34559\_section\_293248246.ag.pdf, doi:10.1093/oxfordhb/9780199734689.013.0013. URL https://doi.org/10.1093/oxfordhb/9780199734689.013.0013 - [44] D. Gentner, Bootstrapping the mind: Analogical processes and symbol systems, Cognitive Science 34 (5) (2010) 752–775. arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/j.1551-6709.2010.01114.x, doi:https://doi.org/10.1111/j.1551-6709.2010.01114.x. URL https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1551-6709.2010.01114.x - [45] S. Carey, Bootstrapping & the origin of concepts, Daedalus 133 (1) (2004) 59–68. arXiv:https://direct.mit.edu/daed/article-pdf/133/1/59/1828762/001152604772746701.pdf, doi:10.1162/001152604772746701. URL https://doi.org/10.1162/001152604772746701 - [46] S. Carey, The Origin of Concepts, Oxford University Press, 2009. doi:10.1093/acprof:oso/9780195367638.001.0001. URL https://doi.org/10.1093/acprof:oso/9780195367638.001.0001 Appendix A Statistical outputs and supplementary figures A.1 Regression results, Semantic Structure experiment We perform a logistic regression with the outcome variable being the raw score (a 0 or 1 for each question). The predictor variables are condition and subject type (restricted to human subjects and GPT-4 only, or human subjects and Claude 3 only). The regression is performed with and without interactions: Without interactions: $smf.logit(formula=respondent\_scores\sim C(subject\_type,Treatment(reference=% human))+C(quiz\_class,Treatment(reference=permuted\_questions)),data=all\_% subjects\_df,).fit(maxiter=1000,method=bfgs)$ With interactions: $smf.logit(formula=respondent\_scores\sim C(subject\_type,Treatment(reference=% human))*C(quiz\_class,Treatment(reference=permuted\_questions)),data=all\_% subjects\_df,).fit(maxiter=1000,method=bfgs)$ The significance of including the interaction between predictors is assessed with a likelihood ratio test with the associated p-value calculated as follows: $p=chi2.sf(lik\_ratio,degfree)$ , with 7 degrees of freedom. The likelihood ratio in the above formula is calculated as follows: $lik\_ratio=degfree*(res\_subjXclass.llf-res\_subjplusclass.llf)$ . In the above, res_subjXclass and res_subjplusclass are the regression outputs with and without interactions respectively. For both comparisons (human subjects compared to GPT-4 and human subjects compared to Claude 3), we find a significant improvement in model fit when interactions between the subject type and experiment condition are included.A likelihood ratio test shows that including interactions between subject type and experiment condition leads to a significantly better fit of the model ( $chi^{2}(7)=115.1871,p<0.001$ ). For the comparison between Claude 3 and human subjects, we again find a significant negative effect of the subject type being Claude 3 when interactions are not included (coef = -0.8706, z = -5.608, p < 0.001) and find that subject type - condition interactions are significant ( $chi^{2}(7)=173.6511,p<0.001$ ). These results are consistent with the observation that the two models exhibit variable performance across conditions, and indicate that the overall performance gap to human subjects is driven by low model accuracy in certain conditions. Simple effects analysis is used below to assess the effect of subject type in particular conditions and groups thereof. Regression outputs are shown in Figure 7 and 8. <details> <summary>x1.png Details</summary>  ### Visual Description ## Logit Regression Results ### Overview The image presents a series of Logit Regression Results, each analyzing the effect of subject type (GPT-4 vs. default) and treatment conditions on respondent scores. The results include model statistics, coefficients, standard errors, z-values, p-values, and confidence intervals. Each analysis focuses on a different set of treatment conditions. ### Components/Axes Each Logit Regression Result section contains the following components: * **Model Information**: * Dep. Variable: respondent scores * No. Observations: Varies (e.g., 1012, 300, 152, 146, 260, 128, 132) * Model: MLE, Df Model * Method: Logit, Df Residuals * Date: Wed, 17 Apr 2024 * Time: 18:34:36 * Pseudo R-squ.: Varies (e.g., 0.2311, 0.01016, 0.05599, 0.1021, 0.1873, 0.1794, 0.2517) * Log-Likelihood: Varies (e.g., -505.58, -148.60, -186.11, -81.628, -166.40, -71.979, -54.70) * Covariance Type: nonrobust * LLR p-value: Varies (e.g., 8.581e-06, 0.08068, 2.622e-06, 1.642e-06, 2.910e-15, 2.012e-08, 1.308e-09) * **Coefficient Table**: * coef: Coefficient estimate * std err: Standard error of the coefficient * z: z-statistic * P>|z|: p-value * [0.025 0.975]: 95% Confidence Interval * **Categorical Variables**: * C(subject type, Treatment[reference=1])[T.GPT-4]: Compares GPT-4 to the reference group. * C(quiz class, Treatment[reference=4])[T.distracted], [T.only the], [T.permuted pairs], [T.random finals], [T.randoms permuted]: Compares different quiz class treatments to the reference group. * **Effect of Subject**: * Effect of subject with only the conditions: Specifies the treatment conditions being analyzed. * **Optimization Information**: * Optimization terminated successfully. * Current function value: Varies * Iterations: Varies * Function evaluations: Varies * Gradient evaluations: Varies ### Detailed Analysis or ### Content Details Here's a breakdown of the key data points from each Logit Regression Result: 1. **First Result (No specific conditions)**: * No. Observations: 1012 * Pseudo R-squ.: 0.2311 * Log-Likelihood: -505.58 * LLR p-value: 8.581e-06 * C(subject type, Treatment[reference=1])[T.GPT-4]: coef = 1.3501, std err = 0.300, z = 4.502, P>|z| = 0.000, [0.025 0.975] = [0.762, 1.938] * C(quiz class, Treatment[reference=4])[T.distracted]: coef = 0.5937, std err = 0.392, z = 1.514, P>|z| = 0.130, [0.025 0.975] = [-0.175, 1.362] * C(quiz class, Treatment[reference=4])[T.only the]: coef = 0.7290, std err = 0.460, z = 1.520, P>|z| = 0.129, [0.025 0.975] = [-0.171, 1.629] * C(quiz class, Treatment[reference=4])[T.permuted pairs]: coef = 0.5306, std err = 0.467, z = 1.136, P>|z| = 0.256, [0.025 0.975] = [-0.385, 1.446] * C(quiz class, Treatment[reference=4])[T.random finals]: coef = 1.2736, std err = 0.638, z = 1.996, P>|z| = 0.046, [0.025 0.975] = [0.025, 2.522] * C(quiz class, Treatment[reference=4])[T.randoms permuted]: coef = -0.0148, std err = 0.585, z = -0.025, P>|z| = 0.980, [0.025 0.975] = [-1.162, 1.132] * C(subject type, Treatment[reference=1])[T.GPT-4]:C(quiz class, Treatment[reference=4])[T.defaults]: coef = -0.4530, std err = 0.608, z = -0.745, P>|z| = 0.456, [0.025 0.975] = [-1.645, 0.739] * C(subject type, Treatment[reference=1])[T.GPT-4]:C(quiz class, Treatment[reference=4])[T.distracted]: coef = -0.5393, std err = 0.518, z = -1.041, P>|z| = 0.298, [0.025 0.975] = [-1.554, 0.476] * C(subject type, Treatment[reference=1])[T.GPT-4]:C(quiz class, Treatment[reference=4])[T.only the]: coef = 0.4354, std err = 0.659, z = 0.661, P>|z| = 0.509, [0.025 0.975] = [-0.856, 1.727] * C(subject type, Treatment[reference=1])[T.GPT-4]:C(quiz class, Treatment[reference=4])[T.permuted pairs]: coef = -1.3833, std err = 0.557, z = -2.481, P>|z| = 0.013, [0.025 0.975] = [-2.474, -0.292] * C(subject type, Treatment[reference=1])[T.GPT-4]:C(quiz class, Treatment[reference=4])[T.random finals]: coef = 2.4663, std err = 0.667, z = 3.696, P>|z| = 0.000, [0.025 0.975] = [1.159, 3.773] * C(subject type, Treatment[reference=1])[T.GPT-4]:C(quiz class, Treatment[reference=4])[T.randoms permuted]: coef = 1.2907, std err = 0.686, z = 1.896, P>|z| = 0.058, [0.025 0.975] = [0.044, 2.644] 2. **Second Result (distracted, permuted pairs)**: * No. Observations: 300 * Pseudo R-squ.: 0.01016 * Log-Likelihood: -148.60 * LLR p-value: 0.08068 * C(subject type, Treatment[reference=1])[T.GPT-4]: coef = -0.5132, std err = 0.233, z = -2.203, P>|z| = 0.028, [0.025 0.975] = [-0.971, -0.056] 3. **Third Result (distracted, permuted pairs)**: * No. Observations: 298 * Pseudo R-squ.: 0.05599 * Log-Likelihood: -186.11 * LLR p-value: 2.622e-06 * C(subject type, Treatment[reference=1])[T.GPT-4]: coef = -1.1664, std err = 0.256, z = -4.557, P>|z| = 0.000, [0.025 0.975] = [-1.668, -0.665] 4. **Fourth Result (permuted pairs)**: * No. Observations: 152 * Pseudo R-squ.: 0.03185 * Log-Likelihood: -100.46 * LLR p-value: 0.01015 * C(subject type, Treatment[reference=1])[T.GPT-4]: coef = -0.8564, std err = 0.338, z = -2.536, P>|z| = 0.011, [0.025 0.975] = [-1.518, -0.195] 5. **Fifth Result (only the)**: * No. Observations: 146 * Pseudo R-squ.: 0.1021 * Log-Likelihood: -81.628 * LLR p-value: 1.642e-06 * C(subject type, Treatment[reference=1])[T.GPT-4]: coef = -1.6764, std err = 0.422, z = -3.979, P>|z| = 0.000, [0.025 0.975] = [-2.508, -0.851] 6. **Sixth Result (random permuted pairs, randoms, random finals)**: * No. Observations: 152 * Pseudo R-squ.: 0.0004850 * Log-Likelihood: -51.147 * LLR p-value: 0.8237 * C(subject type, Treatment[reference=1])[T.GPT-4]: coef = 0.1178, std err = 0.223, z = 0.529, P>|z| = 0.597, [0.025 0.975] = [-0.319, 0.554] 7. **Seventh Result (randoms, random permuted pairs)**: * No. Observations: 260 * Pseudo R-squ.: 0.1873 * Log-Likelihood: -166.40 * LLR p-value: 2.910e-15 * C(subject type, Treatment[reference=1])[T.GPT-4]: coef = -2.2186, std err = 0.301, z = -7.383, P>|z| = 0.000, [0.025 0.975] = [-2.808, -1.630] 8. **Eighth Result (random finals)**: * No. Observations: 128 * Pseudo R-squ.: 0.1794 * Log-Likelihood: -71.979 * LLR p-value: 2.012e-08 * C(subject type, Treatment[reference=1])[T.GPT-4]: coef = -2.1972, std err = 0.422, z = -5.211, P>|z| = 0.000, [0.025 0.975] = [-3.024, -1.371] 9. **Ninth Result (no conditions specified)**: * No. Observations: 132 * Pseudo R-squ.: 0.2517 * Log-Likelihood: -54.70 * LLR p-value: 1.308e-09 * C(subject type, Treatment[reference=1])[T.GPT-4]: coef = -2.7850, std err = 0.539, z = -5.168, P>|z| = 0.000, [0.025 0.975] = [-3.841, -1.729] ### Key Observations * The effect of subject type (GPT-4 vs. default) varies significantly depending on the treatment conditions. * In some conditions, GPT-4 subjects have significantly higher scores (positive coefficients), while in others, they have significantly lower scores (negative coefficients). * The p-values indicate the statistical significance of the coefficients. Lower p-values suggest a stronger effect. * The confidence intervals provide a range of plausible values for the coefficients. ### Interpretation The Logit Regression Results suggest that the impact of using GPT-4 on respondent scores is highly dependent on the specific treatment conditions applied. The interaction terms in the first result indicate that the effect of GPT-4 varies across different quiz class treatments. The subsequent results, which focus on specific combinations of treatment conditions, further highlight this dependency. For example, in the first result, the positive coefficient for C(subject type, Treatment[reference=1])[T.GPT-4] suggests that, on average, GPT-4 subjects tend to have higher scores. However, the interaction terms reveal that this effect is not consistent across all quiz class treatments. The negative coefficients for C(subject type, Treatment[reference=1])[T.GPT-4] in the later results (e.g., when considering 'randoms, random permuted pairs' or 'random finals') suggest that under these specific conditions, GPT-4 subjects tend to have lower scores compared to the reference group. These findings underscore the importance of considering the context and specific conditions when evaluating the impact of GPT-4 on respondent scores. The results suggest that GPT-4 may not always lead to improved performance and that its effectiveness may depend on the specific task or treatment being applied. </details> Figure 7: Regression outputs, GPT-4 compared to human subjects in the Semantic Structure experiment. <details> <summary>x2.png Details</summary>  ### Visual Description ## Logit Regression Results: Multiple Models ### Overview The image presents a series of Logit Regression Results, likely from statistical modeling software. Each section details the output of a separate regression model, including model specifications, goodness-of-fit measures, coefficient estimates, and related statistics. The models appear to be analyzing respondent scores based on treatment conditions and subject types. ### Components/Axes Each model output includes the following components: * **Model Specifications:** * `Dep. Variable`: Dependent variable (respondent scores). * `Model`: Logit Df Residuals. * `Method`: MLE Df Model. * `Date`: Date of analysis (Wed, 17 Apr 2024). * `Time`: Time of analysis (e.g., 18:27:37). * `Pseudo R-squ`: Pseudo R-squared value (measure of model fit). * `Log-Likelihood`: Log-likelihood value. * `True LL-Null`: True Log-Likelihood Null. * `Covariance Type`: Covariance type (nonrobust). * `LLR p-value`: LLR p-value. * `No. Observations`: Number of observations. * **Coefficient Estimates:** * `Intercept`: Intercept term. * `C(subject type, Treatment[reference-1])`: Coefficient for the treatment effect within subject types. The reference category varies across models. * `coef`: Estimated coefficient value. * `std err`: Standard error of the coefficient. * `z`: z-statistic. * `P>|z|`: p-value associated with the z-statistic. * `[0.025 0.975]`: 95% confidence interval for the coefficient. * **Model Evaluation:** * `Effect of subject with only the conditions`: Specifies the conditions under which the subject effect is evaluated. * `Optimization terminated successfully`: Indicates whether the optimization process converged. * `Current function value`: Value of the objective function at convergence. * `Iterations`: Number of iterations required for convergence. * `Function evaluations`: Number of function evaluations. * `Gradient evaluations`: Number of gradient evaluations. ### Detailed Analysis or Content Details Here's a breakdown of the information extracted from each model output: **Model 1:** * `No. Observations`: 1012 * `Pseudo R-squ`: 0.1935 * `Log-Likelihood`: -509.26 * `LLR p-value`: 4.755e-43 * `Intercept`: * `coef`: 1.3499 * `std err`: 0.300 * `z`: 4.501 * `P>|z|`: 0.000 * `[0.025 0.975]`: 0.762, 1.938 * `C(subject type, Treatment[reference-1])[T.Claude-3-Opus]`: * `coef`: 0.7307 * `std err`: 0.480 * `z`: 1.521 * `P>|z|`: 0.128 * `[0.025 0.975]`: -0.211, 1.672 * `C(quiz class, Treatment[reference-4])[T.distracted]`: * `coef`: 0.7298 * `std err`: 0.480 * `z`: 1.520 * `P>|z|`: 0.129 * `[0.025 0.975]`: -0.211, 1.671 * `C(quiz class, Treatment[reference-4])[T.only_rhs]`: * `coef`: 0.5285 * `std err`: 0.467 * `z`: 1.132 * `P>|z|`: 0.257 * `[0.025 0.975]`: -0.386, 1.443 * `C(quiz class, Treatment[reference-4])[T.permuted_pairs]`: * `coef`: 1.2738 * `std err`: 0.461 * `z`: 2.764 * `P>|z|`: 0.006 * `[0.025 0.975]`: 0.370, 2.177 * `C(quiz class, Treatment[reference-4])[T.random_finals]`: * `coef`: 0.540 * `std err`: 0.475 * `z`: 1.136 * `P>|z|`: 0.256 * `[0.025 0.975]`: -0.390, 1.471 * `C(quiz class, Treatment[reference-4])[T.random_permuted_pairs]`: * `coef`: -0.0123 * `std err`: 0.586 * `z`: -0.021 * `P>|z|`: 0.983 * `[0.025 0.975]`: -1.160, 1.136 * `C(subject type, Treatment[reference-1])[T.Claude-3-Opus]:C(quiz class, Treatment[reference-4])[T.distracted]`: * `coef`: -3.6309 * `std err`: 1.150 * `z`: -3.157 * `P>|z|`: 0.002 * `[0.025 0.975]`: -5.885, -1.377 * `C(subject type, Treatment[reference-1])[T.Claude-3-Opus]:C(quiz class, Treatment[reference-4])[T.only_rhs]`: * `coef`: -3.9300 * `std err`: 1.144 * `z`: -3.435 * `P>|z|`: 0.001 * `[0.025 0.975]`: -6.173, -1.687 * `C(subject type, Treatment[reference-1])[T.Claude-3-Opus]:C(quiz class, Treatment[reference-4])[T.permuted_pairs]`: * `coef`: -4.7848 * `std err`: 1.130 * `z`: -4.232 * `P>|z|`: 0.000 * `[0.025 0.975]`: -7.000, -2.579 * `C(subject type, Treatment[reference-1])[T.Claude-3-Opus]:C(quiz class, Treatment[reference-4])[T.random_finals]`: * `coef`: -4.0614 * `std err`: 1.113 * `z`: -3.648 * `P>|z|`: 0.000 * `[0.025 0.975]`: -6.243, -1.880 * `C(subject type, Treatment[reference-1])[T.Claude-3-Opus]:C(quiz class, Treatment[reference-4])[T.random_permuted_pairs]`: * `coef`: -5.2922 * `std err`: 1.201 * `z`: -4.393 * `P>|z|`: 0.000 * `[0.025 0.975]`: -7.658, -2.926 **Model 2:** * `No. Observations`: 300 * `Pseudo R-squ`: 0.009655 * `Log-Likelihood`: -112.90 * `LLR p-value`: 0.1379 * `Intercept`: * `coef`: 0.5796 * `std err`: 0.232 * `z`: 2.494 * `P>|z|`: 0.013 * `[0.025 0.975]`: 0.125, 1.034 * `C(subject type, Treatment[reference-1])[T.Claude-3-Opus]`: * `coef`: 0.2115 * `std err`: 0.341 * `z`: 0.619 * `P>|z|`: 0.536 * `[0.025 0.975]`: -0.458, 0.881 * `Effect of subject with only the conditions`: distracted, permuted pairs **Model 3:** * `No. Observations`: 300 * `Pseudo R-squ`: 0.03850 * `Log-Likelihood`: -184.91 * `LLR p-value`: 0.0001190 * `Intercept`: * `coef`: 1.0965 * `std err`: 0.257 * `z`: 4.263 * `P>|z|`: 0.000 * `[0.025 0.975]`: 0.593, 1.600 * `C(subject type, Treatment[reference-1])[T.Claude-3-Opus]`: * `coef`: -0.9651 * `std err`: 0.257 * `z`: -3.759 * `P>|z|`: 0.000 * `[0.025 0.975]`: -1.468, -0.462 * `Effect of subject with only the conditions`: distracted **Model 4:** * `No. Observations`: 152 * `Pseudo R-squ`: 0.005370 * `Log-Likelihood`: -99.480 * `LLR p-value`: 0.3013 * `Intercept`: * `coef`: 0.7564 * `std err`: 0.253 * `z`: 2.989 * `P>|z|`: 0.003 * `[0.025 0.975]`: 0.261, 1.252 * `C(subject type, Treatment[reference-1])[T.Claude-3-Opus]`: * `coef`: -0.2619 * `std err`: 0.341 * `z`: -0.768 * `P>|z|`: 0.443 * `[0.025 0.975]`: -1.018, 0.316 * `Effect of subject with only the conditions`: permuted pairs **Model 5:** * `No. Observations`: 148 * `Pseudo R-squ`: 0.1143 * `Log-Likelihood`: -92.503 * `LLR p-value`: 4.248e-06 * `Intercept`: * `coef`: 1.8803 * `std err`: 0.358 * `z`: 5.254 * `P>|z|`: 0.000 * `[0.025 0.975]`: 1.179, 2.582 * `C(subject type, Treatment[reference-1])[T.Claude-3-Opus]`: * `coef`: -1.7802 * `std err`: 0.341 * `z`: -5.221 * `P>|z|`: 0.000 * `[0.025 0.975]`: -2.608, -0.953 * `Effect of subject with only the conditions`: only the **Model 6:** * `No. Observations`: 152 * `Pseudo R-squ`: 0.02999 * `Log-Likelihood`: -71.103 * `LLR p-value`: 0.03890 * `Intercept`: * `coef`: 0.9135 * `std err`: 0.458 * `z`: 1.994 * `P>|z|`: 0.046 * `[0.025 0.975]`: 0.016, 1.810 * `C(subject type, Treatment[reference-1])[T.Claude-3-Opus]`: * `coef`: -0.9130 * `std err`: 0.458 * `z`: -1.994 * `P>|z|`: 0.046 * `[0.025 0.975]`: -1.810, -0.016 * `Effect of subject with only the conditions`: random permuted pairs, randoms, random finals **Model 7:** * `No. Observations`: 260 * `Pseudo R-squ`: 0.09155 * `Log-Likelihood`: -160.00 * `LLR p-value`: 1.356e-08 * `Intercept`: * `coef`: 0.5130 * `std err`: 0.207 * `z`: 2.472 * `P>|z|`: 0.014 * `[0.025 0.975]`: 0.106, 0.938 * `C(subject type, Treatment[reference-1])[T.Claude-3-Opus]`: * `coef`: -1.5016 * `std err`: 0.272 * `z`: -5.511 * `P>|z|`: 0.000 * `[0.025 0.975]`: -2.036, -0.968 * `Effect of subject with only the conditions`: randoms, random permuted pairs **Model 8:** * `No. Observations`: 128 * `Pseudo R-squ`: 0.1601 * `Log-Likelihood`: -74.04 * `LLR p-value`: 1.078e-07 * `Intercept`: * `coef`: 1.0986 * `std err`: 0.333 * `z`: 3.296 * `P>|z|`: 0.001 * `[0.025 0.975]`: 0.445, 1.752 * `C(subject type, Treatment[reference-1])[T.Claude-3-Opus]`: * `coef`: -2.0619 * `std err`: 0.714 * `z`: -2.885 * `P>|z|`: 0.004 * `[0.025 0.975]`: -1.251 * `Effect of subject with only the conditions`: random, finals **Model 9:** * `No. Observations`: 132 * `Pseudo R-squ`: 0.04604 * `Log-Likelihood`: -83.269 * `LLR p-value`: 0.004636 * `Intercept`: * `coef`: 0.7837 * `std err`: 0.374 * `z`: 2.099 * `P>|z|`: 0.036 * `[0.025 0.975]`: 0.049, 1.518 * `C(subject type, Treatment[reference-1])[T.Claude-3-Opus]`: * `coef`: -1.0464 * `std err`: 0.374 * `z`: -2.799 * `P>|z|`: 0.005 * `[0.025 0.975]`: -1.779, -0.314 ### Key Observations * The Pseudo R-squared values vary across models, indicating different levels of model fit. * The coefficients for `C(subject type, Treatment[reference-1])[T.Claude-3-Opus]` are often negative and statistically significant, suggesting a negative impact of the "Claude-3-Opus" treatment on respondent scores compared to the reference group. * The p-values indicate the statistical significance of each coefficient. Lower p-values (typically < 0.05) suggest a statistically significant effect. * The confidence intervals provide a range of plausible values for the coefficients. ### Interpretation The Logit Regression Results suggest that the "Claude-3-Opus" treatment, in combination with different quiz conditions, has a varying impact on respondent scores. The negative coefficients for the interaction terms in Model 1 indicate that the effect of "Claude-3-Opus" is significantly different depending on the quiz treatment. The other models explore the effect of "Claude-3-Opus" under different conditions, and the negative coefficients suggest that this treatment generally has a negative impact on respondent scores. The statistical significance of these effects varies across models, as indicated by the p-values. Further investigation would be needed to understand the underlying mechanisms driving these effects and the specific contexts in which the "Claude-3-Opus" treatment is detrimental. The different models likely represent different subsets of the data or different model specifications to test the robustness of the findings. </details> Figure 8: Regression outputs, Claude 3 compared to human subjects in the Semantic Structure experiment. A.2 Regression results, Semantic Content experiment Regressions are performed in the same manner as for the Semantic Structure experiment, described in Section A.1. Here, the reference condition is Categorial and the degrees of freedom used for the likelihood ratio test is 4. As observed in the Semantic Structure experiment, the performance of GPT-4 in the Semantic Content experiment is human-comparable in some conditions but notably lower in others. When comparing a logistic model that uses subject type and experiment condition separately to one that includes their interactions, a likelihood ratio test shows that the model with interactions fits the data significantly better ( $chi^{2}(4)=39.6565,p<0.001$ ). For the comparison between Claude 3 and human subjects, when comparing a logistic model that uses subject type and experiment condition separately to one that includes their interactions, a likelihood ratio test shows that the model with interactions fits the data significantly better ( $chi^{2}(4)=11.6002,p=0.021$ ). Regression outputs are shown in Figure 9 and 10. <details> <summary>x3.png Details</summary>  ### Visual Description ## Logit Regression Results: Multiple Analyses ### Overview The image presents a series of Logit Regression Results, each analyzing the effect of different conditions on respondent scores. Each analysis includes details such as the date, method, convergence status, covariance type, log-likelihood, and pseudo R-squared. The core of each result is a table showing the coefficient, standard error, z-statistic, and p-values for the intercept and treatment variables. The analyses also specify the effect of subject or condition with certain attributes. ### Components/Axes Each analysis block contains the following components: * **Header**: * `Dep. Variable`: Dependent variable (respondent scores). * `Model`: Logit Df Residuals. * `Method`: MLE Df Model. * `Date`: Date of analysis (Tue, 04 Jun 2024). * `Time`: Time of analysis (varies). * `Pseudo R-squ`: Pseudo R-squared value. * `Converged`: Indicates whether the model converged. * `Covariance Type`: Type of covariance used (nonrobust). * `LLR p-value`: Log-Likelihood Ratio p-value. * `No. Observations`: Number of observations. * `Log-Likelihood`: Log-Likelihood value. * **Coefficient Table**: * `coef`: Coefficient value. * `std err`: Standard error. * `z`: z-statistic. * `P>|z|`: p-value. * `[0.025 0.975]`: Confidence interval (2.5% and 97.5% percentiles). * **Effect Description**: * Describes the effect being analyzed (e.g., "Effect of subject with only the conditions [category]"). * `Current function value`: Value of the current function. * `Function evaluations`: Number of function evaluations. * `Gradient evaluations`: Number of gradient evaluations. * `Optimization terminated successfully`: Indicates if optimization was successful. * `Iterations`: Number of iterations. ### Detailed Analysis or ### Content Details Here's a breakdown of the extracted data from each analysis block: 1. **Analysis 1**: * `No. Observations`: 102 * `Log-Likelihood`: -61.618 * `Pseudo R-squ`: 2.006e-07 * `Intercept`: `coef`: -0.2153, `std err`: 0.418, `z`: -0.515, `P>|z|`: 0.607, `[0.025 0.975]`: -1.035, 0.605 * `C(subject_type, Treatment)[T.preference=1][T.OPT=4]`: `coef`: 0.2153, `std err`: 0.289, `z`: 0.742, `P>|z|`: 0.458, `[0.025 0.975]`: -0.352, 0.783 * `C(clazz_class, Treatment)[T.preference=1][T.mult_attribute]`: `coef`: -0.4227, `std err`: 0.437, `z`: -0.967, `P>|z|`: 0.334, `[0.025 0.975]`: -1.280, 0.435 * `C(clazz_class, Treatment)[T.preference=1][T.numeric]`: `coef`: 0.0427, `std err`: 0.437, `z`: 0.098, `P>|z|`: 0.922, `[0.025 0.975]`: -0.814, 0.899 * `C(clazz_class, Treatment)[T.preference=1][T.relational]`: `coef`: 2.429, `std err`: 0.519, `z`: 4.683, `P>|z|`: 0.000, `[0.025 0.975]`: 1.412, 3.446 * `C(clazz_class, Treatment)[T.preference=1][T.OPT=4]`: `coef`: 0.2019, `std err`: 0.505, `z`: 0.400, `P>|z|`: 0.689, `[0.025 0.975]`: -0.788, 1.192 * `C(clazz_class, Treatment)[T.mult_attribute][T.numeric]`: `coef`: 1.919, `std err`: 0.742, `z`: 2.586, `P>|z|`: 0.010, `[0.025 0.975]`: 0.465, 3.373 * `C(clazz_class, Treatment)[T.mult_attribute][T.relational]`: `coef`: 0.2154, `std err`: 0.379, `z`: 0.568, `P>|z|`: 0.570, `[0.025 0.975]`: -0.528, 0.958 * `C(clazz_class, Treatment)[T.mult_attribute][T.OPT=4]`: `coef`: 0.379, `std err`: 0.568, `z`: 0.570, `P>|z|`: 0.528, `[0.025 0.975]`: -0.528, 0.968 * `C(clazz_class, Treatment)[T.numeric][T.relational]`: `coef`: -0.236, `std err`: 0.568, `z`: -0.570, `P>|z|`: 0.528, `[0.025 0.975]`: -0.528, 0.968 * `C(clazz_class, Treatment)[T.numeric][T.OPT=4]`: `coef`: -0.528, `std err`: 0.568, `z`: -0.570, `P>|z|`: 0.528, `[0.025 0.975]`: -0.528, 0.968 * `C(clazz_class, Treatment)[T.relational][T.OPT=4]`: `coef`: -0.811, `std err`: 0.568, `z`: -0.570, `P>|z|`: 0.528, `[0.025 0.975]`: -0.528, 0.968 * Degrees of freedom: 4 * E value for the significance of model improvement when including interaction terms is 5.007466971816891e-08 2. **Analysis 2**: * `No. Observations`: 105 * `Log-Likelihood`: -62.343 * `Pseudo R-squ`: 0.006444 * `Intercept`: `coef`: -1.086, `std err`: 0.396, `z`: -2.743, `P>|z|`: 0.006, `[0.025 0.975]`: -1.862, -0.310 * `C(subject_type, Treatment)[T.preference=1][T.OPT=4]`: `coef`: 0.3749, `std err`: 0.396, `z`: 0.946, `P>|z|`: 0.344, `[0.025 0.975]`: -0.401, 1.151 * Effect of subject with only the conditions [mult_attribute] 3. **Analysis 3**: * `No. Observations`: 96 * `Log-Likelihood`: -64.841 * `Pseudo R-squ`: 0.004863 * `Intercept`: `coef`: -0.692, `std err`: 0.350, `z`: -1.978, `P>|z|`: 0.048, `[0.025 0.975]`: -1.378, -0.006 * `C(subject_type, Treatment)[T.preference=1][T.OPT=4]`: `coef`: 0.3937, `std err`: 0.440, `z`: 0.894, `P>|z|`: 0.371, `[0.025 0.975]`: -0.468, 1.256 * Effect of subject with only the conditions [mult_attribute] 4. **Analysis 4**: * `No. Observations`: 96 * `Log-Likelihood`: -64.841 * `Pseudo R-squ`: 0.01290 * `Intercept`: `coef`: -0.692, `std err`: 0.350, `z`: -1.978, `P>|z|`: 0.048, `[0.025 0.975]`: -1.378, -0.006 * `C(subject_type, Treatment)[T.preference=1][T.OPT=4]`: `coef`: 0.3937, `std err`: 0.440, `z`: 0.894, `P>|z|`: 0.371, `[0.025 0.975]`: -0.468, 1.256 * Effect of subject with only the conditions [mult_attribute] 5. **Analysis 5**: * `No. Observations`: 104 * `Log-Likelihood`: -65.723 * `Pseudo R-squ`: 0.0005009 * `Intercept`: `coef`: -0.2152, `std err`: 0.369, `z`: -0.583, `P>|z|`: 0.559, `[0.025 0.975]`: -0.938, 0.508 * `C(subject_type, Treatment)[T.preference=1][T.OPT=4]`: `coef`: 0.3800, `std err`: 0.425, `z`: 0.894, `P>|z|`: 0.371, `[0.025 0.975]`: -0.453, 1.213 * Effect of subject with only the conditions [numeric] 6. **Analysis 6**: * `No. Observations`: 104 * `Log-Likelihood`: -65.723 * `Pseudo R-squ`: 0.0005009 * `Intercept`: `coef`: -0.2152, `std err`: 0.369, `z`: -0.583, `P>|z|`: 0.559, `[0.025 0.975]`: -0.938, 0.508 * `C(subject_type, Treatment)[T.preference=1][T.OPT=4]`: `coef`: 0.3800, `std err`: 0.425, `z`: 0.894, `P>|z|`: 0.371, `[0.025 0.975]`: -0.453, 1.213 * Effect of subject with only the conditions [numeric] 7. **Analysis 7**: * `No. Observations`: 96 * `Log-Likelihood`: -64.841 * `Pseudo R-squ`: 0.02530 * `Intercept`: `coef`: -1.255, `std err`: 0.401, `z`: -3.130, `P>|z|`: 0.002, `[0.025 0.975]`: -2.041, -0.469 * `C(subject_type, Treatment)[T.preference=1][T.OPT=4]`: `coef`: 0.4781, `std err`: 0.446, `z`: 1.071, `P>|z|`: 0.284, `[0.025 0.975]`: -0.396, 1.353 * Effect of subject with only the conditions [numeric, mult_attribute] 8. **Analysis 8**: * `No. Observations`: 96 * `Log-Likelihood`: -64.841 * `Pseudo R-squ`: 0.02530 * `Intercept`: `coef`: -1.255, `std err`: 0.401, `z`: -3.130, `P>|z|`: 0.002, `[0.025 0.975]`: -2.041, -0.469 * `C(subject_type, Treatment)[T.preference=1][T.OPT=4]`: `coef`: 0.4781, `std err`: 0.446, `z`: 1.071, `P>|z|`: 0.284, `[0.025 0.975]`: -0.396, 1.353 * Effect of subject with only the conditions [numeric, mult_attribute] 9. **Analysis 9**: * `No. Observations`: 102 * `Log-Likelihood`: -71.778 * `Pseudo R-squ`: 0.0005009 * `Intercept`: `coef`: -1.4781, `std err`: 0.396, `z`: -3.732, `P>|z|`: 0.000, `[0.025 0.975]`: -2.254, -0.702 * `C(subject_type, Treatment)[T.preference=1][T.OPT=4]`: `coef`: 0.1396, `std err`: 0.419, `z`: 0.333, `P>|z|`: 0.739, `[0.025 0.975]`: -0.682, 0.961 * Effect of subject with only the conditions [numeric, multi_attribute] 10. **Analysis 10**: * `No. Observations`: 102 * `Log-Likelihood`: -71.778 * `Pseudo R-squ`: 0.0005009 * `Intercept`: `coef`: -1.4781, `std err`: 0.396, `z`: -3.732, `P>|z|`: 0.000, `[0.025 0.975]`: -2.254, -0.702 * `C(subject_type, Treatment)[T.preference=1][T.OPT=4]`: `coef`: 0.1396, `std err`: 0.419, `z`: 0.333, `P>|z|`: 0.739, `[0.025 0.975]`: -0.682, 0.961 * Effect of subject with only the conditions [numeric, multi_attribute] 11. **Analysis 11**: * `No. Observations`: 90 * `Log-Likelihood`: -69.768 * `Pseudo R-squ`: 0.007916 * `Intercept`: `coef`: 2.116, `std err`: 1.291, `z`: 1.639, `P>|z|`: 0.101, `[0.025 0.975]`: -0.415, 4.647 * `C(subject_type, Treatment)[T.preference=1][T.OPT=4]`: `coef`: -1.368, `std err`: 0.448, `z`: -3.055, `P>|z|`: 0.002, `[0.025 0.975]`: -2.241, -0.493 * Effect of condition with only the conditions [categorical, mult_attribute] for human subjects 12. **Analysis 12**: * `No. Observations`: 90 * `Log-Likelihood`: -69.768 * `Pseudo R-squ`: 0.007916 * `Intercept`: `coef`: 2.116, `std err`: 1.291, `z`: 1.639, `P>|z|`: 0.101, `[0.025 0.975]`: -0.415, 4.647 * `C(subject_type, Treatment)[T.preference=1][T.OPT=4]`: `coef`: -1.368, `std err`: 0.448, `z`: -3.055, `P>|z|`: 0.002, `[0.025 0.975]`: -2.241, -0.493 * Effect of condition with only the conditions [categorical, mult_attribute] for human subjects 13. **Analysis 13**: * `No. Observations`: 116 * `Log-Likelihood`: -78.306 * `Pseudo R-squ`: 0.001537 * `Intercept`: `coef`: 0.255, `std err`: 0.369, `z`: 0.691, `P>|z|`: 0.489, `[0.025 0.975]`: -0.469, 0.979 * `C(clazz_class, Treatment)[T.preference=1][T.OPT=4]`: `coef`: -0.402, `std err`: 0.397, `z`: -1.011, `P>|z|`: 0.312, `[0.025 0.975]`: -1.180, 0.376 * Effect of condition with only the conditions [categorical, multi_attribute] for model 14. **Analysis 14**: * `No. Observations`: 116 * `Log-Likelihood`: -78.306 * `Pseudo R-squ`: 0.001537 * `Intercept`: `coef`: 0.255, `std err`: 0.369, `z`: 0.691, `P>|z|`: 0.489, `[0.025 0.975]`: -0.469, 0.979 * `C(clazz_class, Treatment)[T.preference=1][T.OPT=4]`: `coef`: -0.402, `std err`: 0.397, `z`: -1.011, `P>|z|`: 0.312, `[0.025 0.975]`: -1.180, 0.376 * Effect of condition with only the conditions [categorical, multi_attribute] for model 15. **Analysis 15**: * `No. Observations`: 80 * `Log-Likelihood`: -60.329 * `Pseudo R-squ`: 0.000261 * `Intercept`: `coef`: 0.6160, `std err`: 0.332, `z`: 1.856, `P>|z|`: 0.063, `[0.025 0.975]`: -0.035, 1.269 * `C(clazz_class, Treatment)[T.preference=1][T.OPT=4]`: `coef`: 0.2281, `std err`: 0.479, `z`: 0.477, `P>|z|`: 0.634, `[0.025 0.975]`: -0.710, 1.166 * Effect of condition with only the conditions [numeric, numeric, multi_attribute] for human subjects 16. **Analysis 16**: * `No. Observations`: 80 * `Log-Likelihood`: -60.329 * `Pseudo R-squ`: 0.000261 * `Intercept`: `coef`: 0.6160, `std err`: 0.332, `z`: 1.856, `P>|z|`: 0.063, `[0.025 0.975]`: -0.035, 1.269 * `C(clazz_class, Treatment)[T.preference=1][T.OPT=4]`: `coef`: 0.2281, `std err`: 0.479, `z`: 0.477, `P>|z|`: 0.634, `[0.025 0.975]`: -0.710, 1.166 * Effect of condition with only the conditions [numeric, numeric, multi_attribute] for human subjects 17. **Analysis 17**: * `No. Observations`: 120 * `Log-Likelihood`: -80.577 * `Pseudo R-squ`: 0.006427 * `Intercept`: `coef`: 0.3796, `std err`: 0.255, `z`: 1.488, `P>|z|`: 0.137, `[0.025 0.975]`: -0.120, 0.878 * `C(clazz_class, Treatment)[T.preference=1][T.OPT=4]`: `coef`: -0.2705, `std err`: 0.305, `z`: -0.886, `P>|z|`: 0.376, `[0.025 0.975]`: -0.868, 0.328 * Effect of condition with only the conditions [numeric, numeric, multi_attribute] for model 18. **Analysis 18**: * `No. Observations`: 120 * `Log-Likelihood`: -80.577 * `Pseudo R-squ`: 0.006427 * `Intercept`: `coef`: 0.3796, `std err`: 0.255, `z`: 1.488, `P>|z|`: 0.137, `[0.025 0.975]`: -0.120, 0.878 * `C(clazz_class, Treatment)[T.preference=1][T.OPT=4]`: `coef`: -0.2705, `std err`: 0.305, `z`: -0.886, `P>|z|`: 0.376, `[0.025 0.975]`: -0.868, 0.328 * Effect of condition with only the conditions [numeric, numeric, multi_attribute] for model 19. **Analysis 19**: * `No. Observations`: 80 * `Log-Likelihood`: -60.656 * `Pseudo R-squ`: 0.00007012 * `Intercept`: `coef`: -1.0386, `std err`: 0.365, `z`: -2.848, `P>|z|`: 0.004, `[0.025 0.975]`: -1.754, -0.323 * `C(clazz_class, Treatment)[T.preference=1][T.OPT=4]`: `coef`: 0.2920, `std err`: 0.479, `z`: 0.610, `P>|z|`: 0.542, `[0.025 0.975]`: -0.647, 1.231 * Effect of condition with only the conditions [relational] 20. **Analysis 20**: * `No. Observations`: 80 * `Log-Likelihood`: -60.656 * `Pseudo R-squ`: 0.00007012 * `Intercept`: `coef`: -1.0386, `std err`: 0.365, `z`: -2.848, `P>|z|`: 0.004, `[0.025 0.975]`: -1.754, -0.323 * `C(clazz_class, Treatment)[T.preference=1][T.OPT=4]`: `coef`: 0.2920, `std err`: 0.479, `z`: 0.610, `P>|z|`: 0.542, `[0.025 0.975]`: -0.647, 1.231 * Effect of condition with only the conditions [relational] ### Key Observations * The analyses explore the impact of different treatment conditions on respondent scores. * The p-values indicate the statistical significance of the coefficients. Lower p-values (typically < 0.05) suggest a statistically significant effect. * The confidence intervals provide a range within which the true coefficient value is likely to fall. * Pseudo R-squared values are generally low, indicating that the models explain a limited amount of variance in the dependent variable. * The number of observations varies across the analyses. * The "Effect of subject/condition with only the conditions" descriptions specify the particular attribute combinations being examined. ### Interpretation The Logit Regression Results provide insights into how different treatment conditions and subject attributes influence respondent scores. The low pseudo R-squared values across most analyses suggest that the models have limited explanatory power, and other factors not included in the model may be influencing respondent scores. The p-values for the treatment coefficients vary, indicating that some treatment conditions have a statistically significant effect on respondent scores, while others do not. The confidence intervals provide a range of plausible values for the coefficients, which can be used to assess the magnitude and direction of the effects. The "Effect of subject/condition with only the conditions" descriptions provide context for interpreting the results. For example, the analysis of "Effect of subject with only the conditions [category]" examines the impact of the category attribute on respondent scores. Overall, these results provide a detailed examination of the relationship between treatment conditions, subject attributes, and respondent scores, but the limited explanatory power of the models suggests that further investigation is needed to fully understand the factors influencing respondent scores. </details> Figure 9: Regression outputs, GPT-4 compared to human subjects in the Semantic Content experiment. <details> <summary>x4.png Details</summary>  ### Visual Description ## Logit Regression Results ### Overview The image contains a series of Logit Regression Results, each presenting statistical analysis for different models and conditions. Each result includes details such as the dependent variable, number of observations, model type, date, log-likelihood, convergence status, covariance type, and coefficients with standard errors, z-values, and p-values. The results also describe the effect of subjects or conditions with specific attributes. ### Components/Axes Each Logit Regression Result section contains the following components: * **Dep. Variable:** Dependent variable name (e.g., "respondent scores") * **Model:** Model type (e.g., "MLE DI Model") * **Date:** Date of analysis (e.g., "Tue, 04 Jun 2024") * **Log Likelihood:** Log-likelihood value * **Converged:** Indicates whether the model converged * **Covariance Type:** Type of covariance used (e.g., "nonrobust") * **Pseudo R-squared:** Pseudo R-squared value * **LLR p-value:** Likelihood Ratio Test p-value * **No. Observations:** Number of observations * **Intercept:** Intercept coefficient, standard error, z-value, and p-value, and confidence interval * **Coefficient(s):** Coefficient name, standard error, z-value, and p-value, and confidence interval * **Effect of subject/condition:** Description of the effect being analyzed * **Current function value:** Current function value * **Function evaluations:** Number of function evaluations * **Gradient evaluations:** Number of gradient evaluations ### Detailed Analysis or ### Content Details Here's a breakdown of the extracted data from each Logit Regression Result section: **Result 1:** * Dep. Variable: respondent scores * No. Observations: 96 * Model: MLE DI Model * Date: Tue, 04 Jun 2024 * Log Likelihood: -62.83 * Pseudo R-squared: 0.05200 * Converged: True * Covariance Type: nonrobust * LLR p-value: 0.05200 * Intercept: * coef: -0.2121 * stderr: 0.668 * z: -0.318 * P>|z|: 0.751 * [0.025 0.975]: -1.522, 1.098 * C[subject_type, Treatment/reference=d[TClaude::Cisubject_type, Treatment/reference=d[TClaude::Cisubject_type, Treatment/reference=d[Tnumeric::multi_attribute]]: * coef: 0.2101 * stderr: 0.626 * z: 0.336 * P>|z|: 0.737 * [0.025 0.975]: -1.018, 1.438 * C[subject_type, Treatment/reference=d[TClaude::Cisubject_type, Treatment/reference=d[TClaude::Cisubject_type, Treatment/reference=d[Tcategorical::multi_attribute]]: * coef: 0.6696 * stderr: 0.532 * z: 1.259 * P>|z|: 0.208 * [0.025 0.975]: -0.373, 1.712 * C[subject_type, Treatment/reference=d[TClaude::Cisubject_type, Treatment/reference=d[Tnumeric::multi_attribute, Tcategorical::multi_attribute]]: * coef: 0.420 * stderr: 0.420 * z: 1.001 * P>|z|: 0.317 * [0.025 0.975]: -0.403, 1.243 * C[subject_type, Treatment/reference=d[TClaude::Cisubject_type, Treatment/reference=d[TClaude::Cisubject_type, Treatment/reference=d[Talgorithmic::multi_attribute]]: * coef: 1.477 * stderr: 0.740 * z: 2.00 * P>|z|: 0.046 * [0.025 0.975]: 0.027, 2.927 * C[subject_type, Treatment/reference=d[TClaude::Cisubject_type, Treatment/reference=d[Tnumeric::multi_attribute, Talgorithmic::multi_attribute]]: * coef: 0.140 * stderr: 0.577 * z: 0.243 * P>|z|: 0.808 * [0.025 0.975]: -0.991, 1.271 * C[subject_type, Treatment/reference=d[TClaude::Cisubject_type, Treatment/reference=d[Tcategorical::multi_attribute, Talgorithmic::multi_attribute]]: * coef: -0.203 * stderr: 0.526 * z: -0.386 * P>|z|: 0.700 * [0.025 0.975]: -1.234, 0.828 * C[subject_type, Treatment/reference=d[TClaude::Cisubject_type, Treatment/reference=d[Tnumeric::multi_attribute, Tcategorical::multi_attribute, Talgorithmic::multi_attribute]]: * coef: 1.444 * stderr: 0.844 * z: 1.712 * P>|z|: 0.087 * [0.025 0.975]: -0.210, 3.098 * C[subject_type, Treatment/reference=d[Talgorithmic::multi_attribute]]: * coef: 0.2148 * stderr: 0.379 * z: 0.567 * P>|z|: 0.571 * [0.025 0.975]: -0.529, 0.968 * C[subject_type, Treatment/reference=d[Tcategorical::multi_attribute]]: * coef: 0.529 * stderr: 0.567 * z: 0.933 * P>|z|: 0.351 * [0.025 0.975]: -0.582, 1.640 * C[subject_type, Treatment/reference=d[Tnumeric::multi_attribute]]: * coef: -0.612 * stderr: 0.8001 * z: -0.765 * P>|z|: 0.444 * [0.025 0.975]: -2.180, 0.956 * C[subject_type, Treatment/reference=d[Tnumeric::multi_attribute, Tcategorical::multi_attribute]]: * coef: -0.251 * stderr: 0.586 * z: -0.428 * P>|z|: 0.669 * [0.025 0.975]: -1.400, 0.898 * C[subject_type, Treatment/reference=d[Tnumeric::multi_attribute, Talgorithmic::multi_attribute]]: * coef: 1.378 * stderr: 1.168 * z: 1.179 * P>|z|: 0.238 * [0.025 0.975]: -0.912, 3.667 * C[subject_type, Treatment/reference=d[Tcategorical::multi_attribute, Talgorithmic::multi_attribute]]: * coef: 0.947 * stderr: 1.378 * z: 0.687 * P>|z|: 0.492 * [0.025 0.975]: -1.754, 3.648 * Degrees of freedom: 4 * Likelihood ratio is 11.6001 * p value for the significance of model improvement when including interaction terms is 0.020686001787547076 * Effect of subject with only the conditions [categorical] * Current function value: 0.654493 * Optimization terminated successfully * Function evaluations: 12 * Gradient evaluations: 12 **Result 2:** * Dep. Variable: respondent scores * No. Observations: 96 * Model: Logit DI Model * Date: Tue, 04 Jun 2024 * Log Likelihood: -64.104 * Pseudo R-squared: 0.02782 * Converged: True * Covariance Type: nonrobust * LLR p-value: 0.05007 * Intercept: * coef: -1.457 * stderr: 0.733 * z: -1.988 * P>|z|: 0.047 * [0.025 0.975]: -2.905, -0.009 * C[subject_type, Treatment/reference=d[TClaude::Cisubject_type, Treatment/reference=d[TClaude::Cisubject_type, Treatment/reference=d[Tnumeric::multi_attribute]]: * coef: -0.8109 * stderr: 0.430 * z: -1.879 * P>|z|: 0.060 * [0.025 0.975]: -1.657, 0.035 * Effect of subject with only the conditions [multi_attribute] * Current function value: 0.661403 * Optimization terminated successfully * Function evaluations: 9 * Gradient evaluations: 10 **Result 3:** * Dep. Variable: respondent scores * No. Observations: 96 * Model: Logit DI Model * Date: Tue, 04 Jun 2024 * Log Likelihood: -66.521 * Pseudo R-squared: 0.01664 * Converged: True * Covariance Type: nonrobust * LLR p-value: 0.1368 * Intercept: * coef: -0.2889 * stderr: 0.420 * z: -0.688 * P>|z|: 0.491 * [0.025 0.975]: -1.112, 0.534 * C[subject_type, Treatment/reference=d[TClaude::Cisubject_type, Treatment/reference=d[TClaude::Cisubject_type, Treatment/reference=d[Tnumeric::multi_attribute]]: * coef: 0.2152 * stderr: 0.269 * z: 0.800 * P>|z|: 0.423 * [0.025 0.975]: -0.312, 0.742 * Effect of subject with only the conditions [numeric] * Current function value: 0.688033 * Optimization terminated successfully * Function evaluations: 10 * Gradient evaluations: 11 **Result 4:** * Dep. Variable: respondent scores * No. Observations: 104 * Model: Logit DI Model * Date: Tue, 04 Jun 2024 * Log Likelihood: -70.855 * Pseudo R-squared: 0.001360 * Converged: True * Covariance Type: nonrobust * LLR p-value: 0.6607 * Intercept: * coef: -1.491 * stderr: 0.159 * z: -9.375 * P>|z|: 0.000 * [0.025 0.975]: -1.803, -1.179 * C[subject_type, Treatment/reference=d[TClaude::Cisubject_type, Treatment/reference=d[TClaude::Cisubject_type, Treatment/reference=d[Tnumeric::multi_attribute]]: * coef: -0.1748 * stderr: 0.457 * z: -0.382 * P>|z|: 0.702 * [0.025 0.975]: -0.977, 0.619 * Effect of subject with only the conditions [numeric, multi_attribute] * Current function value: 0.688076 * Optimization terminated successfully * Function evaluations: 9 * Gradient evaluations: 10 **Result 5:** * Dep. Variable: respondent scores * No. Observations: 96 * Model: Logit DI Model * Date: Tue, 04 Jun 2024 * Log Likelihood: -66.459 * Pseudo R-squared: 0.005760 * Converged: True * Covariance Type: nonrobust * LLR p-value: 0.6286 * Intercept: * coef: -0.2667 * stderr: 0.300 * z: -0.889 * P>|z|: 0.374 * [0.025 0.975]: -0.854, 0.321 * C[subject_type, Treatment/reference=d[TClaude::Cisubject_type, Treatment/reference=d[TClaude::Cisubject_type, Treatment/reference=d[Tnumeric::multi_attribute]]: * coef: -0.2009 * stderr: 0.415 * z: -0.484 * P>|z|: 0.629 * [0.025 0.975]: -1.015, 0.613 * Effect of subject with only the conditions [relational] * Current function value: 0.683177 * Optimization terminated successfully * Function evaluations: 10 * Gradient evaluations: 11 **Result 6:** * Dep. Variable: respondent scores * No. Observations: 90 * Model: Logit DI Model * Date: Tue, 04 Jun 2024 * Log Likelihood: -67.978 * Pseudo R-squared: 0.001884 * Converged: True * Covariance Type: nonrobust * LLR p-value: 0.6390 * Intercept: * coef: -0.2281 * stderr: 0.261 * z: -0.874 * P>|z|: 0.382 * [0.025 0.975]: -0.742, 0.286 * C[subject_type, Treatment/reference=d[TClaude::Cisubject_type, Treatment/reference=d[TClaude::Cisubject_type, Treatment/reference=d[Tnumeric::multi_attribute]]: * coef: 0.0161 * stderr: 0.452 * z: 0.037 * P>|z|: 0.971 * [0.025 0.975]: -0.874, 0.907 * Effect of condition with only the conditions [categorical, multi_attribute] for human subjects * Current function value: 0.688033 * Optimization terminated successfully * Function evaluations: 11 * Gradient evaluations: 11 **Result 7:** * Dep. Variable: respondent scores * No. Observations: 116 * Model: Logit DI Model * Date: Tue, 04 Jun 2024 * Log Likelihood: -78.306 * Pseudo R-squared: 0.006407 * Converged: True * Covariance Type: nonrobust * LLR p-value: 0.001537 * Intercept: * coef: 0.0115 * stderr: 0.292 * z: 0.039 * P>|z|: 0.969 * [0.025 0.975]: -0.561, 0.584 * C[subject_type, Treatment/reference=d[TClaude::Cisubject_type, Treatment/reference=d[TClaude::Cisubject_type, Treatment/reference=d[Tnumeric::multi_attribute]]: * coef: -1.2887 * stderr: 0.397 * z: -3.245 * P>|z|: 0.001 * [0.025 0.975]: -2.065, -0.512 * Effect of condition with only the conditions [categorical, multi_attribute] for model * Current function value: 0.686076 * Optimization terminated successfully * Function evaluations: 10 * Gradient evaluations: 11 **Result 8:** * Dep. Variable: respondent scores * No. Observations: 78 * Model: Logit DI Model * Date: Tue, 04 Jun 2024 * Log Likelihood: -54.545 * Pseudo R-squared: 0.001876 * Converged: True * Covariance Type: nonrobust * LLR p-value: 0.6510 * Intercept: * coef: 0.2007 * stderr: 0.318 * z: 0.631 * P>|z|: 0.528 * [0.025 0.975]: -0.422, 0.824 * C[subject_type, Treatment/reference=d[TClaude::Cisubject_type, Treatment/reference=d[TClaude::Cisubject_type, Treatment/reference=d[Tnumeric::multi_attribute]]: * coef: 0.2549 * stderr: 0.453 * z: 0.562 * P>|z|: 0.573 * [0.025 0.975]: -0.631, 1.140 * Effect of condition with only the conditions [numeric, numeric, multi_attribute] for human subjects * Current function value: 0.688037 * Optimization terminated successfully * Function evaluations: 11 * Gradient evaluations: 11 **Result 9:** * Dep. Variable: respondent scores * No. Observations: 118 * Model: Logit DI Model * Date: Tue, 04 Jun 2024 * Log Likelihood: -80.577 * Pseudo R-squared: 0.0006427 * Converged: True * Covariance Type: nonrobust * LLR p-value: 0.3029 * Intercept: * coef: 0.3796 * stderr: 0.255 * z: 1.487 * P>|z|: 0.137 * [0.025 0.975]: -0.121, 0.881 * C[subject_type, Treatment/reference=d[TClaude::Cisubject_type, Treatment/reference=d[TClaude::Cisubject_type, Treatment/reference=d[Tnumeric::multi_attribute]]: * coef: -0.2705 * stderr: 0.386 * z: -0.701 * P>|z|: 0.483 * [0.025 0.975]: -1.028, 0.487 * Effect of condition with only the conditions [numeric, numeric, multi_attribute] for model * Current function value: 0.688130 * Optimization terminated successfully * Function evaluations: 11 * Gradient evaluations: 11 **Result 10:** * Dep. Variable: respondent scores * No. Observations: 78 * Model: Logit DI Model * Date: Tue, 04 Jun 2024 * Log Likelihood: -55.452 * Pseudo R-squared: 0.0007226 * Converged: True * Covariance Type: nonrobust * LLR p-value: 0.3707 * Intercept: * coef: 0.2007 * stderr: 0.318 * z: 0.631 * P>|z|: 0.528 * [0.025 0.975]: -0.422, 0.824 * C[subject_type, Treatment/reference=d[TClaude::Cisubject_type, Treatment/reference=d[TClaude::Cisubject_type, Treatment/reference=d[Tnumeric::multi_attribute]]: * coef: -0.1344 * stderr: 0.498 * z: -0.270 * P>|z|: 0.787 * [0.025 0.975]: -1.112, 0.843 * Effect of condition with only the conditions [numeric, multi_attribute] * Current function value: 0.688037 * Optimization terminated successfully * Function evaluations: 11 * Gradient evaluations: 11 ### Key Observations * The models use different combinations of subject types and treatment references. * The LLR p-values vary across the models, indicating different levels of statistical significance. * The coefficients and their significance levels vary, suggesting different impacts of the conditions on the dependent variable. * Most models converged successfully. ### Interpretation The Logit Regression Results provide insights into the relationships between different conditions and respondent scores. The varying LLR p-values suggest that some models are better at explaining the variance in the dependent variable than others. The coefficients indicate the direction and magnitude of the effect of each condition on the respondent scores. The analysis helps in understanding the impact of different factors on the outcome being studied. The models explore different combinations of subject types and treatment references, allowing for a comprehensive understanding of the data. </details> Figure 10: Regression outputs, Claude 3 compared to human subjects in the Semantic Content experiment. A.3 Further details of human performance Figure 11 shows the difference in performance between online subjects recruited through Prolific and in-person University-Name University students in the Semantic Structure experiment conditions. Prolific subjects were paid $1.50 for the task, with Prolific taking an additional $0.50 per subject. This equated to an approximate effective rate of $22 per hour, well above relevant minimum wages. In-person subjects were each paid $10 to reflect the increased time and effort cost. We expect increased performance from the in person subjects for a number of reasons. First, they are more highly remunerated. Second, there may be social pressure to perform well given the presence of a member of the research team. Third, the in-person subjects may not have the decreased attention effects likely experienced by subjects on Prolific who may complete many unrelated and potentially demotivating tasks in a day. Fourth, there is an implicit selection effect on academic performance for students at our university, which is not unrelated to the competencies involved in completing the tasks in the experiment. Indeed, we observe that accuracy increases by approximately 0.1-0.2 for the in-person subjects in all but one condition. The exception to this is the Random Finals condition, in which mean accuracy decreases slightly. However, in this condition it is not clear that a decrease in “accuracy” is objectively worse performance, because in this condition we ask for the drawing corresponding to a final unrelated term, while all previous left-hand terms within the question are related. It is thus not unreasonable to give an answer that differs from what we expect, except insofar as subjects are learning in context from previous questions in the quiz to realize that the final unrelated word should be regarded as irrelevant. Logistic regression analysis confirms that the in-person subjects outperform the online subjects. This is confirmed with the in-person subjects as the reference class and with the independent variable either being jointly the subject type and quiz class, or the subject type alone (respectively coef -1.1738, P 0.015 and coef -0.6462, P 0.000). <details> <summary>extracted/5679376/Images/Prolific_University-Name_Comparison.png Details</summary>  ### Visual Description ## Bar Chart: Accuracy by Condition ### Overview The image is a bar chart comparing the accuracy of two groups, "Prolific" and "University students," across different experimental conditions. The y-axis represents accuracy, ranging from 0.0 to 1.0. The x-axis represents the different conditions: "Defaults," "Distracted," "Permuted Pairs," "Permuted Questions," "Random Permuted Pairs," "Randoms," "Only Rhs," and "Random Finals." Error bars are included on each bar, indicating variability. Small red dots are scattered around each bar, possibly representing individual data points. ### Components/Axes * **Title:** Accuracy by Condition * **Y-axis:** Accuracy, ranging from 0.0 to 1.0 in increments of 0.2. * **X-axis:** * Defaults * Distracted * Permuted Pairs * Permuted Questions * Random Permuted Pairs * Randoms * Only Rhs * Random Finals * **Legend (Top-Right):** * Blue: Prolific * Orange: University students ### Detailed Analysis Here's a breakdown of the accuracy for each condition, including approximate values and trends: * **Defaults:** * Prolific (Blue): Accuracy around 0.66, with error bars extending from approximately 0.6 to 0.72. * University students (Orange): Accuracy around 0.89, with error bars extending from approximately 0.83 to 0.95. * **Distracted:** * Prolific (Blue): Accuracy around 0.52, with error bars extending from approximately 0.46 to 0.58. * University students (Orange): Accuracy around 0.68, with error bars extending from approximately 0.62 to 0.74. * **Permuted Pairs:** * Prolific (Blue): Accuracy around 0.71, with error bars extending from approximately 0.65 to 0.77. * University students (Orange): Accuracy around 0.84, with error bars extending from approximately 0.78 to 0.90. * **Permuted Questions:** * Prolific (Blue): Accuracy around 0.64, with error bars extending from approximately 0.58 to 0.70. * University students (Orange): Accuracy around 0.80, with error bars extending from approximately 0.74 to 0.86. * **Random Permuted Pairs:** * Prolific (Blue): Accuracy around 0.46, with error bars extending from approximately 0.40 to 0.52. * University students (Orange): Accuracy around 0.72, with error bars extending from approximately 0.66 to 0.78. * **Randoms:** * Prolific (Blue): Accuracy around 0.59, with error bars extending from approximately 0.53 to 0.65. * University students (Orange): Accuracy around 0.79, with error bars extending from approximately 0.73 to 0.85. * **Only Rhs:** * Prolific (Blue): Accuracy around 0.75, with error bars extending from approximately 0.69 to 0.81. * University students (Orange): Accuracy around 0.89, with error bars extending from approximately 0.83 to 0.95. * **Random Finals:** * Prolific (Blue): Accuracy around 0.61, with error bars extending from approximately 0.55 to 0.67. * University students (Orange): Accuracy around 0.48, with error bars extending from approximately 0.42 to 0.54. ### Key Observations * University students generally have higher accuracy than Prolific participants across most conditions, except for "Random Finals." * The "Random Permuted Pairs" condition shows the lowest accuracy for both groups. * The "Defaults" and "Only Rhs" conditions show the highest accuracy for University students. ### Interpretation The bar chart suggests that the "University students" group generally performs better in terms of accuracy compared to the "Prolific" group across various experimental conditions. The "Random Permuted Pairs" condition appears to be the most challenging for both groups, indicating that this specific manipulation significantly impacts accuracy. The scattered red dots could represent individual data points, providing a visual representation of the distribution of accuracy within each group and condition. The error bars indicate the variability within each condition, with larger error bars suggesting more variation in accuracy among participants. The "Random Finals" condition is an outlier, where the "Prolific" group shows higher accuracy than the "University students" group. This could be due to various factors, such as differences in test-taking strategies or prior knowledge between the two groups. </details> Figure 11: Accuracy comparison of online subjects recruited through Prolific and in-person University-Name University students in the Semantic Structure experiment conditions. Figure 12 below shows the variation in performance among human subjects completing different quizzes. As can be seen, some conditions aggregate over quizzes in which the mean performance is quite stable (for example, Random Finals). Other conditions aggregate over quizzes in which there is a larger variation in performance (for example, randoms). In the first quiz of the randoms condition, all respondents score 100%. No particular features of this quiz were identified that would explain this occurrence. However, given that the performance of subsequent quiz-takers is independent, that a significant proportion of all quiz-takers score 100%, and that we have 28 quizzes that each sample a number of respondents, it does not seem unlikely for one quiz to have all perfect scores by chance. <details> <summary>extracted/5679376/Images/Human_Accuracy_by_Quiz.png Details</summary>  ### Visual Description ## Bar Chart: Accuracy by Condition ### Overview The image is a bar chart comparing the accuracy of four quizzes under different conditions. The chart displays the mean accuracy for each quiz within each condition, with error bars indicating the variability. ### Components/Axes * **Title:** Accuracy by Condition * **Y-axis:** Accuracy, ranging from 0.0 to 1.0 in increments of 0.2. * **X-axis:** Condition, with the following categories: Defaults, Distracted, Permuted Pairs, Permuted Questions, Random Permuted Pairs, Randoms, Only Rhs, Random Finals. * **Legend:** Located in the top-right corner, the legend identifies the quizzes: * Quiz 1 (Blue) * Quiz 2 (Orange) * Quiz 3 (Green) * Quiz 4 (Red) ### Detailed Analysis The chart presents accuracy data for four quizzes across eight different conditions. Each condition has four bars representing the accuracy of each quiz. Error bars are present on each bar, indicating the standard deviation or confidence interval. **Condition: Defaults** * Quiz 1 (Blue): Accuracy ~0.75 * Quiz 2 (Orange): Accuracy ~0.92 * Quiz 3 (Green): Accuracy ~0.90 * Quiz 4 (Red): Accuracy ~0.93 **Condition: Distracted** * Quiz 1 (Blue): Accuracy ~0.75 * Quiz 2 (Orange): Accuracy ~0.57 * Quiz 3 (Green): Accuracy ~0.69 * Quiz 4 (Red): Accuracy ~0.70 **Condition: Permuted Pairs** * Quiz 1 (Blue): Accuracy ~0.92 * Quiz 2 (Orange): Accuracy ~0.59 * Quiz 3 (Green): Accuracy ~0.93 * Quiz 4 (Red): Accuracy ~0.94 **Condition: Permuted Questions** * Quiz 1 (Blue): Accuracy ~1.0 * Quiz 2 (Orange): Accuracy ~0.65 * Quiz 3 (Green): Accuracy ~0.65 * Quiz 4 (Red): Accuracy ~0.80 **Condition: Random Permuted Pairs** * Quiz 1 (Blue): Accuracy ~0.70 * Quiz 2 (Orange): Accuracy ~0.75 * Quiz 3 (Green): Accuracy ~0.50 * Quiz 4 (Red): Accuracy ~0.70 **Condition: Randoms** * Quiz 1 (Blue): Accuracy ~0.92 * Quiz 2 (Orange): Accuracy ~0.67 * Quiz 3 (Green): Accuracy ~0.0 * Quiz 4 (Red): Accuracy ~0.0 **Condition: Only Rhs** * Quiz 1 (Blue): Accuracy ~0.94 * Quiz 2 (Orange): Accuracy ~0.82 * Quiz 3 (Green): Accuracy ~0.90 * Quiz 4 (Red): Accuracy ~0.90 **Condition: Random Finals** * Quiz 1 (Blue): Accuracy ~0.25 * Quiz 2 (Orange): Accuracy ~0.67 * Quiz 3 (Green): Accuracy ~0.50 * Quiz 4 (Red): Accuracy ~0.25 ### Key Observations * Quiz 1 (Blue) generally shows high accuracy in "Defaults", "Permuted Pairs", "Permuted Questions", "Randoms", and "Only Rhs" conditions, but significantly lower accuracy in "Distracted" and "Random Finals" conditions. * Quiz 2 (Orange) shows relatively consistent accuracy across "Defaults", "Distracted", "Permuted Pairs", "Permuted Questions", and "Random Permuted Pairs" conditions, but lower accuracy in "Randoms" and "Random Finals" conditions. * Quiz 3 (Green) and Quiz 4 (Red) show high accuracy in "Defaults", "Permuted Pairs", and "Only Rhs" conditions. * The "Randoms" condition results in very low accuracy for Quizzes 3 and 4. * The "Random Finals" condition results in low accuracy for Quizzes 1 and 4. ### Interpretation The data suggests that the different conditions have varying impacts on the accuracy of the quizzes. The "Distracted" and "Random Finals" conditions seem to negatively affect the accuracy of Quiz 1, while the "Randoms" condition significantly reduces the accuracy of Quizzes 3 and 4. The "Defaults", "Permuted Pairs", "Permuted Questions", and "Only Rhs" conditions generally result in high accuracy across all quizzes. This indicates that the specific conditions of the quizzes play a crucial role in determining the overall accuracy. The error bars provide insight into the variability of the data, suggesting the reliability of the observed trends. </details> Figure 12: Human accuracy by quiz across conditions. Error bars show standard errors. One highly-specific failure mode is present in the human data and deserves special consideration. In only the “*” grounding, participants quite often introduced separator characters into the response (either just “>” or both separator characters, “=>”). This was observed in 11 instances, thus affecting approximately 10% of the responses in that grounding scheme. This issue was not observed in any other grounding scheme. The reason for this is not entirely clear, but could be related to the short length of the groundings in this scheme (a grounding term is either 1 or 3 characters in this scheme, which is shorter than the other versions). It is possible that the short length of the grounding terms could lead subjects to perceive the separator characters as being part of the grounding term, although it is unclear why this would happen even for subjects who successfully ignore the separator characters in three prior responses (which applies to 7 out of 11 such errors). The error rate of the human subjects by grounding type is shown in Figure 13. The error rate in the “*”, “C K E”, and “Q Z I” grounding schemes was essentially equal, with approximately half of this rate in the “c c” grounding scheme. This is not entirely intuitive, but some possible explanations of this can be offered. First, the “c c” groundings have the fewest number of distinct characters, thus limiting the option space when answering (there are two distinct characters, compared to 3 or 4 for the other groundings). Second, the specific transformations involved (capitalization and adding/removing a letter) are common operations that are encountered more frequently than, say, inserting a special character between existing characters. As can be seen in Table 5, about a fifth of the human participants’ incorrect responses were simply copies of one of the three right-hand terms presented in the question. Very few participants made a mistake that only reordered the correct answer, whereas about half of all incorrect answers were the wrong combination of characters from the right-hand terms of the task. Note that in all of our target domains, any individual right-hand term only uses characters that are found in at least one of the other three. Thus, the third of incorrect responses from human subjects which did not fall into any of the previous categories included characters which had not been presented in one of the three preceding right-hand terms. This can be explained in some cases by the presence of a distractor that confused a participant into including characters from a right-hand term that used other characters, while in others it can be explained by typos or some other confusion. In addition to the task questions, subjects were presented with three follow-up questions that asked them to rate their confidence that their answers were correct, describe what they thought the task involved, and describe their strategy for answering the questions. Subjects employ a mix of strategies in answering the questions. Some subjects explicitly attend to the analogy structure of the left-hand terms. For example, in the distracted condition, one participant reports that “I tried to find the one that resembled the blank one…ie, red/pink, cat/kitten”. By contrast, others focused more on completing the pattern in the right-hand terms. Most subjects robustly ignored the distractor terms in that condition. Some subjects who report a detailed, correct strategy nevertheless fail to attain a high accuracy, thus demonstrating that the task is not trivial even for those who are able to fully grasp what it involves. For example, one subject attains a below-average accuracy of 50% in the distracted condition despite being able to state that the task involves “Looking at other comparable entries to figure out what the answer to the last entry was (dog:puppy::cat:kitten)”, and reporting a strategy in which “I tried to find similar pairs of entries and looked at their meanings.” By contrast, another subject attains 100% accuracy in the distracted condition while responding to what the task involved with “I thought it was fun” and reporting a strategy in which “I just compared answers and tried my best to understand what they were and then tried to guess based on my interpretation of the other answers.” Table 5: The distributions of types of errors made by top-performing participants in the Semantic Structure experiment. | | Copy Context | Scrambled | Wrong Combination | Other | | --- | --- | --- | --- | --- | | Human | 0.192 | 0.020 | 0.556 | 0.232 | | GPT-4 | 0.239 | 0.031 | 0.502 | 0.228 | | Claude 3 Opus | 0.036 | 0.045 | 0.276 | 0.643 | <details> <summary>extracted/5679376/Images/incorrect_answers_by_grounding.png Details</summary>  ### Visual Description ## Bar Chart: Percent of Incorrect Answers by Grounding Type ### Overview The image is a bar chart comparing the percentage of incorrect answers given by humans, Claude 3 Opus, and GPT-4 across different grounding types. The y-axis represents the percentage of incorrect answers, and the x-axis represents the grounding type. ### Components/Axes * **Title:** Percent of Incorrect Answers by Grounding Type * **Y-axis:** Percentage of All Incorrect Answers, ranging from 0.00 to 0.30 in increments of 0.05. * **X-axis:** Grounding Types: \*, CKE, QZI, cc * **Legend:** Located at the top-right of the chart. * Blue: Human * Orange: Claude 3 Opus * Green: GPT-4 ### Detailed Analysis The chart displays the percentage of incorrect answers for each grounding type, broken down by the three agents (Human, Claude 3 Opus, and GPT-4). * **Grounding Type \*:** * Human (Blue): Approximately 0.27 * Claude 3 Opus (Orange): Approximately 0.25 * GPT-4 (Green): Approximately 0.32 * **Grounding Type CKE:** * Human (Blue): Approximately 0.26 * Claude 3 Opus (Orange): Approximately 0.30 * GPT-4 (Green): Approximately 0.24 * **Grounding Type QZI:** * Human (Blue): Approximately 0.31 * Claude 3 Opus (Orange): Approximately 0.28 * GPT-4 (Green): Approximately 0.26 * **Grounding Type cc:** * Human (Blue): Approximately 0.15 * Claude 3 Opus (Orange): Approximately 0.16 * GPT-4 (Green): Approximately 0.17 ### Key Observations * GPT-4 has the highest percentage of incorrect answers for grounding type '\*'. * Claude 3 Opus has the highest percentage of incorrect answers for grounding type 'CKE'. * Human has the highest percentage of incorrect answers for grounding type 'QZI'. * All three agents have the lowest percentage of incorrect answers for grounding type 'cc'. ### Interpretation The bar chart compares the performance of humans, Claude 3 Opus, and GPT-4 in answering questions related to different grounding types. The data suggests that the difficulty of the grounding type significantly impacts the accuracy of all three agents. Grounding type 'cc' appears to be the easiest, while the other grounding types show varying levels of difficulty for each agent. There is no single agent that consistently outperforms the others across all grounding types, indicating that their strengths and weaknesses vary depending on the specific type of grounding required. </details> Figure 13: Percentage of incorrect answers in the Semantic Structure experiment by target domain type. In addition to the comparable mean performance, we find similar patterns in the errors made by humans and GPT-4. In Table 5 and Figure 13, one can see that the distribution of errors is comparable both when broken down by target domain type and when broken down by several error classifications we design. <details> <summary>extracted/5679376/Images/top_performers_best_fit_and_points.png Details</summary>  ### Visual Description ## Chart: Accuracy vs. Question Number by Subject Type and Experiment Condition ### Overview The image presents a series of line plots arranged in a 3x8 grid. Each plot visualizes the accuracy of different subjects (Human, Claude 3 Opus, GPT-4) across four question quartiles (Q1, Q2, Q3, Q4) under various experimental conditions (defaults, distracted, permuted_pairs, permuted_questions, random_permuted_pairs, randoms, only_rhs, random_finals). The plots include error bars indicating variability in the data. ### Components/Axes * **Title:** Accuracy vs. Question Number by Subject Type and Experiment Condition * **Y-axis:** Accuracy, ranging from 0.0 to 1.0 * **X-axis:** Question Quartiles (Q1, Q2, Q3, Q4) * **Subjects (Rows):** Human, Claude 3 Opus, GPT-4 * **Experiment Conditions (Columns):** defaults, distracted, permuted\_pairs, permuted\_questions, random\_permuted\_pairs, randoms, only\_rhs, random\_finals * **Data Points:** Teal dots representing accuracy for each quartile. * **Error Bars:** Teal vertical lines extending above and below each data point, indicating the range of variability. * **Trend Lines:** Blue lines indicating the general trend of accuracy across the quartiles. ### Detailed Analysis **Row 1: Human** * **defaults:** Accuracy starts around 0.95 in Q1 and slightly decreases to approximately 0.9 in Q4. * **distracted:** Accuracy is around 0.7 in Q1 and decreases slightly to about 0.65 in Q4. * **permuted\_pairs:** Accuracy is relatively stable, staying around 0.85-0.9 across all quartiles. * **permuted\_questions:** Accuracy increases from approximately 0.75 in Q1 to 0.9 in Q4. * **random\_permuted\_pairs:** Accuracy increases from approximately 0.65 in Q1 to 0.85 in Q4. * **randoms:** Accuracy increases sharply from approximately 0.5 in Q1 to nearly 1.0 in Q4. * **only\_rhs:** Accuracy is relatively stable, staying around 0.9 across all quartiles. * **random\_finals:** Accuracy increases from approximately 0.3 in Q1 to 0.6 in Q4. **Row 2: Claude 3 Opus** * **defaults:** Accuracy increases from approximately 0.6 in Q1 to nearly 1.0 in Q4. * **distracted:** Accuracy starts around 0.25 in Q1, increases to approximately 0.75 in Q3, and decreases to 0.5 in Q4. * **permuted\_pairs:** Accuracy is relatively stable, staying around 0.6 across all quartiles. * **permuted\_questions:** Accuracy decreases from approximately 0.25 in Q1 to 0.1 in Q4. * **random\_permuted\_pairs:** Accuracy decreases from approximately 0.9 in Q1 to 0.2 in Q4. * **randoms:** Accuracy is relatively stable, staying around 0.8 across all quartiles. * **only\_rhs:** Accuracy decreases from approximately 0.8 in Q1 to 0.6 in Q4. * **random\_finals:** Accuracy increases from approximately 0.1 in Q1 to 0.4 in Q4. **Row 3: GPT-4** * **defaults:** Accuracy increases from approximately 0.6 in Q1 to nearly 1.0 in Q4. * **distracted:** Accuracy starts around 0.25 in Q1 and increases slightly to about 0.5 in Q4. * **permuted\_pairs:** Accuracy decreases from approximately 0.7 in Q1 to 0.5 in Q4. * **permuted\_questions:** Accuracy increases from approximately 0.1 in Q1 to 0.5 in Q4. * **random\_permuted\_pairs:** Accuracy increases from approximately 0.0 in Q1 to 0.9 in Q4. * **randoms:** Accuracy is relatively stable, staying around 0.9 across all quartiles. * **only\_rhs:** Accuracy decreases from approximately 0.9 in Q1 to 0.8 in Q4. * **random\_finals:** Accuracy increases from approximately 0.0 in Q1 to 0.2 in Q4. ### Key Observations * **Human Performance:** Generally high accuracy across most conditions, with some decrease in accuracy under "distracted" conditions. "randoms" condition shows a significant increase in accuracy across quartiles. * **Claude 3 Opus Performance:** More variable performance across conditions. "defaults" shows a significant increase in accuracy across quartiles. "random\_permuted\_pairs" shows a significant decrease in accuracy across quartiles. * **GPT-4 Performance:** Similar to Claude 3 Opus, GPT-4 shows variable performance. "defaults" and "random\_permuted\_pairs" show significant increases in accuracy across quartiles. ### Interpretation The data suggests that different subjects (Human, Claude 3 Opus, GPT-4) respond differently to various experimental conditions related to question types and permutations. Humans generally maintain high accuracy, while the AI models (Claude 3 Opus and GPT-4) exhibit more variability, indicating sensitivity to the specific question types and experimental setups. The "randoms" condition appears to be particularly challenging for the AI models in some cases, while the "defaults" condition seems to improve their accuracy over time (across quartiles). The error bars indicate the variability within each condition, suggesting that some conditions lead to more consistent performance than others. The trends observed in the plots can be used to understand the strengths and weaknesses of each subject under different cognitive loads and question formats. </details> Figure 14: Improvement in human and LLM accuracy by question number across different conditions. Error bars show standard errors. Further, humans and GPT-4 both improve as they see more questions over the course of a quiz. As seen in Figure 14, humans display a positive learning trend in 5 out of 8 conditions. GPT-4 displays a positive learning trend in a comparable 6 out of 8 conditions, with one of the conditions in which it does not display improvement resulting from it displaying near-perfect accuracy from start to finish (in the Only RHS condition). A.4 Further details of LLM performance Figure 15 shows the performance of all tested models in the Semantic Structure experiment. <details> <summary>extracted/5679376/Images/Aggregate_Accuracy_Comparison.png Details</summary>  ### Visual Description ## Bar Chart: Accuracy by Condition ### Overview The image is a bar chart comparing the accuracy of different AI models (GPT-4, GPT-3, Claude 3 Opus, Claude 2, Falcon-40B, Pythia-12B-Deduped) and humans across various conditions (Defaults, Distracted, Permuted Pairs, Permuted Questions, Random Permuted Pairs, Randoms, Only Rhs, Random Finals). The chart displays accuracy on the y-axis, ranging from 0.0 to 1.0, and the condition on the x-axis. Error bars are included on each bar, indicating the variability in the accuracy measurements. ### Components/Axes * **Title:** Accuracy by Condition * **X-axis:** Condition * Categories: Defaults, Distracted, Permuted Pairs, Permuted Questions, Random Permuted Pairs, Randoms, Only Rhs, Random Finals * **Y-axis:** Accuracy, ranging from 0.0 to 1.0 in increments of 0.2. * **Legend:** Located in the top-right corner. * Human (Blue) * GPT-4 (Orange) * GPT-3 (Green) * Claude 3 Opus (Red) * Claude 2 (Purple) * Falcon-40B (Brown) * Pythia-12B-Deduped (Pink) ### Detailed Analysis **Defaults Condition:** * Human (Blue): Accuracy ~0.88 +/- 0.05 * GPT-4 (Orange): Accuracy ~0.79 +/- 0.04 * GPT-3 (Green): Accuracy ~0.48 +/- 0.05 * Claude 3 Opus (Red): Accuracy ~0.82 +/- 0.04 * Claude 2 (Purple): Accuracy ~0.57 +/- 0.05 * Falcon-40B (Brown): Accuracy ~0.10 +/- 0.01 * Pythia-12B-Deduped (Pink): Accuracy ~0.11 +/- 0.01 **Distracted Condition:** * Human (Blue): Accuracy ~0.68 +/- 0.05 * GPT-4 (Orange): Accuracy ~0.48 +/- 0.04 * GPT-3 (Green): Accuracy ~0.30 +/- 0.03 * Claude 3 Opus (Red): Accuracy ~0.60 +/- 0.05 * Claude 2 (Purple): Accuracy ~0.39 +/- 0.04 * Falcon-40B (Brown): Accuracy ~0.09 +/- 0.01 * Pythia-12B-Deduped (Pink): Accuracy ~0.10 +/- 0.01 **Permuted Pairs Condition:** * Human (Blue): Accuracy ~0.84 +/- 0.05 * GPT-4 (Orange): Accuracy ~0.54 +/- 0.04 * GPT-3 (Green): Accuracy ~0.52 +/- 0.03 * Claude 3 Opus (Red): Accuracy ~0.59 +/- 0.05 * Claude 2 (Purple): Accuracy ~0.49 +/- 0.04 * Falcon-40B (Brown): Accuracy ~0.05 +/- 0.01 * Pythia-12B-Deduped (Pink): Accuracy ~0.05 +/- 0.01 **Permuted Questions Condition:** * Human (Blue): Accuracy ~0.80 +/- 0.05 * GPT-4 (Orange): Accuracy ~0.75 +/- 0.04 * GPT-3 (Green): Accuracy ~0.54 +/- 0.03 * Claude 3 Opus (Red): Accuracy ~0.99 +/- 0.05 * Claude 2 (Purple): Accuracy ~0.64 +/- 0.04 * Falcon-40B (Brown): Accuracy ~0.05 +/- 0.01 * Pythia-12B-Deduped (Pink): Accuracy ~0.05 +/- 0.01 **Random Permuted Pairs Condition:** * Human (Blue): Accuracy ~0.72 +/- 0.05 * GPT-4 (Orange): Accuracy ~0.33 +/- 0.04 * GPT-3 (Green): Accuracy ~0.05 +/- 0.03 * Claude 3 Opus (Red): Accuracy ~0.20 +/- 0.05 * Claude 2 (Purple): Accuracy ~0.20 +/- 0.04 * Falcon-40B (Brown): Accuracy ~0.07 +/- 0.01 * Pythia-12B-Deduped (Pink): Accuracy ~0.05 +/- 0.01 **Randoms Condition:** * Human (Blue): Accuracy ~0.80 +/- 0.05 * GPT-4 (Orange): Accuracy ~0.18 +/- 0.04 * GPT-3 (Green): Accuracy ~0.05 +/- 0.03 * Claude 3 Opus (Red): Accuracy ~0.35 +/- 0.05 * Claude 2 (Purple): Accuracy ~0.05 +/- 0.04 * Falcon-40B (Brown): Accuracy ~0.05 +/- 0.01 * Pythia-12B-Deduped (Pink): Accuracy ~0.05 +/- 0.01 **Only Rhs Condition:** * Human (Blue): Accuracy ~0.89 +/- 0.05 * GPT-4 (Orange): Accuracy ~0.77 +/- 0.04 * GPT-3 (Green): Accuracy ~0.39 +/- 0.03 * Claude 3 Opus (Red): Accuracy ~0.24 +/- 0.05 * Claude 2 (Purple): Accuracy ~0.25 +/- 0.04 * Falcon-40B (Brown): Accuracy ~0.06 +/- 0.01 * Pythia-12B-Deduped (Pink): Accuracy ~0.05 +/- 0.01 **Random Finals Condition:** * Human (Blue): Accuracy ~0.48 +/- 0.05 * GPT-4 (Orange): Accuracy ~0.07 +/- 0.04 * GPT-3 (Green): Accuracy ~0.09 +/- 0.03 * Claude 3 Opus (Red): Accuracy ~0.28 +/- 0.05 * Claude 2 (Purple): Accuracy ~0.35 +/- 0.04 * Falcon-40B (Brown): Accuracy ~0.05 +/- 0.01 * Pythia-12B-Deduped (Pink): Accuracy ~0.18 +/- 0.01 ### Key Observations * Human accuracy is generally high across all conditions, with the exception of "Random Finals". * Claude 3 Opus shows the highest accuracy in the "Permuted Questions" condition, reaching nearly 1.0. * Falcon-40B and Pythia-12B-Deduped consistently show low accuracy across all conditions. * The accuracy of all models varies significantly depending on the condition. * The error bars indicate some variability in the accuracy measurements, which should be considered when interpreting the results. ### Interpretation The bar chart provides a comparative analysis of the accuracy of different AI models and humans under various conditions. The data suggests that the performance of AI models is highly dependent on the specific task or condition. For example, Claude 3 Opus excels in "Permuted Questions," while all models struggle with "Random Finals." The consistently low accuracy of Falcon-40B and Pythia-12B-Deduped suggests that these models may not be well-suited for the tasks represented by these conditions. The variability indicated by the error bars highlights the need for caution when drawing definitive conclusions based on these results. The chart demonstrates the strengths and weaknesses of each model, providing valuable insights for selecting the appropriate model for a given task. The human performance serves as a benchmark, highlighting the gap between current AI capabilities and human-level intelligence in these specific scenarios. </details> Figure 15: Performance of all tested models in the Semantic Structure experiment. Error bars show standard errors. Figure 16 shows the variation in the performance of GPT-4 in two conditions (Only RHS and Random Finals) across various small differences in prompting strategy. With small differences in prompting strategy, performance in the Only RHS condition varies between approximately 20% and approximately 100%. Similarly, small differences in the prompting strategy yield performance in the Random Finals condition that varies between near-zero and close to 40%. Such variation is very significant, but appears in general to be fairly explicable. In the Only RHS condition, the majority of the variation appears to come from setting up the prompt in such a way that it is clear that a final term is desired next, as opposed to a new question. In the other conditions, an arrow separator that divides left- and right-hand side terms is the final element of the prompt, thus suggesting that a right-hand term is appropriate as the next token. In the Only RHS condition, this trailing separator was initially not present, and thus the models often responded by beginning a new question rather than by completing the last question presented. Re-introducing arrow separators and making other small changes designed to more clearly indicate when a question has not yet been completed eliminates these kinds of errors and drastically increases performance. In the Random Finals condition, a significant improvement comes from changing the instruction sentence from one that specifies that a drawing of the left-hand side is requested, to an instruction sentence specifying that various patterns will be shown after which the last should be completed. This is reasonable, as in this condition the final left-hand term is misleading and so an instruction focusing attention on it is expected to reduce performance. As expected, no performance improvement is observed when replacing the set of random final words with different ones. Finally, a performance boost is observed when adding additional newlines (instead of only having a clear line between each question, we now also include a clear line between each line of a question). It is not clear why this should improve performance. <details> <summary>extracted/5679376/Images/gpt4_tweak_dependance.png Details</summary>  ### Visual Description ## Bar Chart: GPT-4 Accuracy by Presentation Style ### Overview The image is a bar chart comparing the accuracy of GPT-4 under different presentation styles. The chart displays accuracy on the y-axis (ranging from 0.0 to 1.0) and presentation styles on the x-axis. Error bars are included on each bar, indicating the variability in accuracy. ### Components/Axes * **Title:** GPT-4 Accuracy by Presentation Style * **Y-axis:** * Label: Accuracy * Scale: 0.0 to 1.0, with ticks at 0.0, 0.2, 0.4, 0.6, 0.8, and 1.0 * **X-axis:** * Categories (Presentation Styles): * Only Rhs * Only Rhs + Arrow * Random Finals Drawing * Random Finals Pattern * Random Finals Randomer * Random Finals Newlines * **Bars:** All bars are blue. * **Error Bars:** Black vertical lines extending above and below the top of each bar, indicating the standard deviation or error range. ### Detailed Analysis * **Only Rhs:** Accuracy is approximately 0.25, with a small error bar. * **Only Rhs + Arrow:** Accuracy is approximately 0.84, with a small error bar. * **Random Finals Drawing:** Accuracy is approximately 0.08, with a small error bar. * **Random Finals Pattern:** Accuracy is approximately 0.23, with a small error bar. * **Random Finals Randomer:** Accuracy is approximately 0.23, with a small error bar. * **Random Finals Newlines:** Accuracy is approximately 0.31, with a small error bar. ### Key Observations * The "Only Rhs + Arrow" presentation style yields significantly higher accuracy compared to all other styles. * "Random Finals Drawing" results in the lowest accuracy. * The remaining presentation styles ("Only Rhs", "Random Finals Pattern", "Random Finals Randomer", and "Random Finals Newlines") show relatively low and similar accuracy levels. ### Interpretation The chart suggests that the way information is presented significantly impacts GPT-4's accuracy. Specifically, including arrows along with "Rhs" (Right-hand side?) leads to a substantial improvement in performance. Conversely, using random final drawings degrades performance. The other "Random Finals" variations have a modest impact on accuracy, performing slightly worse than "Only Rhs". This indicates that certain presentation styles are more conducive to accurate processing by GPT-4, highlighting the importance of structured and clear input formatting. </details> Figure 16: Dependence of GPT-4 accuracy on prompt variations. Error bars show standard errors. | Prompt | We are conducting an experiment on general reasoning abilities. Below we will show you various words and drawings of each, after which you will need to complete the last drawing. Respond as concisely as possible with only the last drawing. Question 1: chicken => ! spider => ! cat => * horse => * ant => ! dog => * bee => ! human => * Question 2: car => * tricycle => ! motorcycle => * skateboard => ! bicycle => ! unicycle => ——— SAMPLED RESPONSE SET ——— ! [Continuation omitted] —————————————- ! [Continuation omitted] —————————————- * [Continuation omitted] —————————————- ! [Continuation omitted] —————————————- * [Continuation omitted] | | --- | --- | Table 6: Table showing an illustrative response from Falcon-40B in the Categorial condition of the Semantic Content experiment. Observe that the model provides incorrect and correct responses to the question, seeming to recognize the form of a correct response but to not reason further about correctness. | Prompt | We are conducting an experiment on general reasoning abilities. Below we will show you various words and drawings of each, after which you will need to complete the last drawing. Respond as concisely as possible with only the last drawing. Question 1: spider => * * * * * * * * human => * * cat => * * * * chicken => * * dog => * * * * horse => * * * * bee => * * * * * * Question 2: motorcycle => * * tricycle => * * * bicycle => * * unicycle => * car => ——— SAMPLED RESPONSE SET ——— * * * —————————————- * * —————————————- * * —————————————- * * —————————————- * * | | --- | --- | Table 7: Table showing an illustrative response from GPT-4 in the numerical condition of the Semantic Content experiment. Observe that the model fails to correctly relate the number of characters to the numerical property of the object, in this case the number of wheels that a car has. | Prompt | * * * + * = * * * * * * * * * – * * = * * * * * * * * * – * * * = | | --- | --- | | Responses | * * * * * * (first response) * * * * * * (second response) * * * * * * * (third response) | | Expected result | * * * | Table 8: Table showing a sanity check that GPT-4 fails to reason about the number of characters in the expected way. Settings: temperature 1, maximum length 256, top P 1.