# Emergent Analogical Reasoning in Large Language Models
## Emergent Analogical Reasoning in Large Language Models
Taylor Webb 1,* , Keith J. Holyoak 1 , and Hongjing Lu 1,2
Department of Psychology 2 Department of Statistics University of California, Los Angeles, CA, USA * Correspondence to: taylor.w.webb@gmail.com
## Abstract
The recent advent of large language models has reinvigorated debate over whether human cognitive capacities might emerge in such generic models given sufficient training data. Of particular interest is the ability of these models to reason about novel problems zero-shot , without any direct training. In human cognition, this capacity is closely tied to an ability to reason by analogy. Here, we performed a direct comparison between human reasoners and a large language model (the text-davinci-003 variant of GPT-3) on a range of analogical tasks, including a non-visual matrix reasoning task based on the rule structure of Raven's Standard Progressive Matrices. We found that GPT-3 displayed a surprisingly strong capacity for abstract pattern induction, matching or even surpassing human capabilities in most settings; preliminary tests of GPT-4 indicated even better performance. Our results indicate that large language models such as GPT-3 have acquired an emergent ability to find zero-shot solutions to a broad range of analogy problems.
## 1 Introduction
Analogical reasoning is at the heart of human intelligence and creativity. When confronted with an unfamiliar problem, human reasoners can often identify a reasonable solution through a process of structured comparison to a more familiar situation. 1 This process is an essential part of human reasoning in domains ranging from everyday problem-solving 2 to creative thought and scientific innovation. 3 Indeed, tests of analogical reasoning ability are uniquely effective as measures of fluid intelligence: the capacity to reason about novel problems. 4, 5
Recently, there has been considerable debate about whether and how a capacity for analogical thought might be captured in deep learning systems. 6 Much of this recent work has focused on training neural networks on very large datasets (sometimes containing millions of problems). 7, 8 Though this is a challenging task that has spurred the development of some interesting approaches, 9-12 it does not address the issue of whether analogical reasoning can emerge zero-shot (i.e., without direct training), the capacity most central to human thought.
An alternative approach, also based on deep learning, involves large language models (LLMs). 13 LLMs have recently sparked great interest (and controversy) for their potential to perform few-shot, and even zero-shot, reasoning. These models employ relatively generic neural network architectures with up to billions of parameters, and are trained using a simple predictive objective (predicting the next token in a sequence of text) with massive web-based text corpora consisting of billions of tokens. Though there is significant debate about the capabilities of these models, 14 a potential advantage is their ability to solve problems with little direct training, sometimes requiring only a few examples, or even a simple task instruction (typically without any updating of model parameters). This feature raises the question of whether LLMs might be capable of human-like, zero-shot analogical reasoning.
To answer this question, we evaluated the language model GPT-3 13 on a range of zero-shot analogy tasks, and performed direct comparisons with human behavior. These tasks included a novel text-based matrix reasoning task based on the rule structure of Raven's Standard Progressive Matrices (SPM), 15 a visual analogy problem set commonly viewed as one of the best measures of fluid intelligence. 5 Unlike the original visual SPM problems, our Digit Matrices task was purely text-based so that it could be used to evaluate GPT-3's ability to induce abstract
Published at Nature Human Behaviour (2023) https://doi.org/10.1038/s41562-023-01659-w
Figure 1: Summary of results. Matrix reasoning results show average accuracy on all problems in Digit Matrices problem set, a novel text-based matrix reasoning task designed to emulate Raven's Standard Progressive Matrices (SPM) problems. 15 Note that the Digit Matrices were purely text-based, and therefore do not test for the ability to perform abstract reasoning directly over visual inputs, as in the original SPM. Letter string results show average performance for novel letter string analogy problem set, based on problems from Hofstadter and Mitchell. 16 Both matrix reasoning and letter string results reflect performance on generative task. Verbal analogy results show average performance on UCLA Verbal Analogy Test. 17 Story analogy problems involved identification of analogous stories based on higher-order relations, using materials from Gentner et al. 18 Both verbal and story analogy results reflect multiple-choice accuracy, with chance performance indicated by gray horizontal line. Chance performance for the two generative tasks (matrix reasoning and letter string analogies) is close to zero, due to the very large space of possible generative responses. Black error bars represent standard error of the mean for average performance across participants. Each dot represents accuracy for a single participant (matrix reasoning, N=40; letter string analogies, N=57; verbal analogies, N=57; story analogies, N=54). Gray error bars represent 95% binomial confidence intervals for average performance across multiple problems.
<details>
<summary>Image 1 Details</summary>
### Visual Description
## Bar Chart: GPT-3 vs. Human Accuracy on Analogy Tasks
### Overview
The image is a grouped bar chart comparing the accuracy of two entities, "GPT-3" and "Human," across four different analogy-based reasoning tasks. The chart includes error bars and scattered data points for each condition, indicating variability in performance.
### Components/Axes
* **Chart Type:** Grouped bar chart with overlaid scatter points and error bars.
* **Y-Axis:** Labeled "Accuracy." The scale runs from 0 to 1, with major tick marks at 0, 0.2, 0.4, 0.6, 0.8, and 1.
* **X-Axis:** Four categorical task types are listed:
1. Matrix reasoning
2. Letter string analogies
3. Verbal analogies
4. Story analogies
* **Legend:** Located in the top-right corner of the chart area.
* **Dark Purple Bar:** Labeled "GPT-3"
* **Light Blue Bar:** Labeled "Human"
* **Visual Elements:**
* Each task category has two adjacent bars (GPT-3 left, Human right).
* A thin black vertical line (error bar) extends from the top of each bar.
* Numerous small black dots are scattered vertically above each bar, likely representing individual data points or trials.
### Detailed Analysis
**1. Matrix reasoning:**
* **Trend:** GPT-3 shows higher accuracy than Human.
* **GPT-3 (Dark Purple):** Accuracy is approximately **0.80**. The error bar extends from roughly 0.78 to 0.82.
* **Human (Light Blue):** Accuracy is approximately **0.60**. The error bar extends from roughly 0.58 to 0.62.
* **Data Points:** A dense cluster of dots is visible above both bars, spanning from below 0.2 to above 1.0 for GPT-3, and from ~0.2 to ~0.9 for Human.
**2. Letter string analogies:**
* **Trend:** GPT-3 shows slightly higher accuracy than Human.
* **GPT-3 (Dark Purple):** Accuracy is approximately **0.48**. The error bar extends from roughly 0.46 to 0.50.
* **Human (Light Blue):** Accuracy is approximately **0.43**. The error bar extends from roughly 0.41 to 0.45.
* **Data Points:** Dots are scattered above both bars, ranging from ~0.1 to ~0.9.
**3. Verbal analogies:**
* **Trend:** GPT-3 shows higher accuracy than Human. This is the highest accuracy for GPT-3 across all tasks.
* **GPT-3 (Dark Purple):** Accuracy is approximately **0.92**. The error bar extends from roughly 0.90 to 0.94.
* **Human (Light Blue):** Accuracy is approximately **0.85**. The error bar extends from roughly 0.83 to 0.87.
* **Data Points:** A very dense cluster of dots is visible above both bars, concentrated near the top of the scale (0.8 to 1.0).
**4. Story analogies:**
* **Trend:** Human shows higher accuracy than GPT-3. This is the only task where Human performance exceeds GPT-3.
* **GPT-3 (Dark Purple):** Accuracy is approximately **0.71**. The error bar extends from roughly 0.69 to 0.73.
* **Human (Light Blue):** Accuracy is approximately **0.87**. The error bar extends from roughly 0.85 to 0.89.
* **Data Points:** Dots are scattered above both bars, with a dense cluster for Human near the top.
### Key Observations
1. **Performance Reversal:** The relative performance of GPT-3 and Human flips between task types. GPT-3 leads in Matrix, Letter string, and Verbal analogies, while Human leads in Story analogies.
2. **Highest & Lowest Scores:** The highest overall accuracy is achieved by GPT-3 on Verbal analogies (~0.92). The lowest overall accuracy is by Human on Letter string analogies (~0.43).
3. **Variability:** The scattered dots indicate significant trial-to-trial variability for both entities across all tasks. The error bars suggest the mean accuracy estimates are relatively precise.
4. **Task Difficulty:** Based on the combined performance, "Letter string analogies" appears to be the most difficult task (lowest accuracies), while "Verbal analogies" appears to be the easiest (highest accuracies).
### Interpretation
The data suggests a dissociation in cognitive strengths between the AI model (GPT-3) and humans. GPT-3 demonstrates superior performance on tasks that may rely more on structured pattern recognition and formal linguistic rules (Matrix reasoning, Letter string, Verbal analogies). In contrast, humans outperform GPT-3 on "Story analogies," a task likely requiring deeper narrative comprehension, contextual integration, and perhaps real-world knowledge that is less formally structured.
This pattern implies that while the model excels at certain types of abstract and verbal reasoning, human cognition retains an advantage in tasks grounded in narrative and contextual understanding. The high variability (scattered dots) for both groups indicates that performance on these analogy tasks is not uniform and can fluctuate significantly from one instance to another. The chart effectively highlights that "accuracy" on analogy tasks is not a monolithic measure but is highly dependent on the specific cognitive demands of the task format.
</details>
rules (though not the ability to do so directly from visual inputs). Strikingly, we found that GPT-3 performed as well or better than college students in most conditions, despite receiving no direct training on this task. GPT3 also displayed strong zero-shot performance on letter string analogies, 16 four-term verbal analogies, 17,19-21 and identification of analogies between stories. 18,22,23 These results add to the growing body of work characterizing the emergent capabilities of LLMs, 24-28 and suggest that the most sophisticated LLMs may already possess an emergent capacity to reason by analogy.
## 2 Results
We evaluated the language model GPT-3 on a set of analogy tasks, and compared its performance to human behavior. GPT-3 is a large-scale (175B parameters), transformer-based 29 language model developed by OpenAI. 13 The original base model was trained on a web-based corpus of natural language consisting of over 400 billion tokens, using a training objective based on next-token prediction (given a string of text, the model is trained to predict the token most likely appear next). A number of variants on this base model have since been developed by fine-tuning it in various ways. These include training the model to generate code, 30 and training it to respond appropriately to human prompts, using either supervised learning or reinforcement learning from human feedback (RLHF). 31 Our evaluation focused on the most recent model variant, text-davinci-003 (here referred to simply as 'GPT-3'), which was the first to incorporate RLHF (along with the concurrently released, but distinct, ChatGPT model). We found that text-davinci-003 displayed particularly strong performance on our analogy tasks, but earlier model variants also performed well in some task settings, suggesting that multiple factors contributed to text-davinci-003's analogical capabilities (Supplementary Figures 1- 3). See Section S2 for further discussion.
Our evaluation featured four separate task domains, each designed to probe different aspects of analogical reasoning: 1) text-based matrix reasoning problems, 2) letter-string analogies, 3) four-term verbal analogies, and 4) story analogies. For each task domain, we performed a direct comparison with human behavior, assessing both overall performance and error patterns across a range of conditions relevant to human analogical reasoning. Figure 1 shows a summary of these results. We also performed a qualitative analysis of GPT-3's ability to use analogical reasoning to solve problems.
## 2.1 Matrix reasoning problems
We designed a text-based matrix reasoning task, the Digit Matrices, to emulate the structure of Raven's Standard Progressive Matrices (SPM). 15 The task is illustrated in Figure 2. The dataset was structured similarly to the work of Matzen et al., 32 who created, and behaviorally validated, a visual matrix reasoning dataset with the same rule structure as the original SPM. The Digit Matrices dataset thus has a similar rule structure to SPM, but is guaranteed to be novel for both humans and LLMs.
Digit Matrix problems consisted of either digit transformations (Figures 2b- 2e) or logic problems (Figures 2f- 2g). Transformation problems were defined based on a set of three rule types constant (Figure 2c), distribution-of-3 (Figure 2d), and progression (Figure 2e) - and consisted of one or more rules per problem. When multiple rules were present (Figure 2b), each rule was bound to a different spatial location within each cell (e.g., one rule was bound to the left digit in each cell, and another rule was bound to the right digit). Logic problems were defined based on set relations OR , AND , and XOR - and involved only a single rule per problem. In some logic problems, the corresponding elements were spatially aligned (Figure 2f), whereas in others they were permuted (Figure 2g). We hypothesized that spatial alignment would be beneficial when solving the problems via analogical mapping, as it should highlight the isomorphism . 33 Digit Matrices problems were presented to GPT-3 without any prompt or in-context task examples.
Figure 3 shows zero-shot performance on the Digit Matrices problems for GPT-3 and human participants (N=40, UCLA undergraduates). GPT-3 surpassed the average level of human performance on all problem types, both when generating answers directly (Figure 3a; logistic regression, main effect of GPT-3 vs. human participants: odds ratio (OR) = 1 . 88, p = 0 . 005, 95% confidence intervals (CI) = [1 . 21 , 2 . 91]), and when selecting from a set of answer choices (Figure 3b; main effect of GPT-3 vs. human participants: OR = 6 . 27, p = 2 . 3 Γ 10 -8 , CI = [3 . 28 , 11 . 99]). It is worth emphasizing, however, that participants displayed a range of performance levels on this task, with some participants outperforming GPT-3 (indeed, the best participant answered every problem correctly).
In addition to showing strong overall performance, GPT-3's pattern of performance across problem subtypes was similar to that observed in human participants (correlation analysis: r (30) = 0 . 39, p = 0 . 027). This correlation was driven both by the pattern of performance across major problem types (one-, two-, three-rule, and logic problems; main effect of problem type on generative accuracy: OR = 0 . 5, p = 2 Γ 10 -16 , CI = [0 . 44 , 0 . 56],; main effect of problem type on multiple-choice accuracy: OR = 0 . 56, p = 2 Γ 10 -16 , CI = [0 . 5 , 0 . 64]), and by differences within each problem type. Problems with progression rules were more difficult than those without them (Figure 3c; main effect of progression vs. no progression, human participants: OR = 0 . 41, p = 0 . 0001, CI = [0 . 24 , 0 . 69]; GPT-3: OR = 0 . 07, p = 1 . 9 Γ 10 -5 , CI = [0 . 02 , 0 . 24]); for multi-rule problems, performance was negatively correlated with the number of unique rules in each problem, even when holding constant the number of total rules (Figure 3d; main effect of number of unique rules, human participants: OR = 0 . 61, p = 0 . 0047, CI = [0 . 44 , 0 . 86]; GPT-3: OR = 0 . 25, p = 3 Γ 10 -10 , CI = [0 . 17 , 0 . 39]); and logic problems were more difficult when the corresponding elements were spatially permuted vs. aligned (Figure 3e; main effect of spatial alignment, human participants: OR = 0 . 52, p = 0 . 0017, CI = [0 . 35 , 0 . 79]; GPT-3: OR = 0 . 06, p = 2 Γ 10 -11 , CI = [0 . 03 , 0 . 14]). These effects replicate well-known characteristics of human analogical reasoning: problems defined by relations (e.g., progression) are typically more difficult than problems defined by the features of individual entities (e.g., constant or distribution-of-3); 32,34 problem difficulty is typically driven by the degree of relational complexity, as defined by the number of unique relations; 35 and analogical mapping is easier when a greater number of constraints supports the correct mapping (as is the case in the spatially aligned logic problems). 33 GPT-3's pattern of performance thus displayed many of the characteristics of a human-like analogical mapping process. We also found that GPT-3 was sensitive to contextual information in ways that both improved and impaired its performance, similar to human reasoners (Supplementary Figure 4).
It is important to highlight the differences between the Digit Matrices and traditional visual matrix reasoning problems. In order to solve visual matrix reasoning problems, pixel-level inputs must be parsed into objects, and visual attributes (shape, size, etc.) must be disentangled. In the Digit Matrices, the text-based inputs are already
Figure 2: Matrix reasoning problems. (a) Example problem depicting structure of Raven's Progressive Matrices. 15 Problems consist of a 3 Γ 3 matrix populated with geometric forms, in which each row or column is governed by the same set of abstract rules. Problem solvers must identify these rules, and use them to infer the missing cell in the lower right, by selecting from the set of 8 choices below. (b) Example problem illustrating the novel Digit Matrices problem set. Problems consist of a 3 Γ 3 matrix, in which each cell is demarcated by brackets, and populated by digits. The problems are governed by the same rule structure as Raven's Standard Progressive Matrices. The example problems in (a) and (b) are structurally isomorphic (i.e., governed by the same set of rules). The reader is encouraged to derive the solution to each problem. The solutions to both problems are given in Supplementary Section S1 . Problems were governed either by one or more transformation rules (b-e), or by a single logic rule (f,g). (c) Constant rule: same digit appears across either rows or columns. (d) Distribution-of-3 rule: same set of 3 digits appears in each row or column, but with order varied. (e) Progression rule: digits either increase or decrease, by values of 1 or 2, across rows or columns. In the example shown here, digits increase by 2 across rows. (f) OR rule: the set of digits present in a particular row or column are defined as the union of the sets present in the other rows or columns. In the illustrated example, the digits in the second column are formed from the union of the sets in the first and third columns. This example illustrates how the spatial alignment of the corresponding elements can make it easier to intuitively grasp the underlying rule. (g) More challenging logic problem governed by same rule (OR), but in which the corresponding elements are spatially permuted. Other logic problems were governed either by an AND rule or an XOR rule (not pictured).
<details>
<summary>Image 2 Details</summary>
### Visual Description
## Abstract Reasoning Puzzle Set: Pattern Recognition and Logical Operations
### Overview
The image displays a composite figure containing seven distinct panels (labeled **a** through **g**) that present abstract reasoning puzzles. These puzzles involve visual pattern completion, numerical matrix completion, and the application of specific logical rules (Constant, Distribution-of-3, Progression, OR operations). The overall purpose appears to be the assessment or demonstration of inductive reasoning and rule discovery.
### Components/Axes
The image is segmented into seven labeled panels:
* **Panel a (Top-Left):** A visual matrix puzzle. It consists of a 3x3 grid with the bottom-right cell empty. Below the grid are eight numbered answer choices (1-8), each containing a different arrangement of shapes.
* **Panel b (Top-Right):** A numerical matrix puzzle. It shows a 3x3 grid of number triplets with the bottom-right cell containing a question mark `[ ? ]`. Below are eight numbered answer choices (1-8), each a set of three number triplets.
* **Panels c, d, e, f, g (Bottom Row):** These panels are labeled with specific logical rule names and each contains a 3x3 grid of number triplets with the bottom-right cell missing (marked `[ ? ]`). They serve as examples or test cases for different rule types.
* **c:** Labeled "Constant"
* **d:** Labeled "Distribution-of-3"
* **e:** Labeled "Progression"
* **f:** Labeled "OR (aligned)"
* **g:** Labeled "OR (permuted)"
### Detailed Analysis
**Panel a: Visual Matrix Puzzle**
* **Grid Content (Row-wise):**
* Row 1: [White Triangle], [Black Circle], [Grey Square]
* Row 2: [Two Grey Circles], [Two White Squares], [Two Black Triangles]
* Row 3: [Three Black Squares], [Three Grey Triangles], [ ? ]
* **Answer Choices (1-8):**
1. [Three Grey Squares]
2. [Two Black Circles]
3. [One White Triangle]
4. [Three Grey Triangles]
5. [Three White Circles]
6. [Two Black Triangles]
7. [Three White Squares]
8. [One Grey Circle]
**Panel b: Numerical Matrix Puzzle**
* **Main Grid Content (Row-wise, each cell is a triplet `[x y z]`):**
* Row 1: `[5 9 3]`, `[8 9 2]`, `[1 9 7]`
* Row 2: `[8 4 7]`, `[1 4 3]`, `[5 4 2]`
* Row 3: `[1 2 2]`, `[5 2 7]`, `[ ? ]`
* **Answer Choices (1-8):**
1. `[5 2 3]`
2. `[5 4 2]`
3. `[1 2 7]`
4. `[8 9 7]`
5. `[5 9 3]`
6. `[1 4 3]`
7. `[8 2 3]`
8. `[5 2 7]`
**Panels c-g: Rule-Specific Matrices**
* **c (Constant):** Grid shows `[5][1][9]` in the first two rows. The third row is `[5][1][?]`. The rule implies the third row should be identical to the first two.
* **d (Distribution-of-3):** Grid shows `[6][2][4]`, `[2][4][6]`, `[4][6][?]`. The numbers 2, 4, 6 are distributed across rows/columns.
* **e (Progression):** Grid shows `[3][5][7]`, `[1][3][5]`, `[5][7][?]`. Each row/column shows an arithmetic progression with a common difference of 2.
* **f (OR (aligned)):** Grid shows `[7][74][4]`, `[97][9748][48]`, `[9][98][?]`. This appears to involve a logical OR operation on digits in aligned positions.
* **g (OR (permuted)):** Grid shows `[1][71][7]`, `[10][5071][75]`, `[0][05][?]`. Similar to (f) but with permuted digit positions.
### Key Observations
1. **Dual Modality:** The puzzle set tests both visual-spatial reasoning (Panel a) and numerical-logical reasoning (Panels b-g).
2. **Rule Diversity:** Panels c-g explicitly name different logical operations, suggesting the overarching task is to identify which rule governs a given matrix.
3. **Structural Consistency:** All numerical matrices (b-g) share a 3x3 structure with triplets, while the visual matrix (a) uses a 3x3 grid of single or grouped shapes.
4. **Answer Format:** Both puzzle types (a and b) provide eight discrete multiple-choice options for the missing cell.
5. **Spatial Layout:** Panel a is positioned top-left, Panel b top-right, and the five rule panels (c-g) are arranged in two rows along the bottom.
### Interpretation
This image is a technical illustration of abstract reasoning test items, likely from a cognitive science or psychometrics context. It demonstrates the variety of inductive reasoning problems used to assess fluid intelligence.
* **What it demonstrates:** The figure catalogs different *types* of logical rules that can underlie a pattern. Solving these puzzles requires the test-taker to induce the rule from the given examples (the completed cells) and apply it to find the missing element.
* **Relationship between elements:** The labeled panels (c-g) serve as a "key" or taxonomy of possible rules. Panel b presents a concrete, unlabeled puzzle that likely requires the solver to first deduce which of these rules (or a combination) applies. Panel a shows a parallel task in the visual domain, where the rule might involve shape, color, and quantity progressions.
* **Notable patterns:** The "OR" rules (f, g) are particularly complex, suggesting operations on digit strings rather than simple arithmetic. The "Distribution-of-3" rule (d) implies a constraint-satisfaction logic. The presence of these varied rules indicates the test is designed to discriminate between different levels or types of logical reasoning ability.
* **Underlying purpose:** The image is not a data chart but a schematic of problem types. Its value lies in clearly presenting the structure and variety of inductive reasoning challenges, making it useful for researchers designing tests, educators, or individuals studying for cognitive assessments. The explicit labeling of rules in c-g is especially informative, as it reveals the "answer logic" that a test-taker must discover implicitly.
</details>
parsed and disentangled, essentially providing GPT-3 (which is not capable of visual processing) with pseudo-symbolic inputs. Interestingly, despite these significant differences, we found that overall error rates for human participants
Figure 3: Matrix reasoning results. GPT-3 matched or exceeded human performance for zero-shot Digit Matrices. (a) Generative accuracy for major problem types, including transformation problems with between one and three rules, and logic problems. (b) Multiple-choice accuracy for major problem types. (c) Two-rule problems with at least one progression rule were more difficult than those without. (d) For three-rule problems, performance was a function of the number of unique rules. (e) Spatially permuted logic problems were more difficult than spatially aligned problems. Human results reflect average performance for N=40 participants (UCLA undergraduates). Black error bars represent standard error of the mean across participants. Each dot represents accuracy for a single participant. Gray error bars represent 95% binomial confidence intervals for average performance across multiple problems. Note that the rightmost bar in (d) does not show individual scores because each participant only completed a single problem with three unique rules.
<details>
<summary>Image 3 Details</summary>
### Visual Description
\n
## Bar Charts: GPT-3 vs. Human Performance on Reasoning Problems
### Overview
The image contains five bar charts (labeled a, b, c, d, e) comparing the performance of GPT-3 (dark purple bars) and Human participants (light blue bars) on various types of reasoning problems. The charts measure accuracy (generative or multiple-choice) across different problem categories and conditions. All text is in English.
### Components/Axes
* **Legend:** Located in the top-right corner of subplot **b**. It defines the two data series:
* **GPT-3:** Dark purple bars.
* **Human:** Light blue bars.
* **Subplot a:**
* **Title/Label:** "a" (top-left corner).
* **Y-axis:** "Generative accuracy" (scale 0 to 1).
* **X-axis:** "Problem type" with categories: "1-rule", "2-rule", "3-rule", "Logic".
* **Subplot b:**
* **Title/Label:** "b" (top-left corner).
* **Y-axis:** "Multiple choice accuracy" (scale 0 to 1).
* **X-axis:** "Problem type" with categories: "1-rule", "2-rule", "3-rule", "Logic".
* **Subplot c:**
* **Title/Label:** "c" (top-left corner). **Chart Title:** "Two-rule problems".
* **Y-axis:** "Generative accuracy" (scale 0 to 1).
* **X-axis:** Categories: "No progression", "Progression".
* **Subplot d:**
* **Title/Label:** "d" (top-left corner). **Chart Title:** "Three-rule problems".
* **Y-axis:** "Generative accuracy" (scale 0 to 1).
* **X-axis:** "Number of unique rules" with categories: "1", "2", "3".
* **Subplot e:**
* **Title/Label:** "e" (top-left corner). **Chart Title:** "Logic problems".
* **Y-axis:** "Generative accuracy" (scale 0 to 1).
* **X-axis:** Categories: "Aligned", "Permuted".
### Detailed Analysis
**Subplot a (Generative Accuracy by Problem Type):**
* **Trend:** Accuracy for both GPT-3 and Humans decreases as problem complexity increases (from 1-rule to Logic). GPT-3 consistently outperforms Humans.
* **Data Points (Approximate):**
* **1-rule:** GPT-3 β 1.0, Human β 0.9.
* **2-rule:** GPT-3 β 0.85, Human β 0.62.
* **3-rule:** GPT-3 β 0.70, Human β 0.58.
* **Logic:** GPT-3 β 0.80, Human β 0.42.
**Subplot b (Multiple Choice Accuracy by Problem Type):**
* **Trend:** Similar decreasing trend with complexity. GPT-3 maintains a significant lead over Humans.
* **Data Points (Approximate):**
* **1-rule:** GPT-3 β 1.0, Human β 0.90.
* **2-rule:** GPT-3 β 0.92, Human β 0.72.
* **3-rule:** GPT-3 β 0.94, Human β 0.65.
* **Logic:** GPT-3 β 0.85, Human β 0.55.
**Subplot c (Two-rule Problems - Generative Accuracy):**
* **Trend:** For both groups, accuracy is lower on "Progression" problems compared to "No progression" problems.
* **Data Points (Approximate):**
* **No progression:** GPT-3 β 0.98, Human β 0.72.
* **Progression:** GPT-3 β 0.74, Human β 0.52.
**Subplot d (Three-rule Problems - Generative Accuracy):**
* **Trend:** Accuracy for both groups decreases as the "Number of unique rules" increases from 1 to 3.
* **Data Points (Approximate):**
* **1 unique rule:** GPT-3 β 0.90, Human β 0.68.
* **2 unique rules:** GPT-3 β 0.65, Human β 0.52.
* **3 unique rules:** GPT-3 β 0.38, Human β 0.48.
**Subplot e (Logic Problems - Generative Accuracy):**
* **Trend:** Both groups perform significantly better on "Aligned" problems than on "Permuted" problems.
* **Data Points (Approximate):**
* **Aligned:** GPT-3 β 0.98, Human β 0.50.
* **Permuted:** GPT-3 β 0.64, Human β 0.34.
### Key Observations
1. **Consistent Superiority:** GPT-3's accuracy is higher than Human accuracy across all problem types and conditions shown.
2. **Complexity Impact:** Performance for both entities degrades as the logical complexity of the problem increases (more rules, progression, permutation).
3. **Largest Performance Gap:** The most substantial difference appears in the "Logic" problem category (subplot a) and the "Aligned" condition within logic problems (subplot e).
4. **Human Variability:** The error bars (likely standard error or confidence intervals) for Human performance are generally larger than those for GPT-3, indicating greater variability in human responses.
5. **Notable Crossover:** In subplot d, for "3 unique rules," Human performance (β0.48) appears slightly higher than GPT-3 performance (β0.38), which is an exception to the overall trend.
### Interpretation
The data suggests that GPT-3 possesses a robust capability for solving structured reasoning tasks, outperforming human participants in this specific experimental setup. The consistent decline in accuracy with increasing complexity for both groups validates the experimental design's difficulty scaling. The particularly large gap in "Logic" problems indicates that GPT-3 may have a stronger grasp of abstract, formal logical structures compared to the tested human cohort. The crossover in subplot d (3 unique rules) is a critical outlier, suggesting a potential specific weakness in GPT-3's generative reasoning when faced with a high number of concurrent, unique rules, where human reasoning, while still challenged, may employ different strategies that are slightly more effective in that narrow case. Overall, the charts demonstrate GPT-3's advanced but not infallible reasoning abilities, with performance sensitive to problem structure and complexity.
</details>
were very similar for the Digit Matrices vs. the original image-based SPM problem set, and showed a similar pattern across problem types (Figure 4). These results suggest that, while the Digit Matrices do not engage the visual processes involved in traditional SPM problems (i.e., deriving disentangled representations from pixel-level inputs), they likely engage a similar set of core reasoning processes (i.e., inducing abstract rules from those representations). More generally, performance on verbal, visuospatial, and mathematical analogy problems are known to be highly correlated for people. 5 Accordingly, GPT-3's success on the Digit Matrices can be taken as evidence that it has acquired core capabilities underlying analogy, though it will be important in future work to investigate how these reasoning processes might be integrated with visual processing.
## 2.2 Letter string analogies
A central feature of human analogical reasoning is its flexibility. Human reasoners are capable of identifying abstract similarities between situations even when these situations are superficially quite different. Often this involves a process of re-representation , in which an initial problem representation is revised so as to facilitate the discovery of
Figure 4: Human performance for Digit Matrices vs. Raven's Standard Progressive Matrices (SPM). SPM 15 does not contain three-rule problems, but performance was very similar across one-rule, two-rule, and logic problems. SPM results reflect average performance for N=80 participants (data from 32 ). Digit Matrices results reflect average performance for N=40 participants. Error bars represent standard error of the mean. Each dot represents accuracy for a single participant.
<details>
<summary>Image 4 Details</summary>
### Visual Description
## Bar Chart: Multiple-Choice Accuracy by Problem Type for Raven's and Digit Matrices
### Overview
The image is a grouped bar chart comparing the multiple-choice accuracy of two types of matrix reasoning testsβRaven's Standard Progressive Matrices and Digit Matricesβacross four categories of problem complexity. The chart includes error bars for each data point.
### Components/Axes
* **Chart Type:** Grouped bar chart with error bars.
* **Y-Axis (Vertical):**
* **Label:** "Multiple-choice accuracy"
* **Scale:** Linear, ranging from 0 to 1, with major tick marks at 0, 0.2, 0.4, 0.6, 0.8, and 1.
* **X-Axis (Horizontal):**
* **Label:** "Problem type"
* **Categories (from left to right):** "1-rule", "2-rule", "3-rule", "Logic".
* **Legend:**
* **Position:** Top-right corner of the chart area.
* **Series 1:** "Raven's Standard Progressive Matrices" represented by a dark red (maroon) bar.
* **Series 2:** "Digit Matrices" represented by a light blue (cyan) bar.
* **Data Representation:** For each problem type category on the x-axis, two bars are placed side-by-side: the left bar (red) for Raven's Matrices and the right bar (light blue) for Digit Matrices. Each bar has a vertical black error bar extending above and below its top.
### Detailed Analysis
**Trend Verification:** For both data series (Raven's and Digit Matrices), the visual trend is a clear downward slope in accuracy as the problem type progresses from "1-rule" to "Logic". Accuracy is highest for the simplest problems and lowest for the most complex.
**Data Points (Approximate Values):**
* **1-rule:**
* Raven's (Red): Accuracy β 0.90. Error bar extends from β0.85 to β0.95.
* Digit Matrices (Light Blue): Accuracy β 0.88. Error bar extends from β0.65 to β1.0 (note: the upper bound appears to be capped at the chart maximum of 1).
* **2-rule:**
* Raven's (Red): Accuracy β 0.72. Error bar extends from β0.68 to β0.76.
* Digit Matrices (Light Blue): Accuracy β 0.73. Error bar extends from β0.65 to β0.81.
* **3-rule:**
* Raven's (Red): Accuracy β 0.65. Error bar extends from β0.60 to β0.70.
* Digit Matrices (Light Blue): Accuracy β 0.65. Error bar extends from β0.55 to β0.75.
* **Logic:**
* Raven's (Red): Accuracy β 0.53. Error bar extends from β0.50 to β0.56.
* Digit Matrices (Light Blue): Accuracy β 0.53. Error bar extends from β0.48 to β0.58.
### Key Observations
1. **Performance Parity:** The accuracy levels for Raven's Standard Progressive Matrices and Digit Matrices are remarkably similar across all four problem types. The difference in bar height for any given category is minimal.
2. **Monotonic Decrease:** There is a consistent, stepwise decrease in accuracy for both test types as the problem complexity increases from "1-rule" to "Logic". The drop is most pronounced between "1-rule" and "2-rule".
3. **Error Bar Variability:** The size of the error bars (indicating variability or uncertainty in the measurement) differs between the two test types. Notably, for "1-rule" problems, the Digit Matrices error bar is very large, suggesting high variability in performance on this specific task. For "Logic" problems, the error bars for both are relatively small.
4. **Ceiling Effect:** The "1-rule" accuracy for both tests is near the top of the scale (β0.9), indicating these problems are relatively easy for the test-takers, with little room for improvement.
### Interpretation
This chart demonstrates that the format of the matrix reasoning problemβwhether using traditional abstract shapes (Raven's) or numerical digitsβhas a negligible impact on measured accuracy across a range of rule complexities. The primary factor influencing performance is the inherent difficulty of the problem type itself.
The data suggests a strong inverse relationship between problem complexity (number of rules/logical steps required) and solution accuracy. The near-identical performance curves imply that the cognitive processes assessed by both test formats are highly similar, or that the digit-based adaptation is a valid and equivalent substitute for the traditional Raven's matrices in measuring this specific aspect of fluid intelligence.
The large error bar for Digit Matrices on "1-rule" problems is an interesting anomaly. It could indicate that while the average performance is high, some individuals found the digit format unexpectedly challenging or easy for the simplest problems, leading to a wider spread of scores. This warrants further investigation into individual differences in processing numerical versus abstract visual stimuli.
</details>
an analogy. 36-38
Hofstadter and Mitchell 16,39 introduced the letter string analogy domain to evaluate computational models of analogical reasoning, with a particular emphasis on the process of re-representation. The basic problem structure is illustrated in Figure 5a. In this example, the source string 'a b c d' has been transformed by converting the final letter to its successor, resulting in the string 'a b c e'. This transformation must be identified, and then applied to the target string 'i j k l', yielding the answer 'i j k m'.
Though this example is simple, letter string problems can be made quite complex by introducing various generalizations between the source and target strings. For instance, the target may involve groups of letters rather than individual letters (e.g., 'i i j j k k l l'), or may involve a sequence with a reversed order relative to the source (e.g., 'l k j i'). In these cases, the transformation identified in the source (e.g., a successor transformation applied to the final letter in the sequence) must be generalized to an analogous transformation (e.g., a successor transformation applied to the final group of letters , or a predecessor transformation applied to the first letter). This feature makes letter string analogy problems well-suited to test the capacity for re-representation.
To evaluate GPT-3, we created a novel letter string problem set (Figure 5), and carried out a systematic comparison with human participants (N=57, UCLA undergraduates). The problem set involved a range of different transformation (Figure 5d) and generalization types (Figure 5e). Each transformation type could be combined with any generalization type, and multiple generalization types could be combined together to yield more challenging problems (Figure 5b). Problems were presented to GPT-3 along with a prompt ('Let's try to complete the pattern:'), using a format similar to the Digit Matrices.
Figure 6 shows the results of this evaluation. GPT-3 showed stronger overall performance than human participants (Figure 6a; logistic regression, main effect of GPT-3 vs. human participants: OR = 1 . 76, p = 6 . 3 Γ 10 -5 , CI = [1 . 34 , 2 . 31]), an effect that was driven primarily by stronger performance on zero-generalization problems (main effect of GPT-3 vs. human participants for zero-generalization problems: OR = 1 . 76, p = 0 . 0007, CI = [1 . 27 , 2 . 46]). Performance was strongly affected by the number of generalizations in both GPT-3 and human participants (main effect of number of generalizations, GPT-3: OR = 0 . 51, p = 2 Γ 10 -16 , CI = [0 . 45 , 0 . 57]; human participants: OR = 0 . 66, p = 5 . 9 Γ 10 -16 , CI = [0 . 6 , 0 . 73]). GPT-3 and human participants also showed similar error patterns across transformation types (Figure 6b) and generalization types (Figure 6c), as quantified by a correlation analysis for accuracy across different problem subtypes ( r (39) = 0 . 7, p = 3 . 6 Γ 10 -7 ).
We also investigated a novel variant on letter string problems involving generalization from letters to real-world concepts (Figure 5c). GPT-3 showed strong performance on these problems, though with some discrepancies for different transformation types (Figure 6d). These results suggest that GPT-3 has developed an abstract notion of successorship that can be flexibly generalized between different domains (e.g., alphabetic successorship vs. temper-
Figure 5: Letter string analogy problems. Transformation between source strings must be identified and applied to target string. Mapping between source and target may involve one or more generalizations. (a) Easy problem involving zero generalizations. (b) Difficult problem involving three generalizations (grouping, reversed order, and interleaved distractors). (c) Problem involving generalization from letters to real-world concepts. (d) Transformations were sampled from set of six possible types: sequence extension, successor transformation (applied to the last letter in the string), predecessor transformation (applied to the first letter in the string), removal of a redundant letter, 'fixing' an alphabetic sequence (replacing an out-of-place letter), and sorting. (e) Generalizations were sampled from set of six possible types: letter-to-number, grouping, longer target string, reversed order, interleaved distractors, and larger interval.
<details>
<summary>Image 5 Details</summary>
### Visual Description
## Diagram: Sequence Transformation and Generalization Examples
### Overview
The image is a technical diagram illustrating various types of abstract sequence transformations and generalizations. It is divided into five labeled sections (a, b, c, d, e), each presenting examples of pattern-based reasoning tasks. The diagram uses simple alphabetic and word sequences to demonstrate how an initial sequence is transformed into a target sequence, with some examples posing a question (?) for the viewer to solve.
### Components/Axes
The diagram is organized into five distinct sections, each with a bold lowercase letter label in the top-left corner of its region:
- **Section a**: Top-left region.
- **Section b**: Top-center region.
- **Section c**: Top-right region.
- **Section d**: Middle region, titled "Transformation types".
- **Section e**: Bottom region, titled "Generalization types".
Each section contains one or more examples. Each example consists of:
1. An initial sequence (e.g., `a b c d`).
2. An arrow (`β`) indicating a transformation.
3. A target sequence (e.g., `a b c e`) or a question mark (`?`) indicating the problem to be solved.
### Detailed Analysis
#### Section a
- **Example 1**: `a b c d β a b c e`
- **Transformation**: The fourth element changes from `d` to `e`. This appears to be a "Successor" transformation (changing the last element to its alphabetical successor).
- **Example 2**: `i j k l β ?`
- **Transformation**: Applying the same logic as Example 1, the target would be `i j k m`.
#### Section b
- **Example 1**: `a b c d β a b c e` (Same as Section a, Example 1).
- **Example 2**: `x l x l x k x k x j x j x i x i β ?`
- **Transformation**: This is a complex, interleaved sequence. The pattern appears to be pairs of letters (`x l`, `x k`, `x j`, `x i`) where the second letter in each pair is decreasing alphabetically (`l, k, j, i`). The next pair would logically be `x h`. Therefore, the full target sequence would be `x l x l x k x k x j x j x i x i x h x h`.
#### Section c
- **Example 1**: `a b c β a b c`
- **Transformation**: The sequence remains unchanged. This could represent an "Identity" transformation or a case where no change is required.
- **Example 2**: `cold cool warm β ?`
- **Transformation**: This uses words instead of letters. The pattern is less clear. It may involve semantic relationships (e.g., temperature progression: cold β cool β warm) or a categorical shift. A possible answer could be `hot`, continuing the temperature trend.
#### Section d: Transformation types
This section explicitly names and illustrates six types of sequence transformations.
1. **Extend sequence**: `a b c d β a b c d e` (Adds the next element in sequence).
2. **Successor**: `a b c d β a b c e` (Changes the last element to its successor).
3. **Predecessor**: `b c d e β a c d e` (Changes the first element to its predecessor).
4. **Remove redundant letter**: `a b b c d e β a b c d e` (Removes a duplicate `b`).
5. **Fix alphabetic sequence**: `a b c w e β a b c d e` (Replaces `w` with the correct letter `d` to maintain alphabetical order).
6. **Sort**: `a d c b e β a b c d e` (Rearranges the elements into alphabetical order).
#### Section e: Generalization types
This section names and illustrates six types of generalizations, which apply the core transformation logic to new contexts.
1. **Letter-to-number**: `a b c d β a b c e` / `1 2 3 4 β ?` (Applies the "Successor" rule to numbers. Target: `1 2 3 5`).
2. **Grouping**: `a b c d β a b c e` / `i i j j k k l l β ?` (Applies the rule to grouped pairs. Target: `i i j j k k l l m m`).
3. **Longer target**: `a b c d β a b c e` / `i j k l m n o p β ?` (Applies the rule to a longer sequence. Target: `i j k l m n o q`).
4. **Reversed order**: `a b c d β a b c e` / `l k j i β ?` (Applies the rule to a reversed sequence. Target: `l k j h`).
5. **Interleaved distractor**: `a b c d β a b c e` / `i x j x k x l x β ?` (Applies the rule to a sequence with an interleaved distractor `x`. Target: `i x j x k x l x m x`).
6. **Larger interval**: `a b c d β a b c e` / `i k m o β ?` (Applies the rule to a sequence with a larger step interval. Target: `i k m q`).
### Key Observations
- The diagram progresses from simple, implicit examples (a, b, c) to explicitly categorized types (d, e).
- The core transformation in most examples is the "Successor" rule (changing the last element to the next in sequence), which is then generalized in various ways.
- Section `b` contains the most complex pattern, requiring the solver to identify an interleaved structure and a decreasing subsequence.
- Section `c` introduces a semantic/word-based example, suggesting the rules can apply beyond simple alphabetic sequences.
- The "Generalization types" in Section `e` demonstrate how a single abstract rule (like "Successor") can be tested across different domains (numbers, grouped items, reversed order, etc.).
### Interpretation
This diagram serves as a taxonomy or test bank for abstract reasoning tasks, likely used in cognitive science, artificial intelligence evaluation, or educational psychology. It deconstructs the process of pattern recognition into fundamental transformation operations (like successor, sort, remove redundancy) and then shows how these operations can be generalized to novel situations.
The underlying message is that intelligence involves not just recognizing a pattern in a specific instance, but also extracting the abstract rule governing that pattern and applying it flexibly to new contexts. The progression from `a` to `e` mirrors a learning or testing sequence: first encounter examples, then learn the named categories, and finally see how those categories apply to increasingly complex or varied problems. The presence of the question marks (`?`) frames the entire diagram as a series of problems to be solved, emphasizing its function as a tool for assessment or demonstration of reasoning capabilities.
</details>
ature successorship).
One important caveat is that GPT-3's performance on this task was somewhat sensitive to the way in which problems were formatted. For instance, performance suffered when no prompt was provided (Supplementary Figure 5a), or when problems were presented in the form of a complete sentence (Supplementary Figure 5b). However, even in these cases, GPT-3's zero-shot performance was both within the range of human participants (within one standard deviation), and closely matched the pattern of human performance across problem types (correlation analysis, no prompt: r (39) = 0 . 6, p = 5 . 3 Γ 10 -5 , sentence format: r (39) = 0 . 76, p = 4 . 2 Γ 10 -6 ).
## 2.3 Four-term verbal analogies
Though matrix reasoning and letter string analogies involve a high degree of relational complexity, one limitation is that they consist of highly constrained, synthetic relations, such as alphabetic or numerical successorship. GPT-3's ability to solve problems involving more real-world concepts (e.g., 'a b c β a b d, cold cool warm β ?') suggests that its analogical capabilities may not be limited to such artificial settings. To further evaluate GPT-3's capacity to reason about real-world relational concepts, we tested it on four-term verbal analogy problems involving a broader range of semantic relations.
Figure 6: Letter string analogy results. GPT-3 displayed strong performance on letter string problems, and showed a similar pattern to human participants across conditions. (a) GPT-3 and human performance as a function of the number of generalizations between source and target. (b) Performance on zero-generalization problems as a function of transformation type. (c) Performance on one-generalization problems as a function of generalization type. (d) Performance on problems requiring generalization from letters to real-world concepts. Human results reflect average performance for N=57 participants (UCLA undergraduates). Black error bars represent standard error of the mean across participants. Each dot represents accuracy for a single participant. Note that (b-d) do not show individual participant results because each participant only completed one problem in each condition. Gray error bars represent 95% binomial confidence intervals for average performance across multiple problems.
<details>
<summary>Image 6 Details</summary>
### Visual Description
## Bar Charts: Comparative Generative Accuracy of GPT-3 vs. Humans
### Overview
The image contains four bar charts (labeled a, b, c, d) comparing the generative accuracy of GPT-3 (dark purple bars) and Human participants (light blue bars) across different problem types and generalization levels. All charts share a common y-axis: "Generative accuracy" ranging from 0 to 1. Error bars are present on all data points, indicating variability or confidence intervals.
### Components/Axes
* **Common Y-Axis (All Charts):** "Generative accuracy" (scale: 0, 0.2, 0.4, 0.6, 0.8, 1).
* **Legend (Top-Right of each chart):** Dark purple square = "GPT-3"; Light blue square = "Human".
* **Chart-Specific Titles & X-Axes:**
* **Chart a:** Title: None (implied overall trend). X-axis: "Number of generalizations" (Categories: 0, 1, 2, 3).
* **Chart b:** Title: "Zero-generalization problems". X-axis: "Transformation type" (Categories: Extend sequence, Successor, Predecessor, Remove redundant letter, Fix alphabetic sequence, Sort).
* **Chart c:** Title: "One-generalization problems". X-axis: "Generalization type" (Categories: Letter-to-number, Grouping, Longer target, Reverse order, Interleaved distractor, Larger interval).
* **Chart d:** Title: "Real-world concept problems". X-axis: "Transformation type" (Categories: Extend sequence, Successor, Predecessor, Sort).
### Detailed Analysis
**Chart a: Accuracy vs. Number of Generalizations**
* **Trend:** Generative accuracy for both GPT-3 and Humans decreases as the number of required generalizations increases from 0 to 3.
* **Data Points (Approximate):**
| Number of Generalizations | GPT-3 Accuracy | Human Accuracy |
| :--- | :--- | :--- |
| 0 | β 0.71 | β 0.58 |
| 1 | β 0.50 | β 0.42 |
| 2 | β 0.36 | β 0.35 |
| 3 | β 0.24 | β 0.29 |
**Chart b: Zero-Generalization Problems by Transformation Type**
* **Trend:** GPT-3 outperforms Humans in most transformation types, with the largest gap in "Extend sequence" and "Successor". Performance is similar for "Sort".
* **Data Points (Approximate):**
| Transformation Type | GPT-3 Accuracy | Human Accuracy |
| :--- | :--- | :--- |
| Extend sequence | β 0.96 | β 0.84 |
| Successor | β 0.94 | β 0.79 |
| Predecessor | β 0.78 | β 0.72 |
| Remove redundant letter | β 0.86 | β 0.68 |
| Fix alphabetic sequence | β 0.52 | β 0.24 |
| Sort | β 0.22 | β 0.23 |
**Chart c: One-Generalization Problems by Generalization Type**
* **Trend:** Performance is more varied. GPT-3 leads significantly in "Letter-to-number" and "Reverse order". Humans perform slightly better in "Grouping" and "Larger interval". The "Interleaved distractor" task shows low accuracy for both.
* **Data Points (Approximate):**
| Generalization Type | GPT-3 Accuracy | Human Accuracy |
| :--- | :--- | :--- |
| Letter-to-number | β 0.82 | β 0.63 |
| Grouping | β 0.54 | β 0.56 |
| Longer target | β 0.52 | β 0.48 |
| Reverse order | β 0.56 | β 0.32 |
| Interleaved distractor | β 0.32 | β 0.31 |
| Larger interval | β 0.26 | β 0.23 |
**Chart d: Real-World Concept Problems by Transformation Type**
* **Trend:** GPT-3 shows a perfect score (1.0) on "Extend sequence" and generally outperforms Humans, except for "Predecessor" where Humans are slightly better.
* **Data Points (Approximate):**
| Transformation Type | GPT-3 Accuracy | Human Accuracy |
| :--- | :--- | :--- |
| Extend sequence | = 1.0 | β 0.67 |
| Successor | β 0.76 | β 0.73 |
| Predecessor | β 0.52 | β 0.74 |
| Sort | β 0.12 | β 0.18 |
### Key Observations
1. **Generalization Cost:** Chart (a) clearly demonstrates that requiring more generalizations (0 to 3) significantly reduces accuracy for both AI and humans.
2. **GPT-3's Strengths:** GPT-3 excels in tasks involving sequence extension and successor identification, particularly in zero-generalization (b) and real-world concept (d) contexts.
3. **Human Strengths:** Humans show a relative advantage in the "Predecessor" task within real-world concepts (d) and in "Grouping" and "Larger interval" within one-generalization problems (c).
4. **Shared Difficulty:** Both entities struggle with the "Sort" transformation (b, d) and the "Interleaved distractor" generalization (c), indicating these are inherently challenging tasks.
5. **Performance Gap:** The performance gap between GPT-3 and Humans is most pronounced in tasks that appear more rule-based or sequential (e.g., Extend sequence, Successor).
### Interpretation
The data suggests a nuanced comparison between GPT-3 and human cognitive performance on structured, generative tasks. GPT-3 demonstrates superior performance in tasks that likely rely on pattern completion and applying straightforward sequential rules (e.g., extending a sequence, finding the next item). This aligns with the model's training on vast textual data where such patterns are prevalent.
However, humans show resilience or superiority in tasks that may require different cognitive strategies, such as grouping based on abstract criteria ("Grouping"), working with larger numerical intervals ("Larger interval"), or understanding concepts in reverse ("Predecessor" in real-world contexts). The steep decline in accuracy with increased generalizations (Chart a) highlights a fundamental challenge for both biological and artificial intelligence: moving from specific examples to broader, abstract application.
The near-equal performance on the "Sort" task is notable, suggesting it may be a task where human intuitive heuristics and GPT-3's learned patterns converge on similarly limited effectiveness. Overall, the charts illustrate that while GPT-3 can match or exceed human performance on specific, well-defined generative problems, its advantages are task-dependent, and both systems face common limitations when tasks require higher-order generalization or involve certain types of cognitive operations.
</details>
We evaluated GPT-3 on four separate datasets. 17,19-21 To the best of our knowledge, these constitute an exhaustive set of four-term verbal analogy problems for which human behavioral data is available. 41 Each dataset contains a series of four-term analogy problems in the form 'A:B::C:?', together with a set of answer choices (i.e., potential choices of D). For each problem, GPT-3 was evaluated by presenting the problem together with each potential answer choice, and selecting the option for which GPT-3 assigned a higher log probability. The problem and GPT-3's choice
Figure 7: Verbal analogy results. (a) Results for UCLA Verbal Analogy Test (VAT). 17 Human results reflect average performance for N=57 participants. Black error bars represent standard error of the mean. Each dot represents accuracy for a single participant. (b) Results for dataset from Sternberg and Nigro. 19 Human results reflect average performance for N=20 participants. (c) Results for SAT analogy problems from Turney et al. 20 These problems involve five answer choices, and thus chance performance is 20%. Human results reflect an estimate of the average performance for high school students taking the SAT (see 40 for details). (d) Results for dataset from Jones et al. 21 Human results reflect average performance for N=241 participants. Gray error bars represent 95% binomial confidence intervals for average performance across multiple problems. Gray horizontal lines represent chance performance.
<details>
<summary>Image 7 Details</summary>
### Visual Description
## Grouped Bar Charts: GPT-3 vs. Human Performance on Cognitive Tasks
### Overview
The image contains four subplots (labeled a, b, c, d) displaying grouped bar charts that compare the accuracy of GPT-3 (dark purple bars) and Human participants (light blue bars) across various cognitive and linguistic tasks. Each chart includes a horizontal line at 0.5 accuracy, likely representing chance performance. Error bars are present on all data points, indicating variability or confidence intervals.
### Components/Axes
* **Common Elements:**
* **Y-axis (All Charts):** Labeled "Accuracy," scaled from 0 to 1 with increments of 0.2.
* **Legend:** Located in the top-right corner of subplot **b**. It defines the two data series: "GPT-3" (dark purple) and "Human" (light blue).
* **Chance Line:** A solid horizontal line at y=0.5 across all subplots.
* **Subplot-Specific Elements:**
* **a. UCLA VAT:**
* **X-axis Categories:** "Categorical", "Function", "Antonym", "Synonym".
* **b. Sternberg & Nigro (1980):**
* **X-axis Categories:** "Categorical", "Function", "Antonym", "Synonym", "Linear".
* **c. SAT:**
* **X-axis:** No category labels; presents a single pair of bars.
* **d. Jones et al. (2022):**
* **Left Chart X-axis:** "Relation category" with categories "Categorical", "Compositional", "Causal".
* **Right Chart X-axis:** "Semantic distance" with categories "Near", "Far".
### Detailed Analysis
**Subplot a: UCLA VAT**
* **Trend:** GPT-3 accuracy is consistently higher than Human accuracy across all four categories.
* **Data Points (Approximate):**
* Categorical: GPT-3 β 1.0, Human β 0.85
* Function: GPT-3 β 0.85, Human β 0.80
* Antonym: GPT-3 β 0.90, Human β 0.85
* Synonym: GPT-3 β 0.95, Human β 0.85
**Subplot b: Sternberg & Nigro (1980)**
* **Trend:** GPT-3 maintains near-perfect accuracy across all categories. Human performance is high but shows a slight dip for "Synonym" and "Linear" tasks.
* **Data Points (Approximate):**
* Categorical: GPT-3 β 0.98, Human β 0.92
* Function: GPT-3 β 0.98, Human β 0.96
* Antonym: GPT-3 β 1.0, Human β 0.95
* Synonym: GPT-3 β 1.0, Human β 0.85
* Linear: GPT-3 β 1.0, Human β 0.88
**Subplot c: SAT**
* **Trend:** GPT-3 significantly outperforms Human participants on this task.
* **Data Points (Approximate):**
* GPT-3 β 0.75
* Human β 0.55
**Subplot d: Jones et al. (2022)**
* **Left Chart (Relation category):**
* **Trend:** Performance is closely matched between GPT-3 and Humans for "Categorical" and "Causal" relations. Humans show a slight advantage in "Compositional" relations.
* **Data Points (Approximate):**
* Categorical: GPT-3 β 0.88, Human β 0.92
* Compositional: GPT-3 β 0.72, Human β 0.82
* Causal: GPT-3 β 0.78, Human β 0.78
* **Right Chart (Semantic distance):**
* **Trend:** For "Near" semantic distance, performance is nearly identical. For "Far" semantic distance, Human accuracy is notably higher than GPT-3's.
* **Data Points (Approximate):**
* Near: GPT-3 β 0.88, Human β 0.88
* Far: GPT-3 β 0.70, Human β 0.80
### Key Observations
1. **GPT-3 Dominance:** GPT-3 achieves perfect or near-perfect accuracy (β1.0) in the Sternberg & Nigro (1980) tasks and the UCLA VAT "Categorical" task.
2. **Human Performance Dip:** Human accuracy shows notable decreases in specific tasks: "Synonym" in the Sternberg & Nigro dataset, the SAT task overall, and "Compositional" relations in the Jones et al. data.
3. **Semantic Distance Effect:** The most significant performance gap favoring humans appears in the "Far" semantic distance condition (Jones et al., right chart), suggesting a potential limitation in GPT-3's handling of distant semantic relationships.
4. **Consistency Above Chance:** All recorded accuracies for both GPT-3 and Humans are well above the 0.5 chance line, indicating task engagement and non-random performance.
### Interpretation
The data suggests that GPT-3, a large language model, matches or exceeds human-level performance on a range of structured linguistic and cognitive tasks, particularly those involving categorical judgments and well-defined relations (e.g., antonyms, synonyms in some contexts). Its performance is remarkably consistent and high across the Sternberg & Nigro battery.
However, the model shows relative weaknesses in two areas: 1) tasks requiring compositional reasoning (Jones et al.), and 2) processing semantic relationships that are "far" or less conventional. The SAT result, where GPT-3 outperforms humans, is interesting and may reflect the model's extensive training on text similar to standardized test materials.
The charts collectively illustrate a nuanced picture of AI capability: excelling at pattern recognition and tasks aligned with its training data distribution, while potentially struggling with deeper compositional semantics and novel or distant conceptual links where human intuition or different cognitive mechanisms may have an advantage. The consistent error bars also indicate that while mean performance is high, there is variability in both human and model responses.
</details>
were then appended to the context window for the next problem, thereby simulating any contextual effects that might arise when solving multiple problems in a row, as human participants typically do.
Figure 7 shows the results for all datasets. GPT-3 performed as well or better than human participants (minimum education level of high-school graduation, located in the United States and recruited using Amazon Mechanical Turk) on the UCLA Verbal Analogy Test (VAT), 17 involving categorical, functional, antonym, and synonym relations (Figure 7a), and on a dataset from Sternberg and Nigro 19 involving these same four relation types and linear order relations (Figure 7b). On a dataset of SAT analogy problems from Turney et al., 20 GPT-3 surpassed the estimated average level of performance for high school students taking the SAT (Figure 7c). GPT-3 also showed performance in the same range as human participants (though numerically weaker) on a problem set from Jones et al. 21 involving categorical, compositional, and causal relations (Figure 7d).
In addition to displaying generally strong performance on these problem sets, GPT-3 also displayed sensitivity
to semantic content similar to that observed in human participants. In the dataset from Jones et al. 21 (Figure 7d), participants performed worse on problems in which the analogs were semantically distant (i.e., the A and B terms had low semantic similarity to C and D), an effect that was also displayed by GPT-3 (logistic regression, effect of semantic distance for GPT-3: OR = 3 . 24, p = 0 . 0165, CI = [1 . 24 , 8 . 5]). These results align with a more general phenomenon in which human reasoning is facilitated by semantically meaningful or coherent content. 24,42
Figure 8: Story analogy results. Results for identification of analogies between stories, using materials from Gentner et al. 18 When presented with a source story and two target stories, both GPT-3 and human participants showed a preference for target stories that shared higher-order relations with the source vs. those that only shared first-order relations. Near analogy condition involves within-domain comparison between stories with similar entities. Far analogy condition involves cross-domain comparison between stories with different entities. Human results reflect average performance for N=54 participants (UCLA undergraduates). Black error bars represent standard error of the mean across participants. Each dot represents accuracy for a single participant. Gray error bars represent 95% binomial confidence intervals for average performance across multiple problems. Gray horizontal line represents chance performance.
<details>
<summary>Image 8 Details</summary>
### Visual Description
## Grouped Bar Chart: Analogy Accuracy Comparison (GPT-3 vs. Human)
### Overview
The image is a grouped bar chart comparing the accuracy of two entitiesβGPT-3 and Humanβon two types of analogy tasks: "Near analogy" and "Far analogy." The chart includes error bars for each data point and a horizontal reference line at the 0.5 accuracy level.
### Components/Axes
* **Chart Type:** Grouped bar chart with error bars.
* **Y-Axis:**
* **Label:** "Accuracy"
* **Scale:** Linear, ranging from 0 to 1, with major tick marks at 0, 0.2, 0.4, 0.6, 0.8, and 1.
* **X-Axis:**
* **Categories:** Two primary categories: "Near analogy" (left group) and "Far analogy" (right group).
* **Legend:**
* **Position:** Top-right corner of the chart area.
* **Labels & Colors:**
* **GPT-3:** Represented by dark purple bars.
* **Human:** Represented by light blue bars.
* **Reference Line:** A solid horizontal line at y = 0.5, spanning the width of the chart.
* **Error Bars:** Vertical lines extending above and below the top of each bar, indicating variability or confidence intervals.
### Detailed Analysis
**1. Near Analogy Category (Left Group):**
* **GPT-3 (Dark Purple Bar):**
* **Visual Trend:** The bar is shorter than the adjacent Human bar.
* **Approximate Value:** The top of the bar aligns with an accuracy of approximately **0.75**.
* **Error Bar:** Extends from approximately **0.60** to **0.90**.
* **Human (Light Blue Bar):**
* **Visual Trend:** The bar is taller than the adjacent GPT-3 bar.
* **Approximate Value:** The top of the bar aligns with an accuracy of approximately **0.90**.
* **Error Bar:** Extends from approximately **0.85** to **0.95**.
**2. Far Analogy Category (Right Group):**
* **GPT-3 (Dark Purple Bar):**
* **Visual Trend:** The bar is shorter than the adjacent Human bar and also shorter than the GPT-3 bar in the "Near analogy" category.
* **Approximate Value:** The top of the bar aligns with an accuracy of approximately **0.65**.
* **Error Bar:** Extends from approximately **0.50** to **0.80**.
* **Human (Light Blue Bar):**
* **Visual Trend:** The bar is taller than the adjacent GPT-3 bar but slightly shorter than the Human bar in the "Near analogy" category.
* **Approximate Value:** The top of the bar aligns with an accuracy of approximately **0.85**.
* **Error Bar:** Extends from approximately **0.78** to **0.92**.
### Key Observations
1. **Performance Gap:** Human accuracy is higher than GPT-3 accuracy in both analogy categories.
2. **Task Difficulty:** Accuracy for both entities is lower on "Far analogy" tasks compared to "Near analogy" tasks.
3. **Relative Decline:** The performance drop from "Near" to "Far" analogy is more pronounced for GPT-3 (from ~0.75 to ~0.65, a drop of ~0.10) than for Humans (from ~0.90 to ~0.85, a drop of ~0.05).
4. **Variability:** The error bars for GPT-3 are noticeably larger than those for Humans in both categories, suggesting greater variability or less consistency in GPT-3's performance.
5. **Baseline Comparison:** All four data points (both bars for both entities) are above the 0.5 reference line, indicating performance better than chance for all conditions.
### Interpretation
The data suggests a clear performance advantage for humans over GPT-3 in solving analogies, particularly for "far" analogies which likely require more abstract or distant relational reasoning. The smaller performance drop for humans indicates they are more robust to the increased difficulty of far analogies. The larger error bars for GPT-3 imply its performance is less stable or predictable across different analogy instances. The chart effectively communicates that while GPT-3 performs above chance, there remains a significant gap between its analogical reasoning capabilities and human-level performance, especially for more challenging, abstract tasks. The 0.5 line serves as a crucial benchmark, confirming that both entities are performing meaningfully above random guessing.
</details>
## 2.4 Story analogies
Human reasoners are able not only to form analogies between individual concepts, but can also identify correspondences between complex real-world events, involving many entities and relations. When making such comparisons, human reasoning is especially sensitive to higher-order relations - relations between relations - notably causal relations between events. Such higher-order relations play a central role in some cognitive theories of analogy, 43 and it is thus important to establish whether GPT-3 displays a similar sensitivity to them.
To address this question, we tested GPT-3 on a set of story analogies from Gentner et al. 18 In each set, a source story is compared to two potential target stories, each of which is matched with the source story in terms of first-order relations, but only one of which shares the same causal relations as the source (see Methods Section 4.6.1 for examples). Gentner et al. found that human participants rated the target stories as more similar when they shared the same causal relations as the source story. These problems are further defined by two different comparison conditions. In the near analogy condition (referred to as 'literal similarity' vs. 'mere appearance' by Gentner et al.), the target stories also share the same basic entities as the source story, making for a less abstract, within-domain comparison. In the far analogy condition (referred to as 'true analogy' vs. 'false analogy' by Gentner et al.), the target stories involve different entities from the source story, but share first-order relations, resulting in a more challenging, cross-domain comparison.
To facilitate a direct comparison with GPT-3, we performed a new behavioral study with these materials. For each source story, participants indicated which of two target stories was more analogous. Both GPT-3 and human
participants (N=54, UCLA undergraduates) showed a sensitivity to higher-order relations (Figure 8), most often selecting the target story that shared causal relations with the source (combined near and far analogy; GPT-3, binomial test: p = 0 . 0005; human participants, one-sample t-test: t (53) = 21 . 3, p = 1 . 1 Γ 10 -27 ; null hypotheses for both tests is chance-level performance of 0.5). This effect was significant for both GPT-3 and human participants in the near analogy condition (GPT-3, binomial test: p = 0 . 0039; human participants, one-sample t-test: t (53) = 21 . 5, p = 8 . 5 Γ 10 -28 ), but only human participants showed a significant effect in the far analogy condition (GPT-3, binomial test: p = 0 . 065; human participants, one-sample t-test: t (53) = 16 . 7, p = 9 . 3 Γ 10 -23 ).
Unlike the other task domains considered in the present work, this was a case in which college students clearly outperformed GPT-3 (logistic regression, main effect of GPT-3 vs. human participants: OR = 0 . 37, p = 0 . 0003, CI = [0 . 21 , 0 . 63]). Indeed, a significant number of participants (15/54) selected the analogous story on every trial. However, in an initial investigation of GPT-4, 44 we found that it displays stronger performance on this task, more robustly picking the analogous story even in the far analogy condition, and displaying nearly perfect performance in the near analogy condition (Supplementary Figure 6, Section S4.3 ). It therefore seems likely that further scaling of large language models will enhance their sensitivity to causal relations.
## 2.5 Analogical problem-solving
In everyday thinking and reasoning, analogical comparisons are often made for the purpose of achieving some goal, or solving a novel problem. Thus far, our tests of GPT-3 have assessed its capacity for identifying analogies in textbased inputs with varying formats, but can GPT-3 also use these analogies to derive solutions to novel problems, as human reasoners do?
As a preliminary investigation of this issue, we performed a qualitative evaluation using a paradigm developed by Gick and Holyoak. 22 In that paradigm, participants are presented with a target problem in the form of a story. In the original study, Duncker's radiation problem was used. 45 In that problem, a doctor wants to use radiation to destroy a malignant tumor, but destroying the tumor with a single high-intensity ray will also damage the surrounding healthy tissue. The solution - to use several low-intensity rays that converge at the site of the tumor - is rarely identified spontaneously, but participants are more likely to discover this solution when they are first presented with an analogous source story. In the original study, the source story involved a general who wants to capture a fortress ruled by an evil dictator, but cannot do so by sending his entire army along a single road, which would trigger landmines. The general instead breaks his army up into small groups that approach the fortress from multiple directions, thus avoiding triggering the mines.
We first presented GPT-3 with the target problem in isolation. GPT-3 proposed a solution that involved injecting a radiation source directly into the tumor, rather than identifying the intended solution based on the convergence of multiple low-intensity radiation sources (Supplementary Section S5.1 ). However, when first presented with the general story, followed by the target problem, GPT-3 correctly identified the convergence solution (Supplementary Section S5.2 ). GPT-3 was further able to correctly explain the analogy, and to identify the specific correspondences between the source story and target problem when prompted (e.g., general β doctor, dictator β tumor, army β rays). We also found similar results when using distinct source analogs taken from another study 46 (Supplementary Section S5.3 ).
In a more challenging version of this paradigm, participants were first presented with both the general story, and two other non-analogous stories intended to serve as distractors. In this context, human participants were much less likely to identify the convergence solution. However, when given a prompt to explicitly consider the previously presented stories when trying to solve the radiation problem, participants were often able to correctly identify the analogous general story, and use this analogy to devise the convergence solution. Remarkably, we found that GPT-3 displayed these same effects. When presented with these same distracting, non-analogous stories, GPT-3 no longer identified the convergence solution, instead proposing the same solution that it proposed in response to the radiation problem alone (Supplementary Section S5.4 ). But when prompted to consider the previous stories, GPT-3 both correctly identified the general story as most relevant, and proposed the convergence solution (Supplementary Section S5.5 ).
We also evaluated GPT-3 using materials from a developmental study that employed a similar paradigm. 23 In that study, children were tasked with transferring gumballs from one bowl to another bowl that was out of reach, and provided with a number of materials for doing so (e.g., a posterboard, an aluminum walking cane, a cardboard tube), permitting multiple possible solutions. The key result was that when children were first presented with an analogous source story (about a magical genie trying to transfer jewels between two bottles), they were more likely
to identify a solution to the target problem that was analogous to the events described in the source story.
When presented with this target problem, GPT-3 mostly proposed elaborate, but mechanically nonsensical solutions, with many extraneous steps, and no clear mechanism by which the gumballs would be transferred between the two bowls (Supplementary Sections S5.6 -S5.8 ). However, when asked to explicitly identify an analogy between the source story and target problem, GPT-3 was able to identify all of the major correspondences, even though it could not use this analogy to discover an appropriate solution. This finding suggests that GPT-3's difficulty with this problem likely stems from its lack of physical reasoning skills, rather than being due to a difficulty with analogical mapping per se. It is also worth noting that in the original study, this task was presented to children with real physical objects, which likely aided the physical reasoning process relative to the purely text-based input provided to GPT-3. Overall, these results provide some evidence that GPT-3 is capable of using analogies for the purposes of problem-solving, but its ability to do so is constrained by the content about which it can reason, with particular difficulty in the domain of physical reasoning.
## 3 Discussion
We have presented an extensive evaluation of analogical reasoning in a state-of-the-art large language model. We found that GPT-3 appears to display an emergent ability to reason by analogy, matching or surpassing human performance across a wide range of text-based problem types. These included a novel problem set (Digit Matrices) modeled closely on Raven's Progressive Matrices, where GPT-3 both outperformed human participants, and captured a number of specific signatures of human behavior across problem types. Because we developed the Digit Matrix task specifically for this evaluation, we can be sure GPT-3 had never been exposed to problems of this type, and therefore was performing zero-shot reasoning. GPT-3 also displayed an ability to solve analogies based on more meaningful relations, including four-term verbal analogies and analogies between stories describing complex real-world events.
It is certainly not the case that GPT-3 mimics human analogical reasoning in all respects. Our tests were limited to processes that can be carried out within a local temporal context, but humans are also capable of retrieving potential source analogs from long-term memory, and ultimately of developing new concepts based on the comparison of multiple analogs. Unlike humans, GPT-3 does not have long-term memory for specific episodes. It is therefore unable to search for previously-encountered situations that might create useful analogies with a current problem. For example, GPT-3 can use the general story to guide its solution to the radiation problem, but as soon as its context buffer is emptied, it reverts to giving its non-analogical solution to the problem - the system has learned nothing from processing the analogy. GPT-3's reasoning ability is also limited by its lack of physical understanding of the world, as evidenced by its failure (in comparison with human children) to use an analogy to solve a transfer problem involving construction and use of simple tools. GPT-3's difficulty with this task is likely due at least in part to its purely text-based input, lacking the multimodal experience necessary to build a more integrated world model. 47 Finally, we found GPT-3 was limited in its ability to evaluate analogies based on causal relations, particularly in cross-domain comparisons between stories (far analogy).
But despite these major caveats, our evaluation reveals that GPT-3 exhibits a very general capacity to identify and generalize - in zero-shot fashion - relational patterns to be found within both formal problems and meaningful texts. These results are extremely surprising. It is commonly held that although neural networks can achieve a high level of performance within a narrowly-defined task domain, they cannot robustly generalize what they learn to new problems in the way that human learners do. 6, 48-50 Analogical reasoning is typically viewed as a quintessential example of this human capacity for abstraction and generalization, allowing human reasoners to intelligently approach novel problems zero-shot. Our results indicate that GPT-3 - unlike any other neural network previously tested on analogy problems - displays a capacity for such zero-shot analogical reasoning across a broad range of tasks.
The deep question that now arises is how GPT-3 achieves the analogical capacity that is often considered the core of human intelligence. One possibility is that, perhaps as a result of the sheer size and diversity of GPT-3's training data, it has been forced to develop mechanisms similar to those thought to underlie human analogical reasoning despite not being explicitly trained to do so. The consensus among cognitive scientists working on analogy is that this human ability depends on systematic comparison of knowledge based on explicit relational representations. It is unclear whether and how GPT-3 would implement these processes. Does GPT-3 possess some form of emergent relational representations, and if so, how are they computed? Does it perform a mapping process similar to the type that plays a central role in cognitive theories of analogy 43 ?
A few properties of the transformer architecture, 29 on which GPT-3 and other large language models are based,
are worth considering here. The first is the central role played by similarity . Transformers are built on a self-attention operation, which involves explicitly computing the similarity between each pair of vectors in the inputs to each layer. This pairwise evaluation of similarity is also a key feature of cognitive models of analogy, where it provides the primary constraint guiding the process of analogical mapping. In traditional symbolic models, 51 this takes the form of literal identicality between symbols, but in more recent models, 52,53 a graded similarity function that operates over vector-based inputs is used, much like the self-attention operation in transformers. Second, transformer self-attention employs a form of indirection , in which one set of embeddings is used to reference another set of embeddings (i.e., keys vs. values) - arguably a form of variable binding. Cognitive scientists have long hypothesized that variable binding plays a central role in analogical reasoning, and abstract reasoning more broadly, as it potentially allows generalization of abstract roles across different contexts. 48,54-58 It may be that these features of the transformer make it better equipped to perform zero-shot reasoning than other neural architectures. This possibility aligns with recent evidence that the transformer architecture is an important factor contributing toward the emergence of few-shot learning. 27
But although the mechanisms incorporated into large language models such as GPT-3 may have some important links to building blocks of human reasoning, we must also entertain the possibility that this type of machine intelligence is fundamentally different from the human variety. Humans have evolved to reason within bounds imposed by limited computational power and biological constraints . 59 Thus, we tend to approach complex problems by breaking them into a set of simpler problems that can be solved separately, 60 an approach that plays a particularly important role in solving challenging analogy problems such as Raven's Matrices. 61 It is possible that GPT-3, through sheer computational scale, is able to solve such complex problems in a holistic and massively parallel manner, without the need to segment them into more manageable components.
It must also be noted that, regardless of the extent to which GPT-3 employs human-like mechanisms to perform analogical reasoning, we can be certain that it did not acquire these mechanisms in a human-like manner. LLMs receive orders of magnitude more training data than do individual human beings (at least if we consider linguistic inputs alone), 59 and so they cannot be considered as models of the acquisition of analogical reasoning over the course of human development. Nor can they be considered good models of the evolution of analogical reasoning, as their analogical abilities are derived entirely from being trained to predict human-generated text. Human natural language is replete with analogies; accurately predicting natural language therefore likely requires an ability to appreciate analogies. But there is no reason to suppose that the same system, absent human-generated inputs, would spontaneously develop a disposition to think analogically, as apparently happened at some point in human evolution. 62 Thus, to the extent that large language models capture the analogical abilities of adult human reasoners, their capacity to do so is fundamentally parasitic on natural human intelligence. Nevertheless, the present results indicate that this approach may be sufficient to approximate human-like reasoning abilities, albeit through a radically different route than that taken by biological intelligence.
## 4 Methods
The present research complied with all relevant ethical regulations, and human behavioral experiments were approved by the UCLA Institutional Review Board (IRB protocol #22-000841, approved May 17, 2022).
## 4.1 Code
Most code was written in Python v3.9.6, using the following packages: NumPy v1.24.3, 63 SciPy v1.10.1, 64 statsmodels v0.13.5, 65 Matplotlib v3.7.1, 66 and pandas v2.0.1. 67 Logistic regression analyses were carried out in R v4.2.2. 68 Experimental stimuli for human behavioral experiments were written in JavaScript using jsPsych v7.2.1. 69
## 4.2 GPT-3
We queried GPT-3 in an automated fashion through the OpenAI API. All simulations reported in the main text employed the text-davinci-003 model variant. Additional simulations, reported in the Supplementary Results, also employed the davinci, code-davinci-002, and text-davinci-002 variants. The temperature was set to 0 in all simulations. We set max tokens (the parameter controlling the maximum number of generated tokens for a given prompt) to 10 for Digit Matrices, 40 for letter string analogies, 10 for four-term verbal analogies, and 256 for story analogies and analogical problem-solving. All other parameters were set to their default values.
For each prompt, GPT-3 generates a proposed completion (a string of tokens), and assigns log probabilities to each token in the prompt and the completion. We used these log probabilities to evaluate GPT-3 on multiple-choice problems. For each choice in a given problem, we concatenated the problem with the choice, and treated the average log probability assigned to the choice tokens as a score, selecting the answer choice with the highest score. This approach was used for Digit Matrices and four-term verbal analogies.
## 4.3 Digit Matrices
## 4.3.1 Dataset
The digit matrix problems consisted of two major problem categories: transformation and logic problems. Transformation problems contained anywhere from one to five rules, whereas logic problems each contained only a single rule. Transformation problems were defined using a combination of three rule types: constant, distribution-of-3, and progression. The constant rule was defined by the same digit appearing across either rows or columns. The following example shows an instance of a column-wise constant rule (correct answer: '9'):
$$\begin{array}{ccc}
5 & 1 & 9 \\
5 & 1 & 9 \\
5 & 1 & ?
\end{array}$$
The distribution-of-3 rule was defined by the same set of three digits appearing in each row or column, but with the order permuted. In the following example, the digits 6, 2, and 4 appear in each row (correct answer: '2'):
$$\begin{array}{ccc}
6 & 2 & 4 \\
2 & 4 & 6 \\
4 & 6 & ?
\end{array}$$
The progression rule was defined by a progressive increase or decrease in value, in units of either 1 or 2, across either rows or columns. In the following example, digits increase by units of 2 across rows (correct answer: '9'):
$$\begin{array}{ccc}
3 & 5 & 7 \\
1 & 3 & 5 \\
5 & 7 & ?
\end{array}$$
Transformation rules could be combined to form multi-rule problems, by assigning each rule to a particular spatial location within each cell. The following example shows a two-rule problem, in which the left digit in each cell is governed by a progression rule (digits decrease by units of 1 across columns), and the right digit in each cell is governed by a distribution-of-3 rule (correct answer: '4 9'):
$$\begin{array}{c|cccc}
7 & 1 & 8 & 9 & 6 & 3 \\
\hline
6 & 9 & 7 & 3 & 5 & 1 \\
\hline
5 & 3 & 6 & 1 & ? & \cdot
\end{array}$$
Logic problems were defined by one of three rules: OR, XOR, and AND. In the OR rule, a particular row or column contained all entities that appeared in either of the other rows or columns. In the following example, the middle column contains all entities that appear either in the left or right columns (correct answer: '8'):
$$\left \{ 7 \right \} \left \{ 7 4 \right \} \left \{ 4 \right \} \left \{ 9 7 \right \} \left \{ 9 7 4 8 \right \} \left \{ 4 8 \right \} \left \{ 9 \right \} \left \{ 9 8 \right \} \left \{ ? \right \}$$
The XOR rule was the same, except that entities appearing in both of the other rows or columns were excluded. In the following example, only items that appear in either the left or middle columns, but not both, will appear in the right column (correct answer: '4 3'):
$$\begin{array}{c|cccc}
\hline
64 & 61 & 41 \\
\hline
6 & 3 & ? \\
\hline
\end{array}$$
In the AND rule, a particular row or column contained only entities that appeared in both of the other rows or columns. In the following example, the left column contains only digits that appear in both the left and middle columns (correct answer: '9'):
$$[2 9 7] [1 9 7] [ 9 7 ]$$
For some logic problems, the within-cell spatial position of corresponding elements was aligned, as in the previously presented OR and AND problems. In other logic problems, corresponding elements were spatially permuted. The following example (involving an OR rule) illustrates how this makes it more difficult to intuitively grasp the underlying rule (correct answer: '0'):
$$\left \{ 1 \right \} \left \{ 7 \right \} \left \{ 7 \right \}$$
Within each problem type (one- through five-rule and logic problems), there were a number of specific problem subtypes. There were 6 one-rule subtypes, 6 two-rule subtypes, and 10 subtypes for three-rule, four-rule, five-rule, and logic problems. We generated 100 instances of each subtype (except in the case of progression problems, for which there were fewer possible problem instances). The one-rule problem subtypes consisted of a row-wise constant problem, a column-wise constant problem, two distribution-of-3 problems, and two progression problems (one with an increment of 1 and one with an increment of 2). The two- and three-rule problem subtypes consisted of all possible combinations of two or three rules (allowing for the same rule to be used multiple times within each problem). The four- and five-rule problem subtypes were sampled from the set of all possible combinations of four or five rules. There were five spatially aligned logic problem subtypes, and five spatially permuted logic problem subtypes. Three out of each of these five subtypes were OR problems (defined by the row or column in which the set union appeared), and the other two were AND and XOR problems.
For each problem, we also procedurally generated a set of 7 distractor choices, making for a set of 8 total answer choices. Distractors were generated using different methods for the transformation and logic problems. These methods were chosen based on the approach of Matzen et al., 32 who performed an analysis of the answer choices in the original SPM. For transformation problems, the following methods were used to generate distractors:
1. Sample a random cell from the problem.
2. Sample a random cell from the problem, sample a random digit within that cell, and apply an increment or decrement of either 1 or 2.
3. Start with the correct answer, apply an increment or decrement of either 1 or 2 to a randomly sampled digit.
4. Randomly sample a previously generated distractor for this problem, apply an increment or decrement of either 1 or 2 to a randomly sampled digit.
5. Randomly generate a new answer choice (with the appropriate number of digits given the problem type).
For multi-rule transformation problems, the following additional methods were also used:
1. Start with the correct answer, randomly permute the digits.
2. Sample a random cell from the problem, randomly permute the digits.
3. Randomly sample a previously generated distractor for this problem, randomly permute the digits.
4. Randomly sample digits from multiple cells within the problem and combine.
5. Randomly sample digits from previously generated distractors for this problem and combine.
For logic problems, distractors were generated by sampling from the set of all possible subsets of elements that appeared within the problem, including the empty set (the correct answer was an empty set on some logic problems), but excluding the correct answer. For spatially permuted logic problems, the spatial position of the elements within each distractor was randomly permuted. For spatially aligned logic problems, the order of the elements within each distractor was chosen so as to be consistent with the order that they appeared in the problem.
## 4.3.2 Human behavioral experiments
Human behavioral data was collected in two online experiments. All experiments were approved by the UCLA Institutional Review Board (IRB protocol #22-000841, approved May 17, 2022), and all participants provided informed consent. All participants were UCLA undergraduates. Forty-three participants completed the first experiment, but three participants were excluded from analysis due to the fact that they got nearly every answer incorrect, and produced an apparently random pattern of responses (e.g. random permutations of the same three digits for all problems). The remaining 40 participants (31 female, 18-35 years old, average age = 21.3 years old) were included in our analysis. Forty-seven participants (37 female, 18-42 years old, average age = 21.2 years old) completed the second experiment. No statistical methods were used to pre-determine sample sizes. There was no overlap between the participants in the first and second experiments. Participants received course credit for their participation.
In both experiments, participants were first presented with a set of instructions, and a single one-rule example problem involving a constant rule. For each problem, participants first generated a free-response answer, and then selected from the set of answer choices. Problems were presented in a spatially arranged matrix format, as they appear in Figure 2. Problems remained on the screen until participants made a response.
In the first experiment (Figure 3), participants were presented with one-, two-, three-rule, and logic problems. There were 6 problem subtypes each for the one- and two-rule problems, and 10 problem subtypes each for the threerule and logic problems, making for 32 problem subtypes in total. Participants received these problem subtypes in random order. Each participant received randomly sampled instances of each problem subtype.
In the second experiment (Supplementary Figure 4), participants were presented with one- through five-rule problems. There were 6 problem subtypes each for the one- and two-rule problems, and 10 problem subtypes each for the three- through five-rule problems, making for 42 problem subtypes in total. Problems were presented in order of increasing complexity, with all one-rule problem subtypes first, followed by all two-rule problem subtypes, and so on. For one-rule problems, the two constant problems were presented first, followed by the two distribution-of-3 problems, followed by the two progression problems.
## 4.3.3 Evaluating GPT-3
GPT-3 was evaluated on the Digit Matrices by presenting each complete problem as a prompt, including brackets and line breaks, followed by an open bracket at the start of the final cell. For example, the three-rule problem in Figure 2b would be presented to GPT-3 in the following format:
$$1 4 3 | 5 4 2 | n 1 2 2 | 5 2 7 | \$$
GPT-3's generated responses were truncated at the point where a closing bracket was generated. For logic problems, generated answers were counted as correct if they contained the correct set of digits, regardless of their order. For transformation problems, generated answers were only counted as correct if they contained the correct digits in the correct order. The same criteria were applied when evaluating human responses.
To evaluate GPT-3's multiple-choice performance, for each answer choice, the choice was appended to the problem followed by a closing bracket, and presented to GPT-3 as a prompt. The average log probability of the tokens corresponding to the answer choice (not counting the brackets) was computed. The answer choice with the highest average log probability was treated as GPT-3's selection.
In our primary evaluation (Figure 3), GPT-3 was presented with 40 problem instances from each of the 32 problem subtypes used in the first human behavioral experiment. GPT-3 solved each one zero-shot (without any fine-tuning or in-context learning).
We also evaluated how GPT-3 performed when presented with problems in order of increasing complexity (Supplementary Figure 4). GPT-3 performed 20 runs on this task. For each run, GPT-3 was presented with a series of the same 42 problem subtypes used in the second human behavioral experiment (with different instances of these subtypes in each run). After GPT-3 answered each problem, the selected multiple-choice answer was appended to the problem, and the combined problem and answer choice were recursively appended to the prompt for the next problem. This meant that the size of the prompt grew with each problem. For some of the final five-rule problems, the prompt exceeded the size of GPT-3's context window (4096 tokens). When this occurred, problems from the beginning of the context window were deleted until the entire prompt fit within the window. This resulted in the deletion of a few one-rule problems from the beginning of the prompt. For one-rule problems, the two constant problems were presented first, followed by the two distribution-of-3 rules, followed by the two progression problems.
## 4.3.4 Statistical analyses
Results were analyzed using both regression and correlation analyses. Logistic regression analyses were carried out at the individual trial level, with each data point corresponding to a particular trial from a particular participant (or GPT-3). The dependent variable in all regression analyses was a binary variable coding for whether a particular response was correct or incorrect.
For the first digit matrix experiment, we fit separate regression models for generative vs. multiple-choice responses. Two predictors were used: problem type (one-, two-, three-rule, and logic problems), and a binary predictor coding for GPT-3 vs. human participants. We also performed more fine-grained analyses for generative responses within each problem type. These analyses were performed separately for GPT-3 vs. human responses. For two-rule problems, a single binary predictor coded for whether a problem contained a progression rule. For three-rule problems, a single predictor coded for the number of unique rules present in a given problem. For logic problems, a binary predictor coding for whether a problem was spatially aligned vs. permuted.
We also fit regression models comparing the results of the first and second experiments. These analyses were performed separately for GPT-3 vs. human responses, and only included responses for one- to three-rule problems (since these were the only problem types in common between the two experiments). Two predictors were used: problem type (one-, two-, and three-rule problems), and experiment (experiment 1 vs. 2).
Correlation analyses were carried out by correlating the accuracy for GPT-3 vs. human participants across all 32 problem subtypes.
## 4.4 Letter string analogies
## 4.4.1 Problem set
Each letter string analogy problem involved one of six transformation types: sequence extension, successor, predecessor, removing a redundant letter, fixing an alphabetic sequence, and sorting. In the sequence extension transformation, the source involved an alphabetically ordered sequence of four letters followed by an extension of this sequence involving five letters, as in the following example:
$$\vert a b c d \vert \vert a b c d \vert$$
In the successor transformation, the source involved an alphabetically ordered sequence of four letters, followed by that same sequence, but with the final letter replaced by its successor, as in the following example:
$$\vert a b c d \vert a b c d$$
In the predecessor transformation, the source involved an alphabetically ordered sequence of four letters, followed by that same sequence, but with the first letter replaced by its predecessor, as in the following example:
$$\vert b c d \vert \vert a c d \vert$$
In the transformation involving removal of a redundant letter, the source involved an alphabetically ordered sequence of five letters with one letter repeated, followed by that same sequence with the redundant letter removed, as in the following example:
$$\vert \overrightarrow { a } b c d e \vert = \vert \overrightarrow { a } \vert + \vert \overrightarrow { b } \vert + \vert \overrightarrow { c } \vert + \vert \overrightarrow { d } \vert + \vert \overrightarrow { e } \vert$$
In the transformation involving fixing an alphabetic sequence, the source involved an alphabetically ordered sequence of five letters with one out-of-place letter (not part of the alphabetic sequence), followed by that same sequence with the out-of-place letter replaced, as in the following example:
$$\vert b \vert c \vert w \vert e \vert l \vert a \vert b \vert c \vert d \vert$$
In the sorting transformation, the source involved an alphabetically ordered sequence of five letters with the position of two letters swapped, followed by a sorted version of the same sequence, as in the following example:
$$\vert b \vert c \vert d \vert e \vert f \vert g \vert h \vert i \vert j \vert k \vert l \vert m \vert n \vert o \vert p \vert q \vert r \vert s \vert t \vert u \vert v \vert w \vert x \vert y \vert z$$
Problems involved varying degrees of generalization between the source and target. In the zero-generalization problems, the target involved a different instance of the source transformation (instantiated with different letters). Transformation parameters (e.g., the location of the redundant letter) were independently sampled for source and target.
Generalization problems involved generalizations sampled from the following set of generalization types: generalization from letters to numbers, grouping, generalization to a longer target, reversed order, interleaved distractors, and generalization to a larger interval. In the letter-to-number generalization, target letters were replaced by numbers corresponding to their alphabetic indices, as in the following example:
$$[ a b c d ] [ a b c d e ]$$
In the grouping generalization, target letters were replaced by groups with two instances of each letter, as in the following example:
$$[ a b c d ] [ a b c d e ]$$
In the longer target generalization, the target sequence was replaced with a sequence that was twice as long as the source, as in the following example:
$$[ a b c d ] [ a b c d e ]$$
In the reversed order generalization, the order of the target letters was reversed relative to the source, as in the following example:
$$[ a b c d ] [ a b c d e ]$$
In the interleaved distractor generalization, the letter 'x' was interleaved between each letter in the target sequence, as in the following example:
$$[ a b c d ] [ a b c d e ]$$
In the larger interval generalization, the sequence of target letters was replaced with a sequence involving an interval of size 2, as in the following example:
$$[ a b c d ] [ a b c d e ]$$
Each transformation type could be combined with any generalization type. Multiple generalizations could also be combined together. Generalization problems contained between one and three generalizations. We generated a set of 600 zero-generalization problems (involving 100 problems with each transformation type), 600 one-generalization problems (involving 100 problems with each generalization type, with randomly sampled transformation type), and 600 problems each with two and three generalizations (with randomly sampled combinations of transformation and generalization type).
We also generated a separate set of problems involving generalization from letters to real-world concepts. In these problems, the source instantiated a transformation using letters, and the target instantiated that same transformation using real-world instances of successorship. These problems involved shorter sequences (maximum length of four), due to the difficulty of identifying real-world instances of successorship with more than four points. The following sequences were used:
cold cool warm hot love like dislike hate jack queen king ace penny nickel dime quarter second minute hour day
The transformation types included sequence extension, successor, predecessor, and sorting. No other generalizations were applied to these problems. We generated 100 problems with each transformation type.
## 4.4.2 Evaluating GPT-3
We presented letter string analogies to GPT-3 using the prompt 'Let's try to complete the pattern:', similar to. 70 We also formatted each analogy problem using brackets and line breaks, similar to the presentation format of the Digit Matrices. The presentation format is illustrated in the following example:
Let's try to complete the pattern: \ n \ n[a b c d] [a b c e] \ n[i j k l] [
GPT-3's generated responses were truncated at the point where a closing bracket was generated. We also evaluated GPT-3 with two alternative problem formats: 1) no prompt, and 2) a sentence format, as in the following example:
If a b c d changes to a b c e, then i j k l should change to
For this format, GPT-3's generated responses were truncated at the point where a period was generated. We evaluated GPT-3 on 300 zero-generalization problems (50 problems for each transformation type), 300 one-generalization problems (50 problems for each generalization type), and 300 problems each with two and three generalizations. We also evaluated GPT-3 on 50 real-world concept generalization problems for each transformation type.
## 4.4.3 Human behavioral experiment
Human behavioral data was collected in an online experiment. The experiment was approved by the UCLA Institutional Review Board (IRB protocol #22-000841, approved May 17, 2022), and all participants provided informed consent. All participants were UCLA undergraduates. Fifty-seven participants (50 female, 18-35 years old, average age = 21.1 years old) completed the experiment. No statistical methods were used to pre-determine sample sizes. Participants received course credit for their participation.
Participants were first presented with a set of instructions, and the following example problem (not involving any of the transformations or generalizations employed in the actual experiment):
$$\vert a a a \vert \vert b b b \vert$$
Each participant completed 28 problems, including 6 zero-generalization problems (1 problem for each transformation type), 6 one-generalization problems (1 problem for each generalization type), 6 problems each with two and three generalizations, and 4 real-world concept generalization problems (1 for each transformation type). The specific problem instances were randomly sampled for each participant, and participants received these problems in a random order. Participants generated a free response for each problem.
## 4.4.4 Statistical analyses
Results were analyzed using both regression and correlation analyses. Logistic regression analyses were carried out at the individual trial level, with each data point corresponding to a particular trial from a particular participant (or GPT-3). The dependent variable in all regression analyses was a binary variable coding for whether a particular response was correct or incorrect.
Separate analyses were performed for problems that only involved alphanumeric characters vs. those that involved real-world concepts. For problems involving alphanumeric characters, a regression model was fit with two predictors: number of generalizations (zero to three), and a binary predictor coding for GPT-3 vs. human participants. We also fit regression models at each generalization level with a single binary predictor coding for GPT-3 vs. human participants. For real-world concept problems, a regression model was fit with a predictor coding for GPT-3 vs. human participants.
For correlation analyses, problem subtypes were defined based on each combination of transformation type and generalization type. The accuracy for each subtype was computed for GPT-3 vs. human participants, and these values were subjected to correlation analysis. There were only a few examples of some problem subtypes (across all participants), especially for problems with more generalizations (the space of possible subtypes grows exponentially with the number of generalizations). We only included subtypes for which there were at least five trials from human participants (across all participants) and five trials from GPT-3. Out of the 252 possible problem subtypes, 41 subtypes met this criterion and were included in the analysis.
## 4.5 Four-term verbal analogies
We evaluated GPT-3 on four separate four-term analogy datasets. 17,19-21 The UCLA-VAT dataset contains 80 problems, with four relation types: categorical (B/D is a member of the category A/C), functional (A/C is the function of B/D), antonym, and synonym. There are 20 problems for each relation type. Each problem contains two answer choices for the final term (D and D'). We evaluated GPT-3 by presenting the problem along with each possible answer choice (A:B::C:D or A:B::C:D'), using the standard colon notation, and selected the answer choice for which GPT-3 assigned a higher log probability to the final term. The problem and GPT-3's selected answer were then recursively appended to the prompt for the next problem. The problems were presented in a shuffled order. We compared against human behavioral data from 17 (N=57, UCLA undergraduates). Example problems from each of the four relation categories are shown below:
## Categorical
vegetable :
cabbage ::
insect : ?
1. beetle 2. frog
## Function
drive : car :: burn : ?
1. wood 2. fire
## Antonym
love :
hate ::
rich : ?
1. poor 2. wealthy
## Synonym
rob : steal :: cry : ?
1. weep 2. laugh
The dataset of Sternberg and Nigro 19 contains 200 problems, including 40 problems for each of five relation types: categorical, functional, antonym, synonym, and linear order. We evaluated GPT-3 in the same way that we did for UCLA-VAT, and compared against human behavioral data from 19 (N=20, Yale undergraduates). An example problem illustrating the linear order relation type is shown below (the categorical, functional, antonym, and synonym problems were similar to those from the UCLA VAT):
## Linear order
month : year :: inch : ? 1. foot 2. length
The dataset of SAT problems from Turney et al. 20 contains 374 problems, covering a range of different relation types. Each problem contains five answer choices for both C and D terms (including the correct answer). We evaluated GPT-3 by presenting each of the five possible analogies for each problem, and selecting the choice for which the C and D terms were assigned the highest log probability. The problem, and GPT-3's choice, were then appended to the prompt for the next problem. We compared against an estimate of the average performance level for high school students taking the SAT (see 40 ).
The dataset of Jones et al. 21 contains 120 problems, including 40 problems for each of three relation types: categorical, causal, and compositional. Half of these problems are categorized as semantically near (A and B are similar to C and D), and half are categorized as semantically far (A and B are dissimilar to C and D). Each problem contains two answer choices. We evaluated GPT-3 in the same way that we did for UCLA-VAT, and compared against human behavioral data from 21 (N=241, Wayne State University undergraduates). Example problems for each of the three relation categories are shown below:
## Categorical
diesel :
fuel :: bed : ?
1. furniture 2. pillow
## Causal
motion : sickness :: drought : ?
1. famine 2. rain
## 4.6 Story analogies
## 4.6.1 Materials
All story analogy materials were taken from a problem set created by Gentner et al. 18 (from their Experiment 2), and included in a verbal analogy inventory. 41 These materials involve 18 source stories. Each source story is accompanied by four potential target stories, forming four conditions: correct and incorrect near analogies (respectively termed 'literal similarity' and 'mere appearance' by Gentner et al.), both involving similar entities and first-order relations as the source, while differing from each other in higher-order causal relations; and correct and incorrect far analogies (respectively termed 'true analogy' and 'false analogy' by Gentner et al.), both involving similar first-order relations as the source but distinct entities, while differing from each other in causal relations. An example source story, along with target stories from each condition, is presented below:
Source story: Karla, an old hawk, lived at the top of a tall oak tree. One afternoon, she saw a hunter on the ground with a bow and some crude arrows that had no feathers. The hunter took aim and shot at the hawk but missed. Karla knew the hunter wanted her feathers so she glided down to the hunter and offered to give him a few. The hunter was so grateful that he pledged never to shoot at a hawk again. He went off and shot deer instead.
Near analogy - correct target story: Once there was an eagle named Zerdia who nested on a rocky cliff. One day she saw a sportsman coming with a crossbow and some bolts that had no feathers. The sportsman attacked but the bolts missed. Zerdia realized that the sportsman wanted her tailfeathers so she flew down and donated a few of her tailfeathers to the sportsman. The sportsman was pleased. He promised never to attack eagles again.
Near analogy - incorrect target story: Once there was an eagle named Zerdia who donated a few of her tailfeathers to a sportsman so he would promise never to attack eagles. One day Zerdia was nesting high on a rocky cliff when she saw the sportsman coming with a crossbow. Zerdia flew down to meet the man, but he attacked and felled her with a single bolt. As she fluttered to the ground Zerdia realized that the bolt had her own tailfeathers on it.
Far analogy - correct target story: Once there was a small country called Zerdia that learned to make the world's smartest computer. One day Zerdia was attacked by its warlike neighbor, Gagrach. But the missiles were badly aimed and the attack failed. The Zerdian government realized that Gagrach wanted Zerdian computers so it offered to sell some of its computers to the country. The government of Gagrach was very pleased. It promised never to attack Zerdia again.
Far analogy - incorrect target story: Once there was a small country called Zerdia that learned to make the world's smartest computer. Zerdia sold one of itssupercomputers to its neighbor, Gagrach, so Gagrach would promise never to attack Zerdia. But one day Zerdia was overwhelmed by a surprise attack from Gagrach. As it capitulated the crippled government of Zerdia realized that the attacker's missiles had been guided by Zerdian supercomputers.
## 4.6.2 Human behavioral experiment
Human behavioral data was collected in an online experiment. The experiment was approved by the UCLA Institutional Review Board (IRB protocol #22-000841, approved May 17, 2022), and all participants provided informed consent. All participants were UCLA undergraduates. Fifty-four participants (47 female, 18-44 years old, average age = 20.7 years old) completed the experiment. No statistical methods were used to pre-determine sample sizes. Participants received course credit for their participation.
## Compositional
steel : scissors :: apple : ? 1. cider 2. tree
After receiving instructions, participants were presented with 18 trials, each involving a different source story. On each trial, participants were presented with a source story (referred to as 'Story 1'), followed by two target stories (referred to as 'Story A' and 'Story B'), and asked 'Which of Story A and Story B is a better analogy to Story 1?'. Participants could select either story A or story B, or could indicate that they were both equally analogous. Accuracy was computed as the proportion of trials for which participants selected the correct target story.
On half of the trials, the target stories were from the near analogy conditions. On the other half of the trials, the target stories were from the far analogy conditions. The order of the two target stories was randomly shuffled on all trials.
## 4.6.3 Evaluating GPT-3
GPT-3 was evaluated by entering stories directly into the OpenAI playground. For each source story, GPT-3 was evaluated on both the near analogy comparison, and the far analogy comparison, and was also evaluated on both possible orderings for each pair of target stories, resulting in 18 Γ 2 Γ 2 = 72 total comparisons. For each comparison, the stories were presented in the following format:
Consider the following story:
Story 1: βͺ source story text β«
Now consider two more stories:
Story A: βͺ target story A text β«
Story B: βͺ target story B text β«
Which of Story A and Story B is a better analogy to Story 1? Is the best answer Story A, Story B, or both are equally analogous?
where βͺ source story text β« , βͺ target story A text β« , and βͺ target story B text β« were replaced by the text for the corresponding stories. In addition to answering the forced-choice question, GPT-3 sometimes spontaneously produced explanations, but only the forced-choice response was used in our analysis. GPT-3's context window was cleared after obtaining the results of each comparison.
## 4.6.4 Evaluating GPT-4
GPT-4 was evaluated by entering stories directly into the ChatGPT web interface. GPT-4 was evaluated on the same 72 problems, using the same format as was used for GPT-3. GPT-4's context window was cleared after obtaining the results of each comparison.
## 4.6.5 Statistical analyses
The task performed by both GPT-3 and human participants involved a three-choice discrimination (Story A is more analogous, Story B is more analogous, both are equally analogous). Statistical analyses were carried out to determine whether this discrimination was made at a level greater than expected from chance alone. To be conservative, we assumed a chance performance level of 50% accuracy. For GPT-3, a binomial test was performed (using data at the individual trial level). For human participants, a one-sample t-test was performed (using data averaged at the individual subject level). These analyses were carried out separately for the near analogy and far analogy conditions.
To compare GPT-3 with human performance, a logistic regression analysis was carried out at the individual trial level. The dependent variable was a binary variable coding for whether a particular response was correct or incorrect. A single binary predictor coded for GPT-3 vs. human responses.
## 4.7 Analogical problem-solving
Problems were entered directly into the OpenAI playground. Materials were taken from 22 and. 23 All prompts and responses are shown in Supplementary Section S5 . Each subsection shows the results for a single continuous session, with GPT-3's responses presented in bold text. Responses were not truncated or curated in any way.
## Data Availability
Data for all human behavioral experiments, along with the Digit Matrices, letter string analogy, and UCLA VAT problem sets, can be downloaded from:
https://github.com/taylorwwebb/emergent analogies LLM
The four-term verbal analogy problem sets from Sternberg and Nigro 19 and Jones et al., 21 and the story analogy materials from Gentner et al. 18 can be downloaded from:
http://cvl.psych.ucla.edu/resources/AnalogyInventory.zip
Information about the problem set of SAT four-term verbal analogies from Turney et al. 20 can be found at:
https://aclweb.org/aclwiki/SAT Analogy Questions (State of the art)
## Code Availability
Code for all simulations can be downloaded from:
https://github.com/taylorwwebb/emergent analogies LLM
## Acknowledgements
We would like to thank Bryor Snefjella and Peter Turney for helpful feedback and discussions. Preparation of this paper was supported by NSF grant IIS-1956441 and AFOSR MURI grant FA9550-22-1-0380 to H.L.
## Author Contributions Statement
T.W., K.J.H., and H.L. conceived project and planned experiments. T.W. implemented experiments and analyzed results. T.W., K.J.H., and H.L. drafted manuscript.
## Competing Interests Statement
The authors declare no competing interests.
## References
- 1 Keith J Holyoak. Analogy and relational reasoning. In Keith J Holyoak and Robert G Morrison, editors, Oxford handbook of thinking and reasoning , pages 234-259. Oxford University Press, 2012.
- 2 Miriam Bassok and Laura R Novick. Problem solving. In Keith J Holyoak and Robert G Morrison, editors, Oxford handbook of thinking and reasoning , pages 413-432. Oxford University Press, 2012.
- 3 Kevin N Dunbar and David Klahr. Scientific thinking and reasoning. In Keith J Holyoak and Robert G Morrison, editors, Oxford handbook of thinking and reasoning , pages 701-718. Oxford University Press, 2012.
- 4 Raymond B Cattell. Abilities: Their structure, growth, and action . Houghton Mifflin, 1971.
- 5 Richard E Snow, Patrick C Kyllonen, Brachia Marshalek, et al. The topography of ability and learning correlations. Advances in the psychology of human intelligence , 2(S 47):103, 1984.
- 6 Melanie Mitchell. Abstraction and analogy-making in artificial intelligence. Annals of the New York Academy of Sciences , 1505(1):79-101, 2021.
- 7 David Barrett, Felix Hill, Adam Santoro, Ari Morcos, and Timothy Lillicrap. Measuring abstract reasoning in neural networks. In International conference on machine learning , pages 511-520. PMLR, 2018.
- 8 Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song-Chun Zhu. Raven: A dataset for relational and analogical visual reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5317-5327, 2019.
- 9 Felix Hill, Adam Santoro, David G. T. Barrett, Ari S. Morcos, and Timothy P. Lillicrap. Learning to make analogies by contrasting abstract relational structure. In 7th International Conference on Learning Representations, ICLR , 2019.
- 10 Yuhuai Wu, Honghua Dong, Roger Grosse, and Jimmy Ba. The scattering compositional learner: Discovering objects, attributes, relationships in analogical reasoning. arXiv preprint arXiv:2007.04212 , 2020.
- 11 Michael Hersche, Mustafa Zeqiri, Luca Benini, Abu Sebastian, and Abbas Rahimi. A neuro-vector-symbolic architecture for solving raven's progressive matrices. Nature Machine Intelligence , 2023.
- 12 Shanka Subhra Mondal, Taylor W Webb, and Jonathan D Cohen. Learning to reason over visual objects. In 11th International Conference on Learning Representations, ICLR , 2023.
- 13 Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.
- 14 Kyle Mahowald, Anna A Ivanova, Idan A Blank, Nancy Kanwisher, Joshua B Tenenbaum, and Evelina Fedorenko. Dissociating language and thought in large language models: a cognitive perspective. arXiv preprint arXiv:2301.06627 , 2023.
- 15 John C Raven. Progressive matrices: A perceptual test of intelligence, individual form . London: Lewis, 1938.
- 16 Douglas R Hofstadter and Melanie Mitchell. The copycat project: A model of mental fluidity and analogy-making. In Keith J Holyoak and J A Barnden, editors, Advances in connectionist and neural computation theory , volume 2, page 31-112. Ablex, Norwood, NJ, 1994.
- 17 Hongjing Lu, Ying Nian Wu, and Keith J Holyoak. Emergence of analogy from relation learning. Proceedings of the National Academy of Sciences , 116(10):4176-4181, 2019.
- 18 Dedre Gentner, Mary Jo Rattermann, and Kenneth D Forbus. The roles of similarity in transfer: Separating retrievability from inferential soundness. Cognitive psychology , 25(4):524-575, 1993.
- 19 Robert J Sternberg and Georgia Nigro. Developmental patterns in the solution of verbal analogies. Child Development , pages 27-38, 1980.
- 20 Peter D Turney, Michael L Littman, Jeffrey Bigham, and Victor Shnayder. Combining independent modules to solve multiple-choice synonym and analogy problems. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP-03) , pages 482-489, 2003.
- 21 Lara L Jones, Matthew J Kmiecik, Jessica L Irwin, and Robert G Morrison. Differential effects of semantic distance, distractor salience, and relations in verbal analogy. Psychonomic bulletin & review , 29(4):1480-1491, 2022.
- 22 Mary L Gick and Keith J Holyoak. Analogical problem solving. Cognitive psychology , 12(3):306-355, 1980.
- 23 Keith J Holyoak, Ellen N Junn, and Dorrit O Billman. Development of analogical problem-solving skill. Child development , pages 2042-2055, 1984.
- 24 Ishita Dasgupta, Andrew K Lampinen, Stephanie CY Chan, Antonia Creswell, Dharshan Kumaran, James L McClelland, and Felix Hill. Language models show human-like content effects on reasoning. arXiv preprint arXiv:2207.07051 , 2022.
- 25 Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adri` a Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615 , 34:1877--1901, 2022.
- 26 Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682 , 2022.
- 27 Stephanie CY Chan, Adam Santoro, Andrew Kyle Lampinen, Jane X Wang, Aaditya K Singh, Pierre Harvey Richemond, James McClelland, and Felix Hill. Data distributional properties drive emergent in-context learning in transformers. In Advances in Neural Information Processing Systems , 2022.
- 28 Marcel Binz and Eric Schulz. Using cognitive psychology to understand gpt-3. Proceedings of the National Academy of Sciences , 120(6):e2218523120, 2023.
- 29 Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 31:5998-6008, 2017.
- 30 Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 , 2021.
- 31 Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems , 36:4299-4307, 2022.
- 32 Laura E Matzen, Zachary O Benz, Kevin R Dixon, Jamie Posey, James K Kroger, and Ann E Speed. Recreating Raven's: Software for systematically generating large numbers of Raven-like matrix problems with normed properties. Behavior research methods , 42(2):525-541, 2010.
- 33 Bryan J Matlen, Dedre Gentner, and Steven L Franconeri. Spatial alignment facilitates visual comparison. Journal of Experimental Psychology: Human Perception and Performance , 46(5):443, 2020.
- 34 James K Kroger, Keith J Holyoak, and John E Hummel. Varieties of sameness: The impact of relational complexity on perceptual comparisons. Cognitive Science , 28(3):335-358, 2004.
- 35 Graeme S Halford, William H Wilson, and Steven Phillips. Processing capacity defined by relational complexity: Implications for comparative, developmental, and cognitive psychology. Behavioral and brain sciences , 21(6):803831, 1998.
- 36 David J Chalmers, Robert M French, and Douglas R Hofstadter. High-level perception, representation, and analogy: A critique of artificial intelligence methodology. Journal of Experimental & Theoretical Artificial Intelligence , 4(3):185-211, 1992.
- 37 Douglas R Hofstadter. Fluid concepts and creative analogies: Computer models of the fundamental mechanisms of thought. Basic books, 1995.
- 38 Andrew Lovett and Kenneth Forbus. Modeling visual problem solving as analogical reasoning. Psychological review , 124(1):60, 2017.
- 39 Melanie Mitchell. Analogy-making as perception: A computer model . MIT Press, 1993.
- 40 Peter D Turney and Michael L Littman. Corpus-based learning of analogies and semantic relations. Machine Learning , 60:251-278, 2005.
- 41 Nicholas Ichien, Hongjing Lu, and Keith J Holyoak. Verbal analogy problem sets: An inventory of testing materials. Behavior research methods , 52:1803-1816, 2020.
- 42 Peter C Wason. Reasoning about a rule. Quarterly journal of experimental psychology , 20(3):273-281, 1968.
- 43 Dedre Gentner. Structure-mapping: A theoretical framework for analogy. Cognitive science , 7(2):155-170, 1983.
- 44 OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 , 2023.
- 45 Karl Duncker. On problem-solving. Psychological monographs , 58(5):i, 1945.
- 46 Keith J Holyoak and Kyunghee Koh. Surface and structural similarity in analogical transfer. Memory & cognition , 15(4):332-340, 1987.
- 47 James L McClelland, Felix Hill, Maja Rudolph, Jason Baldridge, and Hinrich Sch¨ utze. Placing language in an integrated understanding system: Next steps toward human-level performance in neural language models. Proceedings of the National Academy of Sciences , 117(42):25966-25974, 2020.
- 48 Gary F Marcus. The algebraic mind: Integrating connectionism and cognitive science . MIT press, 2001.
- 49 Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel J Gershman. Building machines that learn and think like people. Behavioral and brain sciences , 40, 2017.
- 50 Taylor W Webb, Zachary Dulberg, Steven Frankland, Alexander Petrov, Randall O'Reilly, and Jonathan Cohen. Learning representations that support extrapolation. In International conference on machine learning , pages 10136-10146. PMLR, 2020.
- 51 Brian Falkenhainer, Kenneth D Forbus, and Dedre Gentner. The structure-mapping engine: Algorithm and examples. Artificial intelligence , 41(1):1-63, 1989.
- 52 Hongjing Lu, Nicholas Ichien, and Keith J Holyoak. Probabilistic analogical mapping with semantic relation networks. Psychological Review , 2022.
- 53 Taylor W Webb, Shuhao Fu, Trevor Bihl, Keith J Holyoak, and Hongjing Lu. Zero-shot visual reasoning through probabilistic analogical mapping. arXiv preprint arXiv:2209.15087 , 2022.
- 54 Paul Smolensky. Tensor product variable binding and the representation of symbolic structures in connectionist systems. Artificial intelligence , 46(1-2):159-216, 1990.
- 55 Keith J Holyoak and John E Hummel. The proper treatment of symbols in a connectionist architecture. Cognitive dynamics: Conceptual change in humans and machines , 229:263, 2000.
- 56 Trenton Kriete, David C Noelle, Jonathan D Cohen, and Randall C O'Reilly. Indirection and symbol-like processing in the prefrontal cortex and basal ganglia. Proceedings of the National Academy of Sciences , 110(41):16390-16395, 2013.
- 57 Taylor W Webb, Ishan Sinha, and Jonathan D. Cohen. Emergent symbols through binding in external memory. In 9th International Conference on Learning Representations, ICLR , 2021.
- 58 Klaus Greff, Sjoerd Van Steenkiste, and J¨ urgen Schmidhuber. On the binding problem in artificial neural networks. arXiv preprint arXiv:2012.05208 , 2020.
- 59 Thomas L Griffiths. Understanding human intelligence through human limitations. Trends in Cognitive Sciences , 24(11):873-883, 2020.
- 60 Allen Newell, John Calman Shaw, and Herbert A Simon. Elements of a theory of human problem solving. Psychological review , 65(3):151, 1958.
- 61 Patricia A Carpenter, Marcel A Just, and Peter Shell. What one intelligence test measures: a theoretical account of the processing in the raven progressive matrices test. Psychological review , 97(3):404, 1990.
- 62 Derek C Penn, Keith J Holyoak, and Daniel J Povinelli. Darwin's mistake: Explaining the discontinuity between human and nonhuman minds. Behavioral and brain sciences , 31(2):109-130, 2008.
- 63 Charles R. Harris, K. Jarrod Millman, StΒ΄ efan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime FernΒ΄ andez del RΒ΄ Δ±o, Mark Wiebe, Pearu Peterson, Pierre GΒ΄ erard-Marchant, Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant. Array programming with NumPy. Nature , 585(7825):357-362, September 2020.
- 64 Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, StΒ΄ efan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C J Carey, Λ Ilhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, AntΛ onio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods , 17:261-272, 2020.
- 65 Skipper Seabold and Josef Perktold. statsmodels: Econometric and statistical modeling with python. In 9th Python in Science Conference , 2010.
- 66 J. D. Hunter. Matplotlib: A 2d graphics environment. Computing in Science & Engineering , 9(3):90-95, 2007.
- 67 The pandas development team. pandas-dev/pandas: Pandas, February 2020.
- 68 R Core Team. R: A Language and Environment for Statistical Computing . R Foundation for Statistical Computing, Vienna, Austria, 2021.
- 69 Joshua R De Leeuw. jspsych: A javascript library for creating behavioral experiments in a web browser. Behavior research methods , 47(1):1-12, 2015.
- 70 Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. arXiv preprint arXiv:2205.11916 , 2022.
## Supplementary Results
Supplementary Figure 1: Matrix reasoning results for all GPT-3 variants. Zero-shot results on Digit Matrices for four GPT-3 model variants: davinci, code-davinci-002, text-davinci-002, and text-davinci-003. Results reflect generative accuracy for major problem types, including transformation problems with between one and three rules, and logic problems. Human results reflect average performance for N=40 participants. Black error bars represent standard error of the mean across participants. Summary of human results is plotted here for comparison with GPT-3, individual participant data are shown in Main Text Figure 3a. Gray error bars represent 95% binomial confidence intervals for average performance across multiple problems.
<details>
<summary>Image 9 Details</summary>
### Visual Description
## Grouped Bar Chart: Generative Accuracy by Problem Type and Model
### Overview
The image displays a grouped bar chart comparing the generative accuracy of four different GPT-3 model variants and human performance across four categories of problem complexity. The chart illustrates how accuracy changes as the problem type increases in complexity from single-rule to logic-based tasks.
### Components/Axes
* **Chart Type:** Grouped bar chart.
* **Y-Axis:** Labeled "Generative accuracy". The scale runs from 0 to 1, with major tick marks at 0, 0.2, 0.4, 0.6, 0.8, and 1.
* **X-Axis:** Labeled "Problem type". It contains four categorical groups: "1-rule", "2-rule", "3-rule", and "Logic".
* **Legend:** Located in the top-right corner of the chart area. It defines five data series with corresponding colors:
* GPT-3 (davinci): Light pink/magenta
* GPT-3 (code-davinci-002): Medium purple
* GPT-3 (text-davinci-002): Dark purple
* GPT-3 (text-davinci-003): Very dark purple/indigo
* Human: Light blue/cyan
* **Error Bars:** Each bar includes a vertical black error bar extending above and below the top of the bar, indicating variability or confidence intervals in the measurements.
### Detailed Analysis
The chart is segmented into four problem type groups. Each group contains five bars, ordered from left to right as per the legend: GPT-3 (davinci), GPT-3 (code-davinci-002), GPT-3 (text-davinci-002), GPT-3 (text-davinci-003), and Human.
**1. 1-rule Problems:**
* **Trend:** All models and humans achieve very high accuracy, near the maximum value.
* **Approximate Values (with uncertainty):**
* GPT-3 (davinci): ~0.98
* GPT-3 (code-davinci-002): ~0.99
* GPT-3 (text-davinci-002): ~0.99
* GPT-3 (text-davinci-003): ~0.99
* Human: ~0.90 (slightly lower than the models)
**2. 2-rule Problems:**
* **Trend:** A general decrease in accuracy compared to 1-rule problems for all entities. The GPT-3 models remain clustered together, while human performance shows a more pronounced drop.
* **Approximate Values (with uncertainty):**
* GPT-3 (davinci): ~0.83
* GPT-3 (code-davinci-002): ~0.86
* GPT-3 (text-davinci-002): ~0.85
* GPT-3 (text-davinci-003): ~0.85
* Human: ~0.62
**3. 3-rule Problems:**
* **Trend:** A further decline in accuracy. The performance gap between the GPT-3 models widens slightly, and human accuracy continues to fall.
* **Approximate Values (with uncertainty):**
* GPT-3 (davinci): ~0.55
* GPT-3 (code-davinci-002): ~0.73
* GPT-3 (text-davinci-002): ~0.64
* GPT-3 (text-davinci-003): ~0.69
* Human: ~0.56
**4. Logic Problems:**
* **Trend:** This category shows the lowest overall accuracy and the greatest variance between models. GPT-3 (davinci) performs significantly worse than the other models. Human performance is comparable to the lowest-performing model in this category.
* **Approximate Values (with uncertainty):**
* GPT-3 (davinci): ~0.38
* GPT-3 (code-davinci-002): ~0.82
* GPT-3 (text-davinci-002): ~0.77
* GPT-3 (text-davinci-003): ~0.80
* Human: ~0.41
### Key Observations
1. **Inverse Relationship with Complexity:** There is a clear inverse relationship between problem complexity (number of rules/logic) and generative accuracy for all tested entities. Accuracy is highest for "1-rule" problems and lowest for "Logic" problems.
2. **Model Consistency on Rule-Based Tasks:** For rule-based problems (1, 2, and 3-rule), the four GPT-3 model variants perform relatively similarly, with accuracy scores clustered within a ~0.15 range for each category.
3. **Divergence on Logic Tasks:** The "Logic" problem type causes a significant divergence in model performance. GPT-3 (davinci) shows a dramatic drop in accuracy, while the other three models (code-davinci-002, text-davinci-002, text-davinci-003) maintain relatively high accuracy (~0.77-0.82).
4. **Human vs. Model Performance:** Humans outperform all models only in the "1-rule" category (though models are very close). For "2-rule" and "3-rule" problems, the GPT-3 models consistently outperform humans. In "Logic" problems, human performance (~0.41) is comparable to the lowest-performing model (GPT-3 davinci, ~0.38) but significantly below the other models.
### Interpretation
The data suggests that the ability to generate accurate solutions degrades as the underlying problem structure becomes more complex, involving multiple rules or abstract logic. This is a common challenge in both human and machine reasoning.
The consistent performance of the GPT-3 models on rule-based tasks indicates a strong capability for pattern application when the rules are explicit. The significant drop for GPT-3 (davinci) on logic problems, contrasted with the resilience of the later models (code-davinci-002, text-davinci-002/003), highlights a potential advancement in the later models' training or architecture that improves their capacity for logical reasoning or handling more abstract problem spaces.
The human performance curveβstarting high but dropping more steeply than most models on rule-based tasksβmight reflect different cognitive strategies. Humans may excel at simple, direct rule application but find the systematic application of multiple, potentially interacting rules more taxing than the models, which are optimized for such pattern-based tasks. The convergence of human and the weakest model's performance on logic problems suggests that this category represents a fundamental challenge for both biological and artificial intelligence as configured in this test.
</details>
Supplementary Figure 2: Letter string analogy results for all GPT-3 variants. Letter string analogy results for four GPT-3 model variants: davinci, code-davinci-002, text-davinci-002, and text-davinci-003. Results reflect generative accuracy as a function of the number of generalizations between source and target. Human results reflect average performance for N=57 participants. Black error bars represent standard error of the mean across participants. Summary of human results is plotted here for comparison with GPT-3, individual participant data are shown in Main Text Figure 6a. Gray error bars represent 95% binomial confidence intervals for average performance across multiple problems.
<details>
<summary>Image 10 Details</summary>
### Visual Description
## Grouped Bar Chart: Generative Accuracy by Number of Generalizations
### Overview
This is a grouped bar chart comparing the "Generative accuracy" of four different GPT-3 model variants and a "Human" baseline. The performance is measured across four distinct categories based on the "Number of generalizations" required (0, 1, 2, and 3). The chart includes error bars for each data point, indicating variability or confidence intervals.
### Components/Axes
* **Chart Type:** Grouped Bar Chart.
* **X-Axis (Horizontal):** Labeled "Number of generalizations". It has four categorical groups: `0`, `1`, `2`, and `3`.
* **Y-Axis (Vertical):** Labeled "Generative accuracy". It is a linear scale ranging from `0` to `1`, with major tick marks at intervals of `0.2` (0, 0.2, 0.4, 0.6, 0.8, 1).
* **Legend:** Located in the top-right corner of the chart area. It defines five data series with corresponding colors:
* **GPT-3 (davinci):** Light pink/magenta.
* **GPT-3 (code-davinci-002):** Medium purple.
* **GPT-3 (text-davinci-002):** Dark purple.
* **GPT-3 (text-davinci-003):** Dark blue/indigo.
* **Human:** Light cyan/blue.
* **Error Bars:** Each bar has a vertical black line extending from its top, representing the margin of error or standard deviation.
### Detailed Analysis
The following table reconstructs the approximate generative accuracy values for each entity across the four generalization categories. Values are estimated from the bar heights relative to the y-axis. The approximate range of the error bars is noted in parentheses.
| Number of Generalizations | GPT-3 (davinci) | GPT-3 (code-davinci-002) | GPT-3 (text-davinci-002) | GPT-3 (text-davinci-003) | Human |
| :--- | :--- | :--- | :--- | :--- | :--- |
| **0** | ~0.44 (Β±0.05) | ~0.73 (Β±0.04) | ~0.68 (Β±0.05) | ~0.72 (Β±0.04) | ~0.59 (Β±0.03) |
| **1** | ~0.30 (Β±0.04) | ~0.40 (Β±0.04) | ~0.39 (Β±0.04) | ~0.51 (Β±0.05) | ~0.42 (Β±0.03) |
| **2** | ~0.11 (Β±0.03) | ~0.29 (Β±0.04) | ~0.27 (Β±0.04) | ~0.36 (Β±0.04) | ~0.35 (Β±0.03) |
| **3** | ~0.05 (Β±0.02) | ~0.19 (Β±0.03) | ~0.14 (Β±0.03) | ~0.24 (Β±0.03) | ~0.29 (Β±0.03) |
**Trend Verification per Data Series:**
* **GPT-3 (davinci):** Shows a steep, consistent downward trend. Accuracy is highest at 0 generalizations (~0.44) and drops sharply to near zero at 3 generalizations (~0.05).
* **GPT-3 (code-davinci-002):** Also shows a clear downward trend, starting as the top performer at 0 generalizations (~0.73) and declining to ~0.19 at 3.
* **GPT-3 (text-davinci-002):** Follows a similar downward trajectory to `code-davinci-002`, starting at ~0.68 and ending at ~0.14.
* **GPT-3 (text-davinci-003):** Exhibits a downward trend but maintains the highest accuracy among the AI models at 1, 2, and 3 generalizations. It starts at ~0.72 and ends at ~0.24.
* **Human:** Shows a steady downward trend, starting at ~0.59 and ending at ~0.29. Notably, the human performance curve is less steep than most AI models.
### Key Observations
1. **Universal Performance Decline:** All five entities show a monotonic decrease in generative accuracy as the number of generalizations increases from 0 to 3.
2. **Model Hierarchy Shift:** At 0 generalizations, `code-davinci-002` and `text-davinci-003` are the top performers, significantly outperforming humans. However, as generalizations increase, the performance gap narrows and inverts.
3. **Human Resilience:** While human accuracy also declines, it degrades more gracefully than the AI models. At 3 generalizations, the human baseline (~0.29) surpasses all listed GPT-3 variants, with the closest model (`text-davinci-003`) at ~0.24.
4. **Model Comparison:** `text-davinci-003` consistently outperforms `text-davinci-002` and `davinci` across all categories. `code-davinci-002` is the strongest at 0 generalizations but falls behind `text-davinci-003` for higher generalization counts.
5. **Error Bars:** The error bars are relatively consistent across models and categories, suggesting similar levels of variance in the measurements. They do not overlap between the top and bottom performers within a category, indicating the differences are likely statistically significant.
### Interpretation
This chart demonstrates a clear inverse relationship between task complexity (operationalized as the "number of generalizations") and generative accuracy for both advanced AI models and humans. The data suggests that while certain GPT-3 models (particularly `code-davinci-002` and `text-davinci-003`) can achieve superhuman performance on tasks requiring zero generalization, their advantage diminishes rapidly as the need for abstraction or transfer of knowledge increases.
The key insight is the **crossover point in performance**. Humans appear to have a more robust or different mechanism for handling multiple generalizations, allowing them to maintain a higher baseline accuracy under more complex conditions. This implies that the AI models' generative capabilities, while powerful, may be more brittle and specialized to the specific patterns seen in their training data, struggling to generalize as effectively as humans when faced with novel combinations or abstractions. The chart highlights a specific frontier in AI capability: robust, multi-step generalization.
</details>
Supplementary Figure 3: Four-term verbal analogy results for all GPT-3 variants. Results on UCLA Verbal Analogy Test (VAT) for four GPT-3 model variants: davinci, code-davinci-002, text-davinci-002, and text-davinci-003. Results reflect multiple-choice accuracy for problems involving different relation categories. Gray horizontal line represents chance performance. Human results reflect average performance for N=57 participants. Black error bars represent standard error of the mean across participants. Summary of human results is plotted here for comparison with GPT-3, individual participant data are shown in Main Text Figure 7a. Gray error bars represent 95% binomial confidence intervals for average performance across multiple problems.
<details>
<summary>Image 11 Details</summary>
### Visual Description
## Bar Chart: Accuracy Comparison of GPT-3 Models vs. Human Performance
### Overview
This is a grouped bar chart comparing the accuracy of four different GPT-3 model variants and human performance across four distinct linguistic or reasoning tasks. The chart includes error bars for each data point and a horizontal reference line at 0.5 accuracy.
### Components/Axes
* **Chart Type:** Grouped bar chart with error bars.
* **Y-Axis:**
* **Label:** "Accuracy"
* **Scale:** Linear, from 0 to 1, with major tick marks at 0, 0.2, 0.4, 0.6, 0.8, and 1.
* **X-Axis:**
* **Categories (from left to right):** "Categorical", "Function", "Antonym", "Synonym".
* **Legend (Top-Right):**
* **Pink Bar:** GPT-3 (davinci)
* **Purple Bar:** GPT-3 (code-davinci-002)
* **Dark Purple Bar:** GPT-3 (text-davinci-002)
* **Navy Blue Bar:** GPT-3 (text-davinci-003)
* **Light Blue Bar:** Human
* **Reference Line:** A solid gray horizontal line at y = 0.5, spanning the width of the chart.
### Detailed Analysis
**Trend Verification & Data Points (Approximate Values):**
1. **Categorical Task:**
* **GPT-3 (davinci) [Pink]:** High accuracy, ~0.95. Error bar extends from ~0.85 to 1.0.
* **GPT-3 (code-davinci-002) [Purple]:** Significantly lower accuracy, ~0.45. Has a very large error bar extending from ~0.25 to ~0.65.
* **GPT-3 (text-davinci-002) [Dark Purple]:** Perfect or near-perfect accuracy, ~1.0. Error bar is minimal.
* **GPT-3 (text-davinci-003) [Navy Blue]:** High accuracy, ~0.85. Error bar extends from ~0.75 to ~0.95.
* **Human [Light Blue]:** High accuracy, ~0.85. Error bar extends from ~0.8 to ~0.9.
2. **Function Task:**
* **GPT-3 (davinci) [Pink]:** Accuracy ~0.8. Error bar from ~0.7 to ~0.9.
* **GPT-3 (code-davinci-002) [Purple]:** Accuracy ~0.7. Error bar from ~0.6 to ~0.8.
* **GPT-3 (text-davinci-002) [Dark Purple]:** Accuracy ~0.85. Error bar from ~0.75 to ~0.95.
* **GPT-3 (text-davinci-003) [Navy Blue]:** Accuracy ~0.85. Error bar from ~0.75 to ~0.95.
* **Human [Light Blue]:** Accuracy ~0.8. Error bar from ~0.75 to ~0.85.
3. **Antonym Task:**
* **GPT-3 (davinci) [Pink]:** Accuracy ~0.75. Error bar from ~0.6 to ~0.9.
* **GPT-3 (code-davinci-002) [Purple]:** Accuracy ~0.6. Error bar from ~0.4 to ~0.8.
* **GPT-3 (text-davinci-002) [Dark Purple]:** Accuracy ~0.8. Error bar from ~0.7 to ~0.9.
* **GPT-3 (text-davinci-003) [Navy Blue]:** Highest accuracy in this group, ~0.9. Error bar from ~0.8 to 1.0.
* **Human [Light Blue]:** Accuracy ~0.85. Error bar from ~0.8 to ~0.9.
4. **Synonym Task:**
* **GPT-3 (davinci) [Pink]:** High accuracy, ~0.95. Error bar from ~0.85 to 1.0.
* **GPT-3 (code-davinci-002) [Purple]:** Low accuracy, ~0.5. Has a very large error bar extending from ~0.3 to ~0.7.
* **GPT-3 (text-davinci-002) [Dark Purple]:** High accuracy, ~0.95. Error bar from ~0.85 to 1.0.
* **GPT-3 (text-davinci-003) [Navy Blue]:** High accuracy, ~0.95. Error bar from ~0.85 to 1.0.
* **Human [Light Blue]:** Accuracy ~0.85. Error bar from ~0.8 to ~0.9.
### Key Observations
1. **Model Performance Hierarchy:** The `text-davinci-002` and `text-davinci-003` models consistently show the highest and most stable accuracy across all four tasks, often matching or exceeding human performance.
2. **High Variability in `code-davinci-002`:** The `code-davinci-002` model exhibits the lowest average accuracy and the largest error bars (indicating high variance or uncertainty) in the "Categorical" and "Synonym" tasks. Its performance is notably poor in these areas compared to other models.
3. **Human Baseline:** Human performance is relatively consistent across all tasks, hovering around 0.85 accuracy with moderate error bars.
4. **Task Difficulty:** The "Antonym" task appears to be the most challenging overall, as it has the lowest peak accuracy (from `text-davinci-003` at ~0.9) and the lowest average performance across models. The "Categorical" and "Synonym" tasks show the widest performance gaps between models.
5. **Reference Line:** The gray line at 0.5 accuracy serves as a baseline. All models and human performance are above this line in all tasks, except for the lower bound of the error bar for `code-davinci-002` in "Categorical" and "Synonym" tasks, which dips below 0.5.
### Interpretation
This chart demonstrates the evolution and specialization of GPT-3 model variants on specific linguistic benchmarks. The data suggests that the `text-davinci` series (versions 002 and 003) represents a significant improvement in both accuracy and reliability over the earlier `davinci` and `code-davinci-002` models for these tasks.
The `code-davinci-002` model's poor and highly variable performance on "Categorical" and "Synonym" tasks is a notable outlier. This could indicate that this model, potentially optimized for code-related functions, is less robust or specialized for certain types of natural language understanding tasks. Its large error bars suggest inconsistent performance across test samples.
The fact that the latest models (`text-davinci-002/003`) meet or exceed human-level accuracy in three of the four tasks (Categorical, Function, Synonym) highlights the rapid advancement in language model capabilities for these specific domains. The "Antonym" task remains a relative challenge for all entities tested, suggesting it may involve more nuanced reasoning or a different type of linguistic knowledge. The chart effectively communicates that model selection is critical, as performance is not uniform across different task types.
</details>
## S1 Solutions to example matrix reasoning problems
The solution to the example visual matrix reasoning problem in Main Text Figure 2a is option 5. The problem is defined by a constant rule (applied to the number of shapes in each cell), and two distribution-of-3 rules (one applied to color, and one applied to shape). The solution to the example Digit Matrix problem in Main Text Figure 2b is option 7. This problem is also defined by a constant rule (applied to the digits in the center of each cell), and two distribution-of-3 rules (one applied to the digits on the left side of each cell, and one applied to the digits on the right side).
## S2 GPT-3 model variants
Since the initial release of GPT-3, 13 OpenAI has released a number of updates to the original base model. The largest version (175B parameters) of the base model, davinci, was trained exclusively on next-token prediction using a web-based corpus of text data. Code-davinci-002 was further trained on next-token prediction using a dataset of publicly available code from GitHub. 30 Text-davinci-002 and text-davinci-003 were both fine-tuned to respond appropriately to prompts. 31 Text-davinci-002 was initialized with code-davinci-002, and then fine-tuned using supervised learning based on a set of example prompts and responses. Text-davinci-003 was initialized with text-davinci-002, and then further fine-tuned using reinforcement learning from human feedback (RLHF). In RLHF, a reward model (a separate neural network) is first trained to predict human ratings for pairs of human-generated prompts and language-model responses, and this reward model is then used to fine-tune the primary language model through reinforcement learning. More details on the different model variants and training objectives can be found at https://platform.openai.com/docs/model-index-for-researchers.
We evaluated all four of these variants on Digit Matrices (Figure 1), letter string analogies (Figure 2), and fourterm verbal analogies (Figure 3). Text-davinci-003 displayed the best overall performance, but other model variants performed well on a subset of tasks. For instance, code-davinci-002 performed well on the Digit Matrices and letter string problems. These task domains both involve simple alphanumeric characters and highly regular relational structure, similar to computer code. It therefore seems likely that code-davinci-002's strong performance on these
<details>
<summary>Image 12 Details</summary>
### Visual Description
## Bar Charts: Generative Accuracy Comparison (GPT-3 vs. Human)
### Overview
The image contains two bar charts, labeled **a** and **b**, comparing the "Generative accuracy" of GPT-3 (dark blue bars) and Human performance (light blue bars) across different problem types. Both charts share the same y-axis scale (0 to 1) and legend. The data includes error bars, indicating variability or confidence intervals.
### Components/Axes
**Common Elements:**
* **Y-axis (Both Charts):** Label: "Generative accuracy". Scale: 0 to 1, with major ticks at 0.0, 0.2, 0.4, 0.6, 0.8, 1.0.
* **Legend (Top-Right of each chart):**
* Dark Blue Square: "GPT-3"
* Light Blue Square: "Human"
**Chart a (Left):**
* **Title/Label:** "a" (top-left corner).
* **X-axis Label:** "Problem type".
* **X-axis Categories (from left to right):** "1-rule", "2-rule", "3-rule", "4-rule", "5-rule".
* **Data Series:** For each category, two bars are shown side-by-side: GPT-3 (left, dark blue) and Human (right, light blue).
**Chart b (Right):**
* **Title/Label:** "b" (top-left corner).
* **Chart Subtitle:** "One-rule problems" (centered above the chart).
* **X-axis Label:** "Rule type".
* **X-axis Categories (from left to right):** "Constant", "Distribution", "Progression".
* **Data Series:** For each category, two bars are shown side-by-side: GPT-3 (left, dark blue) and Human (right, light blue).
### Detailed Analysis
**Chart a: Accuracy by Number of Rules**
* **Trend Verification:** Both series show a general downward trend as the number of rules increases. GPT-3's decline appears steeper than the Human's.
* **Data Points (Approximate values with estimated error bar ranges):**
* **1-rule:** GPT-3 β 0.82 (error: ~0.78-0.86); Human β 0.89 (error: ~0.85-0.93).
* **2-rule:** GPT-3 β 0.78 (error: ~0.74-0.82); Human β 0.78 (error: ~0.74-0.82).
* **3-rule:** GPT-3 β 0.85 (error: ~0.81-0.89); Human β 0.76 (error: ~0.72-0.80).
* **4-rule:** GPT-3 β 0.73 (error: ~0.69-0.77); Human β 0.76 (error: ~0.72-0.80).
* **5-rule:** GPT-3 β 0.62 (error: ~0.58-0.66); Human β 0.71 (error: ~0.67-0.75).
**Chart b: Accuracy for One-Rule Problems by Rule Type**
* **Trend Verification:** Performance is high and similar for "Constant" and "Distribution" rule types. A significant drop occurs for the "Progression" rule type, especially for GPT-3.
* **Data Points (Approximate values with estimated error bar ranges):**
* **Constant:** GPT-3 β 1.00 (error: ~0.98-1.00); Human β 1.00 (error: ~0.98-1.00).
* **Distribution:** GPT-3 β 1.00 (error: ~0.98-1.00); Human β 0.99 (error: ~0.95-1.00).
* **Progression:** GPT-3 β 0.45 (error: ~0.30-0.60); Human β 0.68 (error: ~0.60-0.76).
### Key Observations
1. **Complexity Impact (Chart a):** Generative accuracy for both GPT-3 and Humans generally decreases as the problem complexity (number of rules) increases from 1 to 5.
2. **AI vs. Human Gap (Chart a):** GPT-3 outperforms Humans on the simplest (1-rule) and most complex (3-rule, 4-rule) problems in this set, but matches or is slightly below on 2-rule and 5-rule problems. The largest performance gap favoring GPT-3 is at 3-rule problems.
3. **Rule Type Specificity (Chart b):** For single-rule problems, both GPT-3 and Humans achieve near-perfect accuracy on "Constant" and "Distribution" rules. However, a major performance divergence occurs on "Progression" rules, where GPT-3's accuracy drops to approximately 0.45, while Human accuracy remains relatively high at ~0.68.
4. **Variability:** Error bars are generally wider for GPT-3 in Chart b's "Progression" category and for both series in Chart a's higher-rule categories, suggesting less consistent performance on more difficult tasks.
### Interpretation
The data suggests that the difficulty of a generative task is not solely a function of the number of rules. The *type* of rule is a critical factor. While both AI and humans handle static ("Constant") or statistical ("Distribution") rules with high accuracy, tasks involving sequential or evolving patterns ("Progression") present a specific challenge, particularly for the AI model tested.
The trends indicate that GPT-3's performance is more sensitive to the specific cognitive demands of a rule type than to a simple count of rules. Its relative strength on 3-rule and 4-rule problems in Chart a, contrasted with its weakness on "Progression" rules in Chart b, implies that the 3- and 4-rule problems may not heavily rely on progression-type logic. This analysis highlights the importance of deconstructing problem complexity into qualitative components (rule type) rather than relying solely on quantitative measures (rule count) when evaluating and comparing AI and human reasoning capabilities.
</details>
Problemtype
Supplementary Figure 4: GPT-3 shows human-like contextual effects. In a separate experiment, we presented both GPT-3 and human participants (N=47, UCLA undergraduates) with Digit Matrix problems in order of increasing complexity (easy-to-hard: one-rule problems, followed by two-rule problems, and so on). (a) Both GPT-3 and human participants were able to generalize the structure inferred from few-rule problems to more complex many-rule problems, resulting in very little decrease in performance even for five-rule problems (compare with the decrease in performance from one- to three-rule problems seen in Main Text Figure 3). (b) One-rule problems were also presented in order of increasing complexity, beginning with constant problems, followed by distribution-of-3 problems, followed by progression problems. Interestingly, this actually impaired performance on progression problems relative to zero-shot (or shuffled) presentation. This was likely due to a tendency to mistake the progression rule for the distribution-of-3 rule in the previously presented problems (which only differ in terms of a single digit). Both GPT-3 and human participants showed this effect. Human results reflect average performance for N=47 participants. Black error bars represent standard error of the mean across participants. Each dot represents accuracy for a single participant. Gray error bars represent 95% binomial confidence intervals for average performance across multiple problems.
tasks was a consequence of having been trained on code. By contrast, code-davinci-002 performed very poorly on four-term verbal analogies (near chance performance), whereas the original davinci model performed relatively well on these problems. This suggests that code-davinci-002's ability to model synthetic code-like structures may have come at the cost of the ability to process more real-world relational concepts. Text-davinci-002, and especially textdavinci-003, seem to have combined both of these abilities, perhaps as a result of prompt training, though these models may have also received additional fine-tuning on the original language modeling task. 31 Finally, it seems likely that prompt training improved text-davinci-002 and text-davinci-003's ability to perform tasks without the need for few-shot task demonstrations, therefore making it easier to evaluate these models' latent capabilities in a zero-shot setting.
We also performed an initial investigation of GPT-4 on the story analogy problems from Gentner et al. 18 (Supplementary Figure 6). GPT-4 showed significant improvement on this task relative to GPT-3, more reliably identifying the target story that shared higher-order relations with the source, and providing more precise explanations (Section S4.3 ). We were not able to test GPT-4 on the other analogy tasks due to a lack of API access. Very little is known about the details of GPT-4, but it is likely that this improvement stems at least in part from increased scale of both model and training set. 44
## S3 Presence of test materials in GPT-3's training data
Given the massive and uncurated nature of GPT-3's training data, it is important to consider the likelihood that our test materials were included in this training data, and the possibility therefore that GPT-3 may have memorized some of these materials (thus undermining their use as a test of zero-shot reasoning).
The Digit Matrices dataset was created specifically for the purposes of our study and therefore certainly was
a
Supplementary Figure 5: Letter string analogy results with alternative formatting. (a) GPT-3 performed worse when letter string problems were presented without prompt, in comparison with results (shown in Main Text Figure 6a) when same problems were formatted in the standard way (using brackets to demarcate the analogs, and with the prompt 'Let's try to complete the pattern'). (b) GPT-3 also performed worse when problems were presented in the form of a sentence (e.g., 'If a b c d changes to a b c e, then i j k l should change to'). Human results reflect average performance for N=57 participants (UCLA undergraduates). Black error bars represent standard error of the mean across participants. Each dot represents accuracy for a single participant. Gray error bars represent 95% binomial confidence intervals for average performance across multiple problems.
<details>
<summary>Image 13 Details</summary>
### Visual Description
## Bar Chart: Generative Accuracy by Number of Generalizations and Prompt Condition
### Overview
The image displays two side-by-side bar charts (labeled **a** and **b**) comparing the "Generative accuracy" of two entitiesβ**GPT-3** and **Human**βacross four levels of task complexity, defined by the "Number of generalizations" (0, 1, 2, 3). The left chart (a) shows results under a "No prompt" condition, while the right chart (b) shows results under a "Sentence format" prompt condition. Both charts share identical axes and a common legend.
### Components/Axes
* **Titles:**
* Chart a (left): "No prompt"
* Chart b (right): "Sentence format"
* **Y-axis (Vertical):** Label: "Generative accuracy". Scale: Linear from 0 to 1, with major tick marks at 0, 0.2, 0.4, 0.6, 0.8, and 1.
* **X-axis (Horizontal):** Label: "Number of generalizations". Categories: 0, 1, 2, 3.
* **Legend:** Located in the top-right corner of chart b.
* **GPT-3:** Dark purple bar.
* **Human:** Light blue bar.
* **Visual Elements:**
* Each data point is represented by a vertical bar with a black error bar extending from its top.
* In the background of both charts, there are several horizontal, gray, dashed lines at various y-values (approximately at 0.17, 0.33, 0.50, 0.67, 0.83). These likely represent statistical significance thresholds or benchmark levels, but are not labeled.
### Detailed Analysis
**Chart a: No prompt**
* **Trend:** For both GPT-3 and Human, generative accuracy decreases as the number of generalizations increases.
* **Data Points (Approximate Values):**
* **0 Generalizations:** GPT-3 β 0.65, Human β 0.60.
* **1 Generalization:** GPT-3 β 0.35, Human β 0.42.
* **2 Generalizations:** GPT-3 β 0.28, Human β 0.35.
* **3 Generalizations:** GPT-3 β 0.22, Human β 0.28.
**Chart b: Sentence format**
* **Trend:** Similar downward trend for both entities as generalizations increase.
* **Data Points (Approximate Values):**
* **0 Generalizations:** GPT-3 β 0.60, Human β 0.58.
* **1 Generalization:** GPT-3 β 0.46, Human β 0.42.
* **2 Generalizations:** GPT-3 β 0.29, Human β 0.35.
* **3 Generalizations:** GPT-3 β 0.21, Human β 0.28.
### Key Observations
1. **Performance Decline:** The most prominent pattern is a consistent decline in accuracy for both GPT-3 and humans as the task requires more generalizations (from 0 to 3).
2. **Prompt Effect:** The "Sentence format" prompt appears to alter the performance gap between GPT-3 and humans, particularly at lower generalization counts.
* At **0 generalizations**, GPT-3's accuracy is slightly higher than Human in the "No prompt" condition, but they are nearly equal in the "Sentence format" condition.
* At **1 generalization**, GPT-3's accuracy is lower than Human in the "No prompt" condition, but higher than Human in the "Sentence format" condition.
3. **Convergence at High Complexity:** At 2 and 3 generalizations, the performance of GPT-3 and humans is relatively similar in both prompt conditions, with humans maintaining a slight edge.
4. **Error Bars:** The presence of error bars on all data points indicates measured variability or confidence intervals around the mean accuracy scores.
### Interpretation
This data suggests a core finding: **the ability to generalize degrades for both artificial (GPT-3) and human intelligence as the number of required generalizations increases.** The task becomes fundamentally harder with added layers of abstraction.
The comparison between the "No prompt" and "Sentence format" conditions provides a nuanced view of how prompting affects model performance relative to humans. The "Sentence format" prompt seems to **modulate the performance differential**. It narrows the gap at the simplest task (0 generalizations) and reverses the gap at one generalization, where GPT-3 outperforms humans. This implies that the format of the instruction can significantly impact how effectively a model like GPT-3 applies its capabilities, sometimes bringing it closer to or even surpassing human baseline performance on specific sub-tasks.
The converging performance at higher generalization counts (2 and 3) suggests a potential **ceiling effect** or a common bottleneck in complex reasoning that affects both biological and artificial systems similarly. The unlabeled dashed lines hint at a statistical framework for evaluating these differences, but without their explicit meaning, we can only infer they mark thresholds for significant performance levels. Overall, the charts illustrate that while raw capability (like GPT-3's at 0 generalizations) can be high, the robustness of that capability across increasingly complex, multi-step reasoning tasks is a critical differentiator, and one where human and model performance can be sensitive to contextual factors like prompt design.
</details>
not included in GPT-3's training data. Furthermore, this problem format itself is, to the best of our knowledge, completely novel, and it is thus extremely unlikely that GPT-3 has been trained on similar problems.
The letter string problem set that we used was also created specifically for the purposes of our study, and so could not have been included in GPT-3's training data. It is possible that GPT-3 has been trained on other letter string analogy problems, as these problems are discussed on a number of webpages (e.g., https://cogsci.indiana.edu/lap.html). However, we were not able to obtain any evidence that GPT-3 knows about these problems, e.g., by asking it to describe or give examples of such problems.
The UCLA Verbal Analogy Test, 17 and the four-term verbal analogy sets from Sternberg and Nigro 19 and Jones et al. 21 are all available in a downloadable supplement in a recently published paper, 41 though to our knowledge they are not published directly in the form of webpages, and are thus unlikely to be included in web crawl data such as the Common Crawl dataset on which GPT-3 was trained. The dataset of SAT problems from Turney et al. 20 is not publicly available, but has been distributed to a relatively small number of researchers. It is possible that any of these datasets were deliberately included in GPT-3's training data (i.e., as a supplement to the web crawl data). However, we were not able to find any evidence that GPT-3 had memorized any of these problems, e.g., by prompting GPT-3 to complete these problems based only on the source analog (the A and B terms alone).
The story analogy materials from Gentner et al. 18 were also included in the downloadable supplement associated with, 41 though to our knowledge have not been published directly to any webpages. We were also not able to obtain any evidence that GPT-3 had memorized any of these stories, e.g., by asking it to complete partial stories.
The analogy between the general story and the radiation problem is discussed in the original publication from Gick and Holyoak, 22 and is discussed on other webpages as well (e.g., http://cognitivepsychology.wikidot.com/problemsolving:analogy). We found evidence that GPT-3 is familiar with this study (it was able to identify the year of publication, and journal in which it was published), but we did not find any evidence that GPT-3 had memorized the details of the analogy itself. GPT-3 was also able to solve the radiation problem via an analogy with lesser known source stories. 46 It seems unlikely that GPT-3's success on these problems is due to prior exposure, but these
<details>
<summary>Image 14 Details</summary>
### Visual Description
\n
## Bar Chart: Accuracy Comparison on Near vs. Far Analogies
### Overview
The image is a grouped bar chart comparing the accuracy of three entitiesβGPT-3, GPT-4, and Humansβon two types of tasks: "Near analogy" and "Far analogy." The chart includes error bars for each data point and a horizontal reference line at the 0.5 accuracy level.
### Components/Axes
* **Chart Type:** Grouped bar chart.
* **Y-Axis:**
* **Label:** "Accuracy"
* **Scale:** Linear, ranging from 0 to 1, with major tick marks at 0, 0.2, 0.4, 0.6, 0.8, and 1.0.
* **X-Axis:**
* **Categories:** Two primary categories are displayed: "Near analogy" (left group) and "Far analogy" (right group).
* **Legend:**
* **Location:** Top-right corner of the chart area.
* **Entries:**
* **GPT-3:** Represented by a dark purple bar.
* **GPT-4:** Represented by a magenta bar.
* **Human:** Represented by a light blue bar.
* **Additional Elements:**
* **Error Bars:** Each bar has a vertical black line extending above and below the top of the bar, indicating variability or confidence intervals.
* **Reference Line:** A solid horizontal gray line is drawn across the chart at the `y = 0.5` mark.
### Detailed Analysis
**1. Near Analogy Task (Left Group):**
* **GPT-3 (Dark Purple):** The bar height is approximately **0.75**. The error bar extends from roughly 0.60 to 0.90.
* **GPT-4 (Magenta):** This is the tallest bar in the chart, with a height very close to **1.0** (approximately 0.98). Its error bar is relatively small, spanning from about 0.95 to 1.0.
* **Human (Light Blue):** The bar height is approximately **0.90**. The error bar extends from about 0.80 to 1.0.
**2. Far Analogy Task (Right Group):**
* **GPT-3 (Dark Purple):** The bar height is approximately **0.65**. The error bar is substantial, extending from roughly 0.50 to 0.80.
* **GPT-4 (Magenta):** The bar height is approximately **0.75**. The error bar spans from about 0.60 to 0.90.
* **Human (Light Blue):** The bar height is approximately **0.85**. The error bar extends from about 0.70 to 1.0.
**Trend Verification:**
* **GPT-3:** Performance drops from ~0.75 (Near) to ~0.65 (Far).
* **GPT-4:** Performance drops significantly from ~0.98 (Near) to ~0.75 (Far).
* **Human:** Performance shows a smaller decline from ~0.90 (Near) to ~0.85 (Far).
### Key Observations
1. **Performance Hierarchy (Near):** GPT-4 > Human > GPT-3. GPT-4 achieves near-perfect accuracy on near analogies.
2. **Performance Hierarchy (Far):** Human > GPT-4 > GPT-3. Humans outperform both models on far analogies.
3. **Largest Performance Drop:** GPT-4 exhibits the most dramatic decrease in accuracy (~23 percentage points) when moving from near to far analogies.
4. **Most Consistent Performance:** Humans show the smallest decline in accuracy between the two tasks (~5 percentage points).
5. **Error Bar Patterns:** Error bars are generally larger for the "Far analogy" task across all entities, suggesting greater variability or uncertainty in performance on more distant analogies.
6. **Reference Line:** All data points are above the 0.5 (chance) line, indicating all entities perform better than random guessing on both tasks.
### Interpretation
The data suggests a clear distinction in how different systems handle analogical reasoning based on the "distance" of the analogy.
* **GPT-4's Specialization:** GPT-4 demonstrates exceptional, near-human-level mastery of **near analogies**, which likely involve surface-level or closely related conceptual mappings. Its performance drop on far analogies indicates a potential limitation in abstracting or transferring knowledge to more distant domains.
* **Human Robustness:** Humans show strong and relatively stable performance across both task types. Their slight edge on far analogies suggests a superior ability for abstract reasoning and flexible knowledge application, which are hallmarks of human cognition.
* **GPT-3's Baseline:** GPT-3 serves as a baseline, showing competent but inferior performance to both GPT-4 and humans on both tasks, with a notable struggle on far analogies.
* **Implication for AI Development:** The chart highlights that while advanced models like GPT-4 can achieve superhuman performance on specific, well-defined tasks (near analogies), bridging the gap to human-like robustness and flexibility in more abstract reasoning (far analogies) remains a challenge. The larger error bars on far analogies for all groups also indicate this is a more difficult and variable problem space.
</details>
Supplementary Figure 6: Story analogy results including test of GPT-4. Results for identification of analogies between stories, including both GPT-3 and GPT-4. Whereas GPT-3 was only reliably able to perform the task in the near analogy condition, GPT-4 displayed much more robust performance. GPT-4 showed nearly perfect performance in the near analogy condition, and displayed some sensitivity to higher-order relations in the far analogy condition (two-sided binomial test: p = 0 . 0039; null hypothesis is chance-level performance of 0.5). Human results reflect average performance for N=54 participants (UCLA undergraduates). Black error bars represent standard error of the mean across participants. Each dot represents accuracy for a single participant. Gray error bars represent 95% binomial confidence intervals for average performance across multiple problems. Gray horizontal line represents chance performance.
results should be interpreted with caution. The other materials used to evaluate analogical problem solving are also discussed in the associated publication, 23 but we did not find any evidence that GPT-3 is familiar with this study, or that it had memorized these materials.
## S4 Story analogy results
The following section contains examples of GPT-3's responses to the story analogy problems. GPT-3's responses are presented in bold text.
## S4.1 Near analogy
The following contains an example from the near analogy condition. GPT-3 identified the correct answer (Story A).
Consider the following story:
Story 1: Karla, an old hawk, lived at the top of a tall oak tree. One afternoon, she saw a hunter on the ground with a bow and some crude arrows that had no feathers. The hunter took aim and shot at the hawk but missed. Karla knew the hunter wanted her feathers so she glided down to the hunter and offered to give him a few. The hunter was so grateful that he pledged never to shoot at a hawk again. He went off and shot deer instead.
Now consider two more stories:
Story A: Once there was an eagle named Zerdia who nested on a rocky cliff. One day she saw a sportsman coming with a crossbow and some bolts that had no feathers. The sportsman attacked but the bolts
missed. Zerdia realized that the sportsman wanted her tailfeathers so she flew down and donated a few of her tailfeathers to the sportsman. The sportsman was pleased. He promised never to attack eagles again.
Story B: Once there was an eagle named Zerdia who donated a few of her tailfeathers to a sportsman so he would promise never to attack eagles. One day Zerdia was nesting high on a rocky cliff when she saw the sportsman coming with a crossbow. Zerdia flew down to meet the man, but he attacked and felled her with a single bolt. As she fluttered to the ground Zerdia realized that the bolt had her own tailfeathers on it.
Which of Story A and Story B is a better analogy to Story 1? Is the best answer Story A, Story B, or both are equally analogous?
Story A is the better analogy to Story 1. Story B is not as analogous because it has a different ending. In Story 1, the hunter is grateful for the hawk's feathers and promises not to shoot at hawks again. In Story B, the hunter attacks the eagle and kills her with her own feathers.
## S4.2 Far analogy
The following contains an example from the far analogy condition. GPT-3 identified the correct answer (Story A).
Consider the following story:
Story 1: Karla, an old hawk, lived at the top of a tall oak tree. One afternoon, she saw a hunter on the ground with a bow and some crude arrows that had no feathers. The hunter took aim and shot at the hawk but missed. Karla knew the hunter wanted her feathers so she glided down to the hunter and offered to give him a few. The hunter was so grateful that he pledged never to shoot at a hawk again. He went off and shot deer instead.
Now consider two more stories:
Story A: Once there was a small country called Zerdia that learned to make the world's smartest computer. One day Zerdia was attacked by its warlike neighbor, Gagrach. But the missiles were badly aimed and the attack failed. The Zerdian government realized that Gagrach wanted Zerdian computers so it offered to sell some of its computers to the country. The government of Gagrach was very pleased. It promised never to attack Zerdia again.
Story B: Once there was a small country called Zerdia that learned to make the world's smartest computer. Zerdia sold one of itssupercomputers to its neighbor, Gagrach, so Gagrach would promise never to attack Zerdia. But one day Zerdia was overwhelmed by a surprise attack from Gagrach. As it capitulated the crippled government of Zerdia realized that the attacker's missiles had been guided by Zerdian supercomputers.
Which of Story A and Story B is a better analogy to Story 1? Is the best answer Story A, Story B, or both are equally analogous?
The best answer is Story A. Story A is a better analogy to Story 1 because it follows a similar pattern of a protagonist offering something of value to an antagonist in exchange for a promise of peace. In Story 1, Karla offers her feathers to the hunter in exchange for a promise not to shoot at hawks again. In Story A, Zerdia offers its computers to Gagrach in exchange for a promise not to attack Zerdia again.
## S4.3 GPT-4
The following contains an example response from GPT-4 in the far analogy condition. GPT-4 both identifies the correct answer (Story A), and offers a precise explanation of the analogy.
Consider the following story:
Story 1: William was a patient in a psychiatric hospital who was confined indoors almost all the time. He could never pass the monthly room inspections so he hated them. He spent most of his time daydreaming about food. A few days before the April inspection William's room was still a mess since he had done nothing but daydream. To provide William with an incentive, the nurse promised him some gingerbread from the cookie shop if he scrubbed his room and put it in order once and for all. William was overjoyed. But there was no longer enough time for him to put it in order. As a result, he did not pass the inspection and did not get any gingerbread. William sulked all day and slammed his door so hard the plaster cracked, but he still didn't get any gingerbread.
Now consider two more stories:
Story A: Karen always did poorly in high school so she despised it. But she loved vacations. She spent most of her time dreaming about going to Hawaii. Not long before the end of her fourth year Karen was not doing at all well in her classes because she had spent all her time daydreaming. To motivate her, Karen's father promised her that if she did well enough during the next few weeks to graduate from high school he would pay for her trip to Hawaii. This made Karen extremely happy. But she was too far behind in her classes. Consequently she failed too many and did not go to Hawaii.
Story B: Karen disliked high school so she always had trouble getting passing grades in her classes. Vacations were her favorite part of the school year. Not long before the end of her fourth year of school, Karen was not doing well at all because she despised school. To motivate her, Karen's father promised her a trip to Hawaii if she would just pass enough of her classes to graduate. This made Karen extremely happy. During the last few weeks of school she spent most of her time dreaming about Hawaii and preparing for her trip there. Consequently, she failed to graduate and she did not get to go.
Which of Story A and Story B is a better analogy to Story 1? Is the best answer Story A, Story B, or both are equally analogous?
The best answer is Story A. Both Story A and Story 1 share the following similarities:
1. The main characters (William and Karen) are not successful in their respective situations (room inspection and graduating high school) due to spending too much time daydreaming.
2. An incentive is provided to motivate the main characters (gingerbread for William and a trip to Hawaii for Karen).
3. Despite being motivated by the incentive, both characters fail to achieve the desired outcome because they are too far behind in their tasks (cleaning the room and catching up on classes).
Story B is less analogous to Story 1 because Karen spends her time daydreaming and preparing for the trip instead of focusing on her classes, which leads to her failure to graduate. The cause of her failure is different from that in Story 1, where William fails due to the lack of time to complete the task.
## S5 Analogical problem-solving results
The following sections contain all of the materials used to evaluate analogical reasoning in natural language problemsolving. GPT-3's responses are presented in bold text.
## S5.1 Solution to radiation problem in isolation
The following contains GPT-3's proposed solution to the radiation problem when presented in isolation.
Target problem:
Suppose you were a doctor faced with a patient suffering from a malignant stomach tumor. The tumor is inoperable, and the patient will die unless it is destroyed. There is a kind of ray that will destroy the tumor if the rays reach it at a high intensity. But unfortunately, such high-intensity rays will also destroy the healthy tissue they pass through on the way to the tumor. At lower intensities the rays will not damage the healthy tissue, but neither will they remove the tumor. How can the doctor use rays to destroy the tumor, while at the same time sparing the healthy tissue?
Solution:
The doctor can use a technique called brachytherapy, which involves placing a radioactive source directly into or near the tumor. This allows the doctor to deliver a high dose of radiation directly to the tumor, while sparing the healthy tissue around it. The radiation source can be placed in the tumor using a catheter, or it can be placed in a sealed container and placed near the tumor. This technique is often used to treat tumors that are inoperable or too close to vital organs to be safely removed.
## S5.2 Solution to radiation problem following general story
The following contains GPT-3's proposed solution to the radiation problem when first presented with the general story, along with responses to followup questions that explicitly probe the analogy between the source story and the target problem.
Source story:
An evil dictator controlled a fortress situated in the center of a small country. Many roads radiated out from the fortress like spokes on a wheel. A general raised an army at the border, vowing to capture the fortress and overthrow the dictator. The general was about to send his entire army down one road to capture the fortress, when he learned that the dictator had mined each road so that although small groups could still pass, a large army would set off an explosion. The general then had a clever idea: he divided his army into small groups and dispatched each group to the head of a different road. Upon his signal, each group charged down a different road. All the groups passed safely to the fortress, where the entire army attacked the fortress in full strength. In this way the general captured the fortress and overthrew the dictator.
Target problem:
Suppose you were a doctor faced with a patient suffering from a malignant stomach tumor. The tumor is inoperable, and the patient will die unless it is destroyed. There is a kind of ray that will destroy the tumor if the rays reach it at a high intensity. But unfortunately, such high-intensity rays will also destroy the healthy tissue they pass through on the way to the tumor. At lower intensities the rays will not damage the healthy tissue, but neither will they remove the tumor. How can the doctor use rays to destroy the tumor, while at the same time sparing the healthy tissue?
Solution:
The doctor could divide the rays into small groups and send each group down a different path to the tumor. By using a lower intensity for each group, the rays will not damage the healthy tissue, but when they all reach the tumor at the same time, the combined intensity
will be enough to destroy it. The doctor could also use imaging technology to map out the paths of the rays and ensure that they all reach the tumor at the same time.
Can you identify an analogy between the source story and the target problem?
The analogy between the source story and the target problem is that both involve a situation in which a large force needs to be divided into smaller groups in order to achieve a desired outcome. In the source story, the general divides his army into small groups in order to avoid setting off the mines and capture the fortress. In the target problem, the doctor needs to divide the rays into smaller groups in order to destroy the tumor without damaging the healthy tissue.
Who or what in the target problem is most analogous to the general in the source story?
The doctor in the target problem is most analogous to the general in the source story. Both are faced with a difficult situation and must come up with a clever solution in order to achieve their desired outcome. In the source story, the general divides his army into small groups in order to avoid setting off the mines and capture the fortress. In the target problem, the doctor needs to divide the rays into smaller groups in order to destroy the tumor without damaging the healthy tissue.
Who or what in the target problem is most analogous to the evil dictator in the source story?
The malignant stomach tumor in the target problem is most analogous to the evil dictator in the source story. Both are obstacles that must be overcome in order to achieve the desired outcome. In the source story, the evil dictator must be overthrown in order for the general to capture the fortress. In the target problem, the malignant stomach tumor must be destroyed in order for the patient to be saved.
Who or what in the target problem is most analogous to the army in the source story?
The rays in the target problem are most analogous to the army in the source story. Both are forces that must be divided into smaller groups in order to achieve the desired outcome. In the source story, the army is divided into small groups in order to avoid setting off the mines and capture the fortress. In the target problem, the rays must be divided into smaller groups in order to destroy the tumor without damaging the healthy tissue.
## S5.3 Solution to radiation problem following alternative source analogs
We also tested GPT-3 using alternative materials for the source analog taken from a different study. 46 These stories each involve a solution based on convergence, but differ from the general story used in the original study of Gick and Holyoak. 22 GPT-3 was also able to identify the convergence solution to the radiation problem when presented with these alternative source analogs. The following contains GPT-3's response to the radiation problem when first presented with an analogous story about a student in a physics lab who uses lasers to repair a lightbulb.
Source story:
In a physics lab at a major university, a very expensive lightbulb which would emit precisely controlled quantities of light was being used in some experiments. Ruth was the research assistant responsible for operating the sensitive lightbulb. One morning she came into the lab and found to her dismay that the lightbulb no longer worked. She realized that she had forgotten to turn it off the previous night. As a result the lightbulb overheated, and the filament inside the bulb had broken into two parts. The surrounding glass bulb was completely sealed, so there was no way to open it. Ruth knew that the lightbulb could be repaired if a brief, high-intensity laser beam could be used to fuse the two parts of
the filament into one. Furthermore, the lab had the necessary equipment to do the job. However, a high-intensity laser beam would also break the fragile glass surrounding the filament. At lower intensities the laser would not break the glass, but neither would it fuse the filament. So it seemed that the lightbulb could not be repaired, and a costly replacement would be required. Ruth was about to give up when she had an idea. She placed several lasers in a circle around the lightbulb, and administered low-intensity laser beams from several directions all at once. The beams all converged on the filament, where their combined effect was enough to fuse it. Since each spot on the surrounding glass received only a lowintensity beam from one laser, the glass was left intact. Ruth was greatly relieved that the lightbulb was repaired, and she then went on to successfully complete the experiment.
## Target problem:
Suppose you were a doctor faced with a patient suffering from a malignant stomach tumor. The tumor is inoperable, and the patient will die unless it is destroyed. There is a kind of ray that will destroy the tumor if the rays reach it at a high intensity. But unfortunately, such high-intensity rays will also destroy the healthy tissue they pass through on the way to the tumor. At lower intensities the rays will not damage the healthy tissue, but neither will they remove the tumor. How can the doctor use rays to destroy the tumor, while at the same time sparing the healthy tissue?
## Solution:
The doctor could use multiple low-intensity rays from several directions to converge on the tumor. The combined effect of the rays would be enough to destroy the tumor, while each spot on the healthy tissue would only receive a low-intensity beam from one ray, thus leaving it intact.
The following contains GPT-3's response to the radiation problem when first presented with a different story involving ultrasound instead of lasers. GPT-3 was still able to use this story to identify the convergence solution, despite the fact that it was designed to have lower surface similarity to the radiation problem.
## Source story:
In a physics lab at a major university, a very expensive lightbulb which would emit precisely controlled quantities of light was being used in some experiments. Ruth was the research assistant responsible for operating the sensitive lightbulb. One morning she came into the lab and found to her dismay that the lightbulb no longer worked. She realized that she had forgotten to turn it off the previous night. As a result the lightbulb overheated, and the two wires in the filament inside the bulb fused together. The surrounding glass bulb was completely sealed, so there was no way to open it. Ruth knew that the lightbulb could be repaired if a brief, high-intensity ultrasound wave could be used to jar apart the fused parts. Furthermore, the lab had the necessary equipment to do the job. However, a high-intensity ultrasound wave would also break the fragile glass surrounding the filament. At lower intensities the ultrasound wave would not break the glass, but neither would it jar apart the fused parts. So it seemed that the lightbulb could not be repaired, and a costly replacement would be required. Ruth was about to give up when she had an idea. She placed several ultrasound machines in a circle around the lightbulb, and administered low-intensity ultrasound waves from several directions all at once. The waves all converged on the filament, where their combined effect was enough to jar apart the fused parts. Since each spot on the surrounding glass received only a low-intensity wave from one ultrasound machine, the glass was left intact. Ruth was greatly relieved that the lightbulb was repaired, and she then went on to successfully complete the experiment.
## Target problem:
Suppose you were a doctor faced with a patient suffering from a malignant stomach tumor. The tumor is inoperable, and the patient will die unless it is destroyed. There is a kind of ray that will destroy
the tumor if the rays reach it at a high intensity. But unfortunately, such high-intensity rays will also destroy the healthy tissue they pass through on the way to the tumor. At lower intensities the rays will not damage the healthy tissue, but neither will they remove the tumor. How can the doctor use rays to destroy the tumor, while at the same time sparing the healthy tissue?
Solution:
The doctor could use multiple low-intensity rays from several directions all at once. The rays would converge on the tumor, where their combined effect would be enough to destroy it. Since each spot on the healthy tissue would receive only a low-intensity ray from one source, the healthy tissue would be left intact.
## S5.4 Solution to radiation problem with distracting stories
The following contains GPT-3's proposed solution to the radiation problem when first presented with both the general story and two distracting (i.e., non-analogous) stories. GPT-3 did not identify the convergence solution in this case, instead proposing the same solution as it did when presented with the radiation problem in isolation.
Story #1 - The Wine Merchants:
One day a rich man found that his wine cellar was empty. So he sent out messengers to announce a generous offer. The first person to bring the rich man a barrel of wine would be given a brick of solid gold. However, the offer would expire at sundown. Two wine merchants heard the news. Each had a horse-drawn cart loaded with large barrels of wine. They both set out for the duke's palace at once. An hour before sundown they came to a place where the bridge had been washed out by a raging river. The first merchant drove his horses and cart into the flood in a desperate attempt to reach the other side. But the horses were already exhausted and could not fight the current. The cart overturned, and the horses, wine, and driver were washed away. The second merchant tried a different tactic. He poured the wine out of all but one of his barrels, and lashed them together to form a raft; then he loaded the one full barrel, a horse, and himself on top. He set the raft adrift and floated downstream. In a few minutes the raft came to rest on the shore in front of the town where the rich man lived. The merchant disembarked, loaded the wine barrel on the horse, and led it to the rich man's house. He arrived just as the sun was setting, and collected the gold brick as a reward for his efforts.
Story #2 - The General:
An evil dictator controlled a fortress situated in the center of a small country. Many roads radiated out from the fortress like spokes on a wheel. A general raised an army at the border, vowing to capture the fortress and overthrow the dictator. The general was about to send his entire army down one road to capture the fortress, when he learned that the dictator had mined each road so that although small groups could still pass, a large army would set off an explosion. The general then had a clever idea: he divided his army into small groups and dispatched each group to the head of a different road. Upon his signal, each group charged down a different road. All the groups passed safely to the fortress, where the entire army attacked the fortress in full strength. In this way the general captured the fortress and overthrew the dictator.
Story #3 - The Identical Twins:
Once there were identical twins who were continually playing pranks on their family, friends, and teachers. The annual school picnic was always a big event for the twins. There were races and other athletic events in which the twins won lots of prizes. One year a new student arrived who was a star runner. The twins wanted to win the main event: the 2-mile race through the woods behind the school. So they secretly devised a plan which would enable them to outdo the newcomer. The day of the race arrived. Each runner was to pick his own path through the woods to a clearing, where a teacher stood posted to
determine the winner. One twin entered the race, while the other excused himself on the grounds that he had hurt his leg in an earlier broadjumping event. The race began and the students rushed into the woods. The twin rushed into the woods and waited until the others had passed out of sight. Then he went back to the school using a path hidden from the picnic area. Shortly after, the other twin, who had been hiding behind a rock near the finish line of the race, burst out and ran into the clearing ahead of the other runners. The teacher named him the winner and marveled at the speed of his running. Next year the twins switched places and thereafter maintained their status on this event.
Target problem:
Suppose you were a doctor faced with a patient suffering from a malignant stomach tumor. The tumor is inoperable, and the patient will die unless it is destroyed. There is a kind of ray that will destroy the tumor if the rays reach it at a high intensity. But unfortunately, such high-intensity rays will also destroy the healthy tissue they pass through on the way to the tumor. At lower intensities the rays will not damage the healthy tissue, but neither will they remove the tumor. How can the doctor use rays to destroy the tumor, while at the same time sparing the healthy tissue?
Solution:
The doctor can use a technique called brachytherapy. This involves placing a radioactive source directly into the tumor. The radiation from the source will be of a high enough intensity to destroy the tumor, but the radiation will not travel far enough to damage the healthy tissue. The doctor can also use a technique called stereotactic radiotherapy. This involves using a machine to deliver a high-intensity beam of radiation to the tumor from outside the body. The beam is focused on the tumor, so that the healthy tissue is spared.
## S5.5 Solution to radiation problem with distracting stories and prompt
When presented with distracting stories, GPT-3 did not propose a solution to the radiation problem based on the analogy with the general story. We therefore provided a prompt to encourage GPT-3 to consider the previously presented stories when generating its solution. This same prompt was used in the original human behavioral study, 22 where it was found to increase the likelihood that participants would identify the analogous convergence solution. The source stories and target problem were presented in the exact same manner, but the prompt used to elicit GPT3's solution was changed. When presented with this prompt, GPT-3 correctly identified the convergence solution, and specifically identified the general story as the relevant source.
Solution (in solving this problem you may find that one of the stories you read before will give you a hint for a solution of this problem):
The doctor could use the same tactic as the general in the second story. He could divide the rays into small groups and send each group down a different path to the tumor. The rays would be of low intensity, so they would not damage the healthy tissue, but when they all reached the tumor at the same time, the combined intensity would be enough to destroy it.
## S5.6 Solution to gumball problem in isolation
The following contains GPT-3's proposed solution to the gumball problem (from 23 ) when presented in isolation (enumeration is part of the response provided by GPT-3).
Target problem:
You are seated in front of a table. Two bowls are on the table, one within your reach and one farther away. The closer bowl contains a number of small gumballs, and the farther one is empty. Also on the table are an aluminum walking cane, a large rectangular sheet of heavy paper (posterboard), a hollow
cardboard tube long enough to reach the farther bowl, scissors, string, tape, paper clips, and rubber bands. Using the materials provided, how can you transfer the balls from the filled to the empty bowl without leaving your seat?
Solution:
1. Cut the posterboard into a long strip and tape it to the end of the cardboard tube.
2. Tie one end of the string to the end of the tube and the other end to the handle of the cane.
3. Place the tube in the bowl of gumballs and use the cane to pull the tube and gumballs towards you.
4. Once the tube is close enough, use the paper clips to secure the tube to the edge of the table.
5. Use the rubber bands to secure the tube to the cane.
6. Pull the cane towards you, using the string, and the gumballs will be transferred to the empty bowl.
## S5.7 Solution to gumball problem following magic staff story
The following contains GPT-3's proposed solution to the gumball problem when first presented with the 'magic staff story'. This story involves a magical genie who has moved from his old home (a bottle) to a new home (a different bottle), and wants to transport his collection of jewels between the bottles. To do so, he uses a magic staff to pull the new bottle over next to the old bottle and transfers the jewels by hand. When first presented with this story, children most often identify a solution to the gumball problem that involves using the aluminum walking cane to pull the empty bowl over next to the bowl with the gumballs and then transferring the gumballs by hand. GPT-3 was not able to discover this solution, but was able to identify the high-level analogy between the source story and the target problem (though not the specific analogy between the magic staff and the walking cane).
Source story:
Once upon a time there lived a magical genie. He was a very old, wise, and rich genie indeed. One day while he was polishing his home, which was actually a bottle, he decided he would like to find an even bigger and better home to live in. So he began searching far and wide for another bottle. Finally he found the perfect home. It was larger, prettier, and not too far away from his old bottle. The genie was very excited and began moving his belongings right away. But now the genie had a problem. He had a great many beautiful and very precious jewels in his old home. He had to somehow get all the jewels from his old bottle to the new bottle without dropping or losing a single jewel.
After thinking a bit, the genie came up with a wonderful idea. He began searching for his magic staff, or wand. He then commanded his staff to stretch itself from his old home to his new home. Next, the genie tugged and pulled on his magical staff until at last he pulled the new bottle right up next to his old bottle. At once, the genie began gathering his jewels together in his old home and simply dropped them carefully into his new home right next to him. When all his jewels were safely tucked away in his new home, the genie settled in happily. He invited his friend to come in and admire his new home. I'm sure you can still find the genie sitting in his new bigger and better bottle with all his jewels and smiling contentedly even today!
Target problem:
You are seated in front of a table. Two bowls are on the table, one within your reach and one farther away. The closer bowl contains a number of small gumballs, and the farther one is empty. Also on the
table are an aluminum walking cane, a large rectangular sheet of heavy paper (posterboard), a hollow cardboard tube long enough to reach the farther bowl, scissors, string, tape, paper clips, and rubber bands. Using the materials provided, how can you transfer the balls from the filled to the empty bowl without leaving your seat?
Solution:
Take the cardboard tube and cut it in half. Place one half of the tube in the bowl with the gumballs and the other half in the empty bowl. Secure the two halves together with the rubber bands. Take the posterboard and cut it into strips. Tape the strips together to form a long rope. Tie one end of the rope to the handle of the walking cane and the other end to the rubber bands. Place the walking cane on the table and hold it in place. Gently pull the cane towards you, and the gumballs will travel through the tube and into the empty bowl. When all the gumballs have been transferred, remove the tube and rope and enjoy your gumballs!
Can you identify an analogy between the source story and the target problem?
The source story and the target problem both involve transferring something from one place to another. In the source story, the genie is transferring his jewels from one bottle to another, and in the target problem, the gumballs are being transferred from one bowl to another. In both cases, the characters must use creative problem-solving to figure out how to move the items without losing any.
Does this analogy suggest a solution to the target problem?
Yes, the analogy suggests a solution to the target problem. The genie used a magical staff to stretch from one bottle to the other, and the target problem suggests using a rope and a walking cane to transfer the gumballs from one bowl to the other.
Who or what in the target problem is most analogous to the genie's old home in the source story?
The bowl with the gumballs is most analogous to the genie's old home in the source story. Both contain something valuable that needs to be transferred to a new location.
Who or what in the target problem is most analogous to the genie's new home in the source story?
The empty bowl is most analogous to the genie's new home in the source story. Both are the destination for something valuable that needs to be transferred from an old location.
Who or what in the target problem is most analogous to the jewels in the source story?
The gumballs are most analogous to the jewels in the source story. Both are valuable items that need to be transferred from one place to another.
Who or what in the target problem is most analogous to the magic staff in the source story?
The rope and walking cane are most analogous to the magic staff in the source story. Both are used to transfer something valuable from one place to another.
## S5.8 Solution to gumball problem following magic carpet story
The following contains GPT-3's proposed solution to the gumball problem when first presented with the 'magic carpet story'. This story has the same setup as the magic staff story - a magical genie wants to transport his jewels
from his old bottle to his new bottle - but the genie's solution in this story is to roll up his magic carpet and use it as a bridge between the two bottles. When first presented with this story, children most often identify a solution to the gumball problem that involves rolling up the posterboard and rolling the gumballs through it into the empty bowl. It is interesting to note that children do this despite having access to an already formed cardboard tube, likely due to the analogy between the rolled up posterboard and the rolled up magic carpet. GPT-3 identified essentially this same solution, but added some unnecessary steps, including first cutting the posterboard in half. GPT-3 correctly identified the analogy between the source story and the target problem.
Source story:
Once upon a time there lived a magical genie. He was a very old, wise, and rich genie indeed. One day while he was polishing his home, which was actually a bottle, he decided he would like to find an even bigger and better home to live in. So he began searching far and wide for another bottle. Finally he found the perfect home. It was larger, prettier, and not too far away from his old bottle. The genie was very excited and began moving his belongings right away. But now the genie had a problem. He had a great many beautiful and very precious jewels in his old home. He had to somehow get all the jewels from his old bottle to the new bottle without dropping or losing a single jewel.
After thinking a bit, the genie came up with a wonderful idea. He searched for his magic carpet. Then he commanded it to roll itself up into a long hollow tube. Next the genie commanded his flying carpet to place one end at his old home and the other end at his new home so that it formed a sort of hollow bridge between the two bottles. Then, the genie very carefully took one jewel from inside his old home and placed it into the opening of his carpet. At once, the jewel began tumbling and rolling through the carpet tube until it reached his new home and plopped safely inside. The genie grinned happily and began rolling all his jewels through the carpet into his new home. In fact, I'm sure you can still find him sitting in his new, bigger and better bottle with all his jewels and smiling contentedly even today!
Target problem:
You are seated in front of a table. Two bowls are on the table, one within your reach and one farther away. The closer bowl contains a number of small gumballs, and the farther one is empty. Also on the table are an aluminum walking cane, a large rectangular sheet of heavy paper (posterboard), a hollow cardboard tube long enough to reach the farther bowl, scissors, string, tape, paper clips, and rubber bands. Using the materials provided, how can you transfer the balls from the filled to the empty bowl without leaving your seat?
Solution:
Take the posterboard and cut it into two pieces. Take one of the pieces and roll it up into a tube. Secure the tube with the rubber bands, tape, and paper clips. Place one end of the tube into the bowl with the gumballs and the other end into the empty bowl. Then, use the walking cane to push the gumballs through the tube until they reach the empty bowl.
Can you identify an analogy between the source story and the target problem?
The source story and the target problem both involve transferring items from one place to another. In the source story, the genie transfers jewels from one bottle to another using a flying carpet. In the target problem, the person transfers gumballs from one bowl to another using a posterboard tube. In both cases, the items are transferred without leaving the original location.
Does this analogy suggest a solution to the target problem?
Yes, the analogy suggests a solution to the target problem. The solution is to use a posterboard tube to transfer the gumballs from one bowl to the other. This is similar to how the genie used a flying carpet to transfer the jewels from one bottle to the other. Who or what in the target problem is most analogous to the genie's old home in the source story? The bowl with the gumballs is most analogous to the genie's old home in the source story. Both contain items that need to be transferred to a new location. Who or what in the target problem is most analogous to the genie's new home in the source story? The empty bowl is most analogous to the genie's new home in the source story. Both are the destination for items that need to be transferred from an old location. Who or what in the target problem is most analogous to the jewels in the source story? The gumballs in the target problem are most analogous to the jewels in the source story. Both are items that need to be transferred from one location to another.
Who or what in the target problem is most analogous to the magic staff in the source story?
The posterboard tube in the target problem is most analogous to the magic carpet in the source story. Both are used to transfer items from one location to another.