## Heatmap: Zero-shot - Core Generalization - o3-mini ### Overview The image is a heatmap visualizing the accuracy (%) of a model across different "Types" and "Lengths". The color intensity represents the accuracy, with darker blue indicating higher accuracy and lighter blue indicating lower accuracy. The heatmap shows how the model's performance varies depending on the type and length of the input. ### Components/Axes * **Title:** Zero-shot - Core Generalization - o3-mini * **Y-axis:** "Type" with categories 1, 2, 3, 4, 5, 6, and 7. * **X-axis:** "Length" ranging from 0 to 19. * **Color Scale (Legend):** "Accuracy (%)" ranging from 0 to 100, with darker blues representing higher accuracy and lighter blues representing lower accuracy. ### Detailed Analysis The heatmap displays accuracy values for each combination of "Type" and "Length". The values are explicitly written on each cell of the heatmap. * **Type 1:** High accuracy (80-100%) for lengths 0-9, then drops off. * Length 0: 99.0% * Length 1: 99.0% * Length 2: 95.0% * Length 3: 93.0% * Length 4: 89.0% * Length 5: 87.0% * Length 6: 80.0% * Length 7: 84.0% * Length 8: 75.0% * Length 9: 79.0% * **Type 2:** Consistently high accuracy (94-100%) across lengths 0-11. * Length 0: 99.0% * Length 1: 100.0% * Length 2: 100.0% * Length 3: 98.0% * Length 4: 97.0% * Length 5: 97.0% * Length 6: 99.0% * Length 7: 98.0% * Length 8: 96.0% * Length 9: 94.0% * **Type 3:** Lower accuracy (9-58%) overall, with some increase in accuracy between lengths 0-5, then decreases. * Length 0: 9.0% * Length 1: 32.0% * Length 2: 38.0% * Length 3: 51.0% * Length 4: 53.0% * Length 5: 58.0% * Length 6: 43.0% * Length 7: 52.0% * Length 8: 52.0% * Length 9: 51.0% * Length 10: 43.0% * Length 11: 52.0% * Length 12: 43.0% * Length 13: 44.0% * Length 14: 39.0% * Length 15: 30.0% * Length 16: 29.0% * Length 17: 34.0% * Length 18: 32.0% * Length 19: 30.0% * **Type 4:** Low to moderate accuracy (24-42%), with a peak around length 2. * Length 1: 24.0% * Length 2: 36.0% * Length 3: 40.0% * Length 4: 32.0% * Length 5: 34.0% * Length 6: 29.0% * Length 7: 26.0% * Length 8: 36.0% * Length 9: 34.0% * Length 10: 36.0% * Length 11: 42.0% * **Type 5:** Moderate accuracy (30-75%), with higher accuracy for lengths 17-19. * Length 7: 66.0% * Length 8: 56.0% * Length 9: 57.0% * Length 10: 55.0% * Length 11: 50.0% * Length 12: 41.0% * Length 13: 50.0% * Length 14: 44.0% * Length 15: 34.0% * Length 16: 30.0% * Length 17: 75.0% * Length 18: 66.0% * Length 19: 73.0% * **Type 6:** High accuracy (70-99%) across lengths 0-16, then drops off. * Length 0: 96.0% * Length 1: 98.0% * Length 2: 98.0% * Length 3: 97.0% * Length 4: 93.0% * Length 5: 95.0% * Length 6: 88.0% * Length 7: 99.0% * Length 8: 93.0% * Length 9: 85.0% * Length 10: 83.0% * Length 11: 86.0% * Length 12: 78.0% * Length 13: 82.0% * Length 14: 70.0% * Length 15: 82.0% * Length 16: 74.0% * Length 17: 75.0% * Length 18: 72.0% * **Type 7:** High accuracy (70-99%) across lengths 0-13, then drops off. * Length 0: 98.0% * Length 1: 98.0% * Length 2: 99.0% * Length 3: 94.0% * Length 4: 92.0% * Length 5: 86.0% * Length 6: 89.0% * Length 7: 87.0% * Length 8: 78.0% * Length 9: 87.0% * Length 10: 75.0% * Length 11: 83.0% * Length 12: 75.0% * Length 13: 70.0% ### Key Observations * Types 1, 2, 6, and 7 generally exhibit higher accuracy compared to Types 3, 4, and 5. * Accuracy tends to vary with length, with some types showing a decrease in accuracy as length increases. * Type 2 shows the most consistent high accuracy across all lengths tested. * Type 3 has the lowest accuracy overall. ### Interpretation The heatmap provides insights into the zero-shot core generalization performance of the model. The model performs well on certain types (1, 2, 6, 7) regardless of length, while its performance on other types (3, 4, 5) is more sensitive to the length of the input. This suggests that the model may have learned certain patterns or features that are more relevant to some types than others. The drop in accuracy for some types as length increases could indicate limitations in the model's ability to handle longer sequences or a mismatch between the training data and the longer sequences used for evaluation. The data suggests that the model's generalization ability is not uniform across all types and lengths, highlighting areas for potential improvement.

## Heatmap: Zero-shot - Core Generalization - o3-mini ### Overview This image presents a heatmap visualizing the accuracy of a model ("o3-mini") in a zero-shot core generalization task. The heatmap displays accuracy percentages based on two dimensions: "Type" and "Length". The "Type" dimension represents different categories (h-, 2-, m-, 4-, un-, o-, >), while "Length" represents the length of the input, ranging from 0 to 19. The color intensity of each cell corresponds to the accuracy percentage, with darker blues indicating higher accuracy and lighter shades indicating lower accuracy. ### Components/Axes * **Title:** "Zero-shot - Core Generalization - o3-mini" (Top-center) * **X-axis:** "Length" - ranging from 0 to 19, with integer values. (Bottom) * **Y-axis:** "Type" - with the following categories: * h- * 2- * m- * 4- * un- * o- * >- (Bottom-left) * **Color Scale/Legend:** A vertical color bar on the right side of the heatmap, representing accuracy percentages from 0% to 100%. (Right) ### Detailed Analysis The heatmap is a 7x20 grid. Each cell represents the accuracy for a specific combination of "Type" and "Length". The values are approximate, based on visual estimation. * **h- Type:** * Length 0: ~99.0% * Length 1: ~99.0% * Length 2: ~95.0% * Length 3: ~89.0% * Length 4: ~87.0% * Length 5: ~80.0% * Length 6: ~75.0% * Length 7: ~79.0% * **2- Type:** * Length 0: ~99.3% * Length 1: ~100.0% * Length 2: ~98.0% * Length 3: ~97.0% * Length 4: ~98.0% * Length 5: ~96.0% * Length 6: ~94.0% * **m- Type:** * Length 0: ~32.0% * Length 1: ~38.0% * Length 2: ~51.0% * Length 3: ~58.0% * Length 4: ~43.0% * Length 5: ~52.0% * Length 6: ~41.0% * Length 7: ~52.0% * **4- Type:** * Length 0: ~24.0% * Length 1: ~36.0% * Length 2: ~40.0% * Length 3: ~34.0% * Length 4: ~26.0% * Length 5: ~36.0% * Length 6: ~36.0% * Length 7: ~42.0% * **un- Type:** * Length 6: ~66.0% * Length 7: ~56.0% * Length 8: ~57.0% * Length 9: ~50.0% * Length 10: ~41.0% * Length 11: ~44.0% * Length 12: ~30.0% * Length 13: ~75.0% * Length 14: ~66.0% * Length 15: ~73.0% * **o- Type:** * Length 0: ~96.0% * Length 1: ~98.0% * Length 2: ~97.0% * Length 3: ~95.0% * Length 4: ~88.0% * Length 5: ~89.0% * Length 6: ~83.0% * Length 7: ~85.0% * **>- Type:** * Length 0: ~98.0% * Length 1: ~94.0% * Length 2: ~92.0% * Length 3: ~86.0% * Length 4: ~87.0% * Length 5: ~78.0% * Length 6: ~83.0% * Length 7: ~75.0% **Trends:** * For "h-" and "2-" types, accuracy is generally high (above 80%) and tends to decrease slightly as length increases. * "m-" and "4-" types exhibit significantly lower accuracy, generally below 60%, with some fluctuations. * "un-" type shows a complex pattern, with accuracy initially decreasing and then increasing again at higher lengths. * "o-" and ">-" types show high accuracy, similar to "h-" and "2-", but with more noticeable decreases at higher lengths. ### Key Observations * The "h-" and "2-" types consistently demonstrate the highest accuracy across all lengths. * The "m-" and "4-" types have the lowest accuracy, indicating the model struggles with these categories. * The "un-" type shows a non-monotonic relationship between length and accuracy, suggesting a more complex interaction. * Accuracy generally decreases as the length of the input increases, but the rate of decrease varies significantly between types. ### Interpretation The heatmap reveals that the "o3-mini" model performs well on certain types of inputs ("h-" and "2-") in a zero-shot setting, achieving high accuracy even with increasing length. However, it struggles with other types ("m-" and "4-"), indicating potential biases or limitations in its generalization capabilities. The varying trends across different types suggest that the model's performance is sensitive to the specific characteristics of the input data. The non-monotonic behavior of the "un-" type warrants further investigation to understand the underlying factors influencing its accuracy. This data suggests that the model is not universally capable of generalizing to all core types without any prior training. The performance differences between types highlight the importance of considering the diversity of input data when evaluating and deploying zero-shot learning models. The decrease in accuracy with increasing length could be due to the model's limited capacity to process longer sequences or the increased difficulty of maintaining context over longer inputs.

## Heatmap: Zero-shot Core Generalization Performance of o3-mini Model ### Overview This image is a heatmap titled "Zero-shot - Core Generalization - o3-mini". It visualizes the accuracy percentage of an AI model (o3-mini) across seven different task "Types" (y-axis) and varying input "Lengths" (x-axis). The chart uses a blue color gradient to represent accuracy, with darker blue indicating higher accuracy. The data appears to be from a technical evaluation of the model's zero-shot generalization capabilities. ### Components/Axes * **Title:** "Zero-shot - Core Generalization - o3-mini" (Top center) * **Y-Axis (Vertical):** Labeled "Type". Contains 7 discrete categories numbered 1 through 7. * **X-Axis (Horizontal):** Labeled "Length". Contains 20 discrete categories numbered 0 through 19. * **Color Bar/Legend:** Located on the right side. Labeled "Accuracy (%)". It is a vertical gradient bar ranging from 0 (lightest blue/white) to 100 (darkest blue). Key markers are at 0, 20, 40, 60, 80, and 100. * **Data Cells:** Each cell in the grid contains a numerical value representing the accuracy percentage for a specific Type-Length combination. The cell's background color corresponds to this value per the color bar. ### Detailed Analysis The following table reconstructs the accuracy data from the heatmap. Empty cells indicate no data was recorded for that Type-Length combination. | Type | Length 0 | Length 1 | Length 2 | Length 3 | Length 4 | Length 5 | Length 6 | Length 7 | Length 8 | Length 9 | Length 10 | Length 11 | Length 12 | Length 13 | Length 14 | Length 15 | Length 16 | Length 17 | Length 18 | Length 19 | | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | | **1** | 99.0 | 99.0 | 95.0 | 93.0 | 89.0 | 87.0 | 80.0 | 84.0 | 75.0 | 79.0 | | | | | | | | | | | | **2** | | 99.0 | 100.0 | 100.0 | 98.0 | 97.0 | 97.0 | 99.0 | 98.0 | 96.0 | 94.0 | | | | | | | | | | | **3** | 9.0 | 32.0 | 38.0 | 51.0 | 53.0 | 58.0 | 43.0 | 52.0 | 52.0 | 51.0 | 43.0 | 52.0 | 43.0 | 44.0 | 39.0 | 30.0 | 29.0 | 34.0 | 32.0 | 30.0 | | **4** | | 24.0 | 36.0 | 40.0 | 32.0 | 34.0 | 29.0 | 26.0 | 36.0 | 34.0 | 36.0 | 42.0 | | | | | | | | | | **5** | | | | | | | | 66.0 | 56.0 | 57.0 | 55.0 | 50.0 | 41.0 | 50.0 | 44.0 | 34.0 | 30.0 | 75.0 | 66.0 | 73.0 | | **6** | 96.0 | 98.0 | 98.0 | 97.0 | 93.0 | 95.0 | 88.0 | 99.0 | 93.0 | 85.0 | 83.0 | 86.0 | 78.0 | 82.0 | 70.0 | 82.0 | 74.0 | 75.0 | 72.0 | | | **7** | 98.0 | 98.0 | 99.0 | 94.0 | 92.0 | 86.0 | 89.0 | 87.0 | 78.0 | 87.0 | 75.0 | 83.0 | 75.0 | 70.0 | | | | | | | **Trend Verification by Type:** * **Type 1:** Shows a gradual downward trend in accuracy as length increases, starting at 99% (Length 0) and ending at 79% (Length 9). * **Type 2:** Maintains exceptionally high accuracy (94-100%) across its measured lengths (1-10), with no significant downward trend. * **Type 3:** Exhibits a complex trend. Accuracy starts very low (9% at Length 0), rises to a peak of 58% at Length 5, then generally declines with fluctuations, ending at 30% (Length 19). * **Type 4:** Shows moderate, relatively stable accuracy in the 24-42% range across lengths 1-11, with no strong directional trend. * **Type 5:** Displays a U-shaped or volatile trend. Accuracy is higher at the start (66% at Length 7) and end (73-75% at Lengths 17-19) of its range, with a dip in the middle lengths (as low as 30% at Length 16). * **Type 6:** Maintains high accuracy (mostly 70-99%) across a wide range of lengths (0-18), with a slight overall decreasing trend. * **Type 7:** Similar to Type 6, shows high accuracy (70-99%) for lengths 0-13, with a slight downward trend as length increases. ### Key Observations 1. **Performance Disparity:** There is a stark contrast in performance between task types. Types 1, 2, 6, and 7 consistently achieve high accuracy (often >80%), while Types 3 and 4 struggle, with accuracies frequently below 50%. 2. **Length Sensitivity:** The impact of increasing "Length" varies dramatically by type. Types 1, 6, and 7 show a mild negative correlation. Type 3 is highly sensitive, with performance peaking at mid-lengths. Types 2 and 4 are relatively insensitive to length within their measured ranges. 3. **Data Coverage:** The evaluation is not uniform. Some types (e.g., Type 3) are tested across all lengths (0-19), while others have limited ranges (e.g., Type 1 only up to Length 9). This suggests the tasks or their applicable lengths differ. 4. **Outliers:** The 9.0% accuracy for Type 3 at Length 0 is a significant low outlier. The 100.0% accuracy for Type 2 at Lengths 2 and 3 represents perfect performance. ### Interpretation This heatmap provides a diagnostic view of the o3-mini model's zero-shot reasoning capabilities. The "Type" axis likely represents different categories of logical or cognitive tasks (e.g., arithmetic, spatial reasoning, syllogisms), while "Length" probably corresponds to problem complexity, such as the number of steps, variables, or tokens in the input. The data suggests the model has robust, length-invariant performance on certain core task types (2, 6, 7), indicating strong foundational generalization for those domains. In contrast, its poor and variable performance on Type 3 reveals a specific weakness, possibly in a task requiring sequential or compositional reasoning where performance degrades with problem scale. The U-shape in Type 5 is intriguing, potentially indicating that the model uses different strategies for short vs. long problems within that category, or that the task distribution has distinct clusters. For a developer or researcher, this chart is crucial for identifying which capabilities are reliable and which require further training or architectural improvement. It moves beyond a single accuracy score to show *where* and *how* the model's generalization breaks down.

# Technical Document Extraction: Zero-shot - Core Generalization - o3-mini ## 1. Title - **Title**: Zero-shot - Core Generalization - o3-mini ## 2. Axes and Labels - **X-axis (Horizontal)**: - **Label**: Length - **Values**: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19 - **Y-axis (Vertical)**: - **Label**: Type - **Values**: 1, 2, 3, 4, 5, 6, 7 ## 3. Color Scale - **Legend**: Accuracy (%) - **Range**: 0% (lightest blue) to 100% (darkest blue) - **Placement**: Right side of the heatmap ## 4. Data Table ### Heatmap Values (Type vs. Length) | Type \ Length | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | |---------------|------|------|------|------|------|------|------|------|------|------|------|------|------|------|------|------|------|------|------|------| | **1** | 99.0 | 99.0 | 95.0 | 93.0 | 89.0 | 87.0 | 80.0 | 84.0 | 75.0 | 79.0 | - | - | - | - | - | - | - | - | - | - | | **2** | - | 99.0 | 100.0| 97.0 | 98.0 | 97.0 | 97.0 | 99.0 | 98.0 | 96.0 | 94.0 | - | - | - | - | - | - | - | - | - | | **3** | 9.0 | 32.0 | 38.0 | 51.0 | 53.0 | 58.0 | 43.0 | 52.0 | 52.0 | 51.0 | 43.0 | 52.0 | 43.0 | 44.0 | 39.0 | 30.0 | 29.0 | 34.0 | 32.0 | 30.0 | | **4** | 24.0 | 36.0 | 40.0 | 32.0 | 34.0 | 29.0 | 26.0 | 36.0 | 34.0 | 36.0 | 42.0 | - | - | - | - | - | - | - | - | - | | **5** | - | - | - | - | - | - | 66.0 | 56.0 | 57.0 | 55.0 | 50.0 | 41.0 | 50.0 | 44.0 | 34.0 | 30.0 | 75.0 | 66.0 | 73.0 | | **6** | 96.0 | 98.0 | 98.0 | 97.0 | 93.0 | 95.0 | 88.0 | 99.0 | 93.0 | 85.0 | 83.0 | 86.0 | 78.0 | 82.0 | 70.0 | 82.0 | 74.0 | 75.0 | 72.0 | - | | **7** | 98.0 | 98.0 | 99.0 | 94.0 | 92.0 | 86.0 | 89.0 | 87.0 | 78.0 | 87.0 | 75.0 | 83.0 | 75.0 | 70.0 | - | - | - | - | - | - | ## 5. Key Trends 1. **Type 1**: - High accuracy (99-79%) across all lengths. - Gradual decline with increasing length. 2. **Type 2**: - Consistently high accuracy (94-100%). - Peaks at Length 2 (100%) and Length 3 (97%). 3. **Type 3**: - Low accuracy (9-58%) across all lengths. - Peaks at Length 5 (58%) and Length 6 (43%). 4. **Type 4**: - Moderate accuracy (24-42%) across all lengths. - Peaks at Length 10 (42%). 5. **Type 5**: - Moderate accuracy (30-75%) across lengths 7-19. - Peaks at Length 17 (75%). 6. **Type 6**: - High accuracy (70-99%) across lengths 0-16. - Peaks at Length 0 (96%) and Length 7 (99%). 7. **Type 7**: - High accuracy (70-99%) across lengths 0-13. - Peaks at Length 0 (98%) and Length 2 (99%). ## 6. Spatial Grounding - **Legend**: Located on the right side of the heatmap. - **Data Points**: Numerical values embedded in cells match the color intensity of the legend. ## 7. Trend Verification - **Type 1**: Slopes downward from 99% (Length 0) to 79% (Length 9). - **Type 2**: Peaks at Length 2 (100%) and declines to 94% (Length 10). - **Type 3**: Slopes upward from 9% (Length 0) to 58% (Length 5), then declines. - **Type 4**: Slopes upward from 24% (Length 0) to 42% (Length 10). - **Type 5**: Slopes downward from 66% (Length 7) to 30% (Length 15), then rises to 75% (Length 17). - **Type 6**: Slopes downward from 96% (Length 0) to 74% (Length 16). - **Type 7**: Slopes downward from 98% (Length 0) to 70% (Length 13). ## 8. Component Isolation - **Header**: Title and axis labels. - **Main Chart**: Heatmap with embedded numerical values. - **Footer**: Color scale legend. ## 9. Language - **Primary Language**: English - **Translated Text**: None (all text is in English). ## 10. Missing Data - Dashes (`-`) indicate missing values for certain Type-Length combinations.