## Heatmap: Baseline - Core Generalization - Qwen-2.5 3B ### Overview The image is a heatmap visualizing the accuracy of a model (Qwen-2.5 3B) across different types and lengths. The heatmap uses a color gradient from light blue to dark blue to represent accuracy percentages, ranging from 0% to 100%. The x-axis represents the length, and the y-axis represents the type. ### Components/Axes * **Title:** Baseline - Core Generalization - Qwen-2.5 3B * **X-axis:** Length (ranging from 0 to 19) * **Y-axis:** Type (ranging from 1 to 7) * **Color Legend (right side):** Accuracy (%) * Darkest Blue: 100% * Lightest Blue: 0% * Intermediate markers: 80%, 60%, 40%, 20% ### Detailed Analysis The heatmap displays accuracy values for different "Types" (1-7) at varying "Lengths" (0-19). Each cell in the heatmap contains a numerical value representing the accuracy percentage. * **Type 1:** * Length 0: 99.3% * Length 1: 96.7% * Length 2: 98.3% * Length 3: 92.7% * Length 4: 87.7% * Length 5: 83.0% * Length 6: 82.0% * Length 7: 86.3% * Length 8: 86.0% * Length 9: 83.7% * **Type 2:** * Length 1: 100.0% * Length 2: 99.7% * Length 3: 99.7% * Length 4: 98.3% * Length 5: 98.0% * Length 6: 99.3% * Length 7: 98.7% * Length 8: 99.3% * Length 9: 97.3% * Length 10: 97.7% * **Type 3:** * Length 0: 99.7% * Length 1: 98.0% * Length 2: 94.0% * Length 3: 95.0% * Length 4: 95.7% * Length 5: 89.7% * Length 6: 86.0% * Length 7: 88.3% * Length 8: 90.3% * Length 9: 86.7% * Length 10: 86.0% * Length 11: 89.3% * Length 12: 89.3% * Length 13: 86.0% * Length 14: 90.0% * Length 15: 89.0% * Length 16: 90.0% * Length 17: 90.7% * Length 18: 90.0% * Length 19: 89.0% * **Type 4:** * Length 0: 98.3% * Length 1: 98.3% * Length 2: 91.0% * Length 3: 91.7% * Length 4: 92.0% * Length 5: 91.0% * Length 6: 92.0% * Length 7: 92.3% * Length 8: 92.7% * Length 9: 92.7% * Length 10: 90.7% * **Type 5:** * Length 7: 80.3% * Length 8: 84.3% * Length 9: 81.3% * Length 10: 87.3% * Length 11: 87.3% * Length 12: 85.7% * Length 13: 89.0% * Length 14: 90.0% * Length 15: 87.0% * Length 16: 85.0% * Length 17: 87.3% * Length 18: 86.0% * Length 19: 89.7% * **Type 6:** * Length 0: 100.0% * Length 1: 99.3% * Length 2: 99.7% * Length 3: 99.7% * Length 4: 99.0% * Length 5: 100.0% * Length 6: 98.3% * Length 7: 99.3% * Length 8: 99.3% * Length 9: 98.3% * Length 10: 98.3% * Length 11: 98.7% * Length 12: 98.0% * Length 13: 97.7% * Length 14: 97.7% * Length 15: 98.7% * Length 16: 98.7% * Length 17: 98.3% * Length 18: 97.7% * **Type 7:** * Length 0: 99.7% * Length 1: 99.7% * Length 2: 98.7% * Length 3: 98.0% * Length 4: 98.7% * Length 5: 96.0% * Length 6: 95.7% * Length 7: 95.0% * Length 8: 92.3% * Length 9: 91.0% * Length 10: 88.3% * Length 11: 84.7% * Length 12: 82.7% * Length 13: 87.3% ### Key Observations * Types 2 and 6 generally exhibit high accuracy across different lengths. * Type 5 has data only for lengths 7-19, with accuracy values generally lower than Types 2 and 6. * Types 1, 3, 4, and 7 show a trend of decreasing accuracy as the length increases, especially after length 5. ### Interpretation The heatmap provides insights into the performance of the Qwen-2.5 3B model under different conditions. The "Type" could represent different categories or tasks, and the "Length" could refer to the input sequence length. The data suggests that the model performs well on Types 2 and 6, maintaining high accuracy even with increasing length. However, for Types 1, 3, 4, and 7, the model's accuracy decreases as the input length increases, indicating potential challenges in handling longer sequences for these specific types. Type 5 appears to be a special case, with data only available for longer lengths, and its performance is generally lower compared to Types 2 and 6. This information can be used to further investigate the model's strengths and weaknesses and guide future improvements.

## Heatmap: Baseline - Core Generalization - Qwen-2.5B ### Overview This image presents a heatmap visualizing the accuracy of a model (Qwen-2.5B) across different sequence lengths and input types. The heatmap uses a color gradient to represent accuracy percentages, ranging from approximately 0% (white) to 100% (dark blue). The x-axis represents sequence length, and the y-axis represents input type. ### Components/Axes * **Title:** Baseline - Core Generalization - Qwen-2.5B (positioned at the top-center) * **X-axis Label:** Length (positioned at the bottom-center) * **X-axis Markers:** 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19 * **Y-axis Label:** Type (positioned at the left-center) * **Y-axis Markers:** h, ~, m, 4, u, o, > * **Color Scale/Legend:** A vertical color bar on the right side of the heatmap, representing accuracy percentages. * 0% is represented by white. * 100% is represented by dark blue. * Intermediate values are represented by shades of blue. * **Data Points:** Each cell in the heatmap represents the accuracy for a specific combination of input type and sequence length. The values are displayed as percentages within each cell. ### Detailed Analysis The heatmap displays accuracy values for 7 input types (h, ~, m, 4, u, o, >) across 20 sequence lengths (0 to 19). The values are as follows (approximated to one decimal place): * **Type 'h':** * Length 0: 96.7% * Length 1: 96.3% * Length 2: 95.3% * Length 3: 87.7% * Length 4: 83.0% * Length 5: 82.0% * Length 6: 86.0% * Length 7: 86.0% * Length 8: 83.7% * **Type '~':** * Length 0: 100.0% * Length 1: 99.7% * Length 2: 99.3% * Length 3: 98.7% * Length 4: 98.0% * Length 5: 99.3% * Length 6: 99.3% * Length 7: 97.3% * Length 8: 97.7% * **Type 'm':** * Length 0: 98.0% * Length 1: 94.0% * Length 2: 95.7% * Length 3: 89.7% * Length 4: 86.0% * Length 5: 88.3% * Length 6: 90.3% * Length 7: 86.7% * Length 8: 89.3% * Length 9: 86.0% * Length 10: 90.0% * Length 11: 89.0% * Length 12: 90.0% * Length 13: 90.0% * Length 14: 89.0% * Length 15: 90.0% * Length 16: 89.0% * Length 17: 86.0% * Length 18: 89.7% * **Type '4':** * Length 0: 98.3% * Length 1: 96.3% * Length 2: 91.0% * Length 3: 91.7% * Length 4: 92.0% * Length 5: 91.0% * Length 6: 92.3% * Length 7: 92.7% * Length 8: 90.7% * **Type 'u':** * Length 0: 80.3% * Length 1: 84.3% * Length 2: 81.3% * Length 3: 87.3% * Length 4: 85.7% * Length 5: 89.0% * Length 6: 90.0% * Length 7: 85.0% * Length 8: 87.3% * Length 9: 86.0% * Length 10: 89.7% * **Type 'o':** * Length 0: 100.0% * Length 1: 99.3% * Length 2: 99.7% * Length 3: 99.0% * Length 4: 100.0% * Length 5: 98.3% * Length 6: 99.3% * Length 7: 98.3% * Length 8: 98.7% * Length 9: 97.7% * Length 10: 98.7% * Length 11: 98.3% * Length 12: 97.7% * **Type '>':** * Length 0: 99.7% * Length 1: 98.7% * Length 2: 98.0% * Length 3: 96.0% * Length 4: 95.7% * Length 5: 95.0% * Length 6: 92.3% * Length 7: 91.0% * Length 8: 84.7% * Length 9: 82.7% * Length 10: 87.3% ### Key Observations * Generally, accuracy is high for shorter sequence lengths (0-5) across all input types. * Accuracy tends to decrease as sequence length increases, particularly for input types 'h', 'm', '4', 'u', and '>'. * Input type '~' consistently exhibits very high accuracy (close to 100%) across all sequence lengths. * Input type 'o' also shows consistently high accuracy, generally above 98%. * Input type 'u' has the lowest overall accuracy, especially for longer sequence lengths. * There is a noticeable dip in accuracy for type 'h' at length 3 and 4. ### Interpretation The heatmap demonstrates the performance of the Qwen-2.5B model on different input types and sequence lengths. The model performs best on shorter sequences and certain input types ('~' and 'o'). The decline in accuracy with increasing sequence length suggests that the model may struggle with long-range dependencies or have limitations in processing longer contexts. The variation in performance across input types indicates that the model is sensitive to the characteristics of the input data. The heatmap provides valuable insights into the model's strengths and weaknesses, which can inform further development and optimization efforts. The consistent high performance of type '~' suggests it may be a particularly well-suited input format for this model. The lower performance of type 'u' could indicate a need for more training data or architectural adjustments to better handle that type of input.

## Heatmap: Baseline - Core Generalization - Qwen-2.5 3B ### Overview This image is a heatmap visualizing the accuracy (in percentage) of a model named "Qwen-2.5 3B" on a "Core Generalization" task. The performance is broken down by two dimensions: "Type" (y-axis, categories 1-7) and "Length" (x-axis, values 0-19). The color intensity represents accuracy, with a scale from 0% (lightest) to 100% (darkest blue). The chart shows how model performance varies across different task types and sequence lengths. ### Components/Axes * **Title:** "Baseline - Core Generalization - Qwen-2.5 3B" (Top center). * **Y-Axis (Vertical):** Labeled "Type". Categories are numbered 1 through 7 from top to bottom. * **X-Axis (Horizontal):** Labeled "Length". Values range from 0 to 19 from left to right. * **Color Bar/Legend:** Located on the right side. It is a vertical gradient bar labeled "Accuracy (%)". The scale runs from 0 at the bottom to 100 at the top, with tick marks at 0, 20, 40, 60, 80, and 100. Darker blue corresponds to higher accuracy. * **Data Cells:** Each cell in the grid contains a numerical accuracy value. Cells with no data are left blank (white). ### Detailed Analysis The following table reconstructs the accuracy data for each Type across the available Lengths. Empty cells indicate no data for that Type-Length combination. | Type | Length 0 | Length 1 | Length 2 | Length 3 | Length 4 | Length 5 | Length 6 | Length 7 | Length 8 | Length 9 | Length 10 | Length 11 | Length 12 | Length 13 | Length 14 | Length 15 | Length 16 | Length 17 | Length 18 | Length 19 | | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | | **1** | 99.3 | 96.7 | 98.3 | 92.7 | 87.7 | 83.0 | 82.0 | 86.3 | 86.0 | 83.7 | | | | | | | | | | | | **2** | | 100.0 | 99.7 | 99.7 | 98.3 | 98.0 | 99.3 | 98.7 | 99.3 | 97.3 | 97.7 | | | | | | | | | | | **3** | 99.7 | 98.0 | 94.0 | 95.0 | 95.7 | 89.7 | 86.0 | 88.3 | 90.3 | 86.7 | 86.0 | 89.3 | 89.3 | 86.0 | 90.0 | 89.0 | 90.0 | 90.7 | 90.0 | 89.0 | | **4** | | 98.3 | 98.3 | 91.0 | 91.7 | 92.0 | 91.0 | 92.0 | 92.3 | 92.7 | 92.7 | 90.7 | | | | | | | | | | **5** | | | | | | | | 80.3 | 84.3 | 81.3 | 87.3 | 87.3 | 85.7 | 89.0 | 90.0 | 87.0 | 85.0 | 87.3 | 86.0 | 89.7 | | **6** | 100.0 | 99.3 | 99.7 | 99.7 | 99.0 | 100.0 | 98.3 | 99.3 | 99.3 | 98.3 | 98.3 | 98.7 | 98.0 | 97.7 | 97.7 | 98.7 | 98.7 | 98.3 | 97.7 | | | **7** | 99.7 | 99.7 | 98.7 | 98.0 | 98.7 | 96.0 | 95.7 | 95.0 | 92.3 | 91.0 | 88.3 | 84.7 | 82.7 | 87.3 | | | | | | | **Trend Verification by Type:** * **Type 1:** Shows a general downward trend. Accuracy starts very high (99.3% at Length 0) and declines to the low 80s by Length 9. * **Type 2:** Maintains exceptionally high accuracy (97.3% - 100.0%) across its available lengths (1-10), with minimal degradation. * **Type 3:** Exhibits a fluctuating but relatively stable trend after an initial drop. Accuracy starts at 99.7%, dips into the mid-80s, and then stabilizes in the 86-90% range for longer lengths. * **Type 4:** Shows stable performance, mostly in the 91-92% range, with a slight peak at Lengths 9-10 (92.7%). * **Type 5:** Starts at a lower accuracy (80.3% at Length 7) and shows a slight, inconsistent upward trend, reaching 89.7% at Length 19. * **Type 6:** Demonstrates the most consistent and highest performance, with accuracy almost exclusively between 97.7% and 100.0% across all measured lengths (0-18). * **Type 7:** Shows a clear downward trend. Accuracy begins at 99.7% and steadily decreases to 82.7% at Length 12, with a slight recovery at Length 13 (87.3%). ### Key Observations 1. **Performance Variability:** There is significant variability in performance across the different "Types". Type 6 is the top performer, while Type 5 shows the lowest initial accuracy. 2. **Length Sensitivity:** Some types are highly sensitive to length (e.g., Types 1 and 7 show clear degradation), while others are robust (e.g., Types 2 and 6 maintain high accuracy). 3. **Data Coverage:** Not all Types have data for all Lengths. Type 3 has the most complete data (Lengths 0-19). Types 1, 2, 4, and 7 have data only for shorter to medium lengths. Type 5 only has data for longer lengths (7-19). 4. **High-Accuracy Clusters:** The darkest blue cells (accuracy >98%) are concentrated in the top-left region of the chart (shorter lengths for Types 1, 2, 3, 6, 7) and throughout Type 6. ### Interpretation This heatmap provides a diagnostic view of the Qwen-2.5 3B model's generalization capabilities. The "Type" axis likely represents different categories or tasks within the "Core Generalization" benchmark, while "Length" probably refers to the sequence length or complexity of the input. * **Model Strengths:** The model exhibits strong and robust performance on Type 6 tasks across all lengths, suggesting a particular strength in that category. It also performs very well on shorter sequences for most types. * **Model Weaknesses:** The model struggles with Type 5 tasks, especially at shorter lengths. It also shows a clear vulnerability to increasing sequence length for Types 1 and 7, where accuracy drops by over 10 percentage points. * **Generalization Pattern:** The data suggests that the model's ability to generalize is not uniform. Its performance is highly dependent on the specific nature of the task (Type) and the length of the input. The degradation with length for some types indicates a potential limitation in handling long-range dependencies or maintaining context for those specific tasks. * **Practical Implication:** For users of this model, this chart indicates that performance will be most reliable for Type 6 tasks and for shorter inputs across most categories. When dealing with Type 5 tasks or long sequences of Type 1 or 7, one should expect lower and potentially declining accuracy.

# Technical Document Extraction: Baseline - Core Generalization - Qwen-2.5 3B ## 1. Chart Identification - **Type**: Heatmap - **Title**: "Baseline - Core Generalization - Qwen-2.5 3B" - **Color Scale**: Accuracy (%) from 0% (lightest) to 100% (darkest blue) ## 2. Axis Labels & Markers - **X-axis (Horizontal)**: - Label: "Length" - Values: 0 to 19 (integer increments) - **Y-axis (Vertical)**: - Label: "Type" - Values: 1 to 7 (integer increments) - **Colorbar**: - Label: "Accuracy (%)" - Range: 0% (lightest) to 100% (darkest blue) ## 3. Data Categories - **Types (Rows)**: 1, 2, 3, 4, 5, 6, 7 - **Lengths (Columns)**: 0 to 19 ## 4. Data Table Reconstruction | Type \ Length | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | |---------------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------| | 1 | 99.3 | 96.7 | 98.3 | 92.7 | 87.7 | 83.0 | 82.0 | 86.3 | 86.0 | 83.7 | - | - | - | - | - | - | - | - | - | - | | 2 | - | 100.0 | 99.7 | 99.7 | 98.3 | 98.0 | 99.3 | 98.7 | 99.3 | 97.3 | 97.7 | - | - | - | - | - | - | - | - | - | | 3 | 99.7 | 98.0 | 94.0 | 95.0 | 95.7 | 89.7 | 86.0 | 88.3 | 90.3 | 86.7 | 86.0 | 89.3 | 89.3 | 86.0 | 90.0 | 89.0 | 90.0 | 90.7 | 90.0 | 89.0 | | 4 | - | 98.3 | 98.3 | 91.0 | 91.7 | 92.0 | 91.0 | 92.0 | 92.3 | 92.7 | 92.7 | 90.7 | - | - | - | - | - | - | - | - | | 5 | - | - | - | - | - | - | - | 80.3 | 84.3 | 81.3 | 87.3 | 87.3 | 85.7 | 89.0 | 90.0 | 87.0 | 85.0 | 87.3 | 86.0 | 89.7 | | 6 | 100.0 | 99.3 | 99.7 | 99.7 | 99.0 | 100.0 | 98.3 | 99.3 | 99.3 | 98.3 | 98.3 | 98.7 | 98.0 | 97.7 | 98.7 | 98.7 | 98.7 | 98.3 | 97.7 | - | | 7 | 99.7 | 99.7 | 98.7 | 98.0 | 98.7 | 96.0 | 95.7 | 95.0 | 92.3 | 91.0 | 88.3 | 84.7 | 82.7 | 87.3 | - | - | - | - | - | - | ## 5. Key Trends 1. **General Pattern**: Accuracy decreases as Length increases for all Types. 2. **Type 2**: - Highest accuracy at Length 0 (100%) - Gradual decline to 97.7% at Length 19 3. **Type 5**: - Sharp drop from 90.0% (Length 14) to 85.0% (Length 16) - Recovery to 89.7% at Length 19 4. **Type 6**: - Maintains >97% accuracy until Length 18 (97.7%) - Final drop to 98.3% at Length 19 5. **Type 7**: - Steepest decline (99.7% → 87.3% between Lengths 0-13) - Partial recovery to 89.7% at Length 19 ## 6. Spatial Grounding - **Legend Position**: Right side of chart (colorbar) - **Color Consistency**: Darker blues correspond to higher accuracy values (e.g., 100% = darkest blue, 80% = medium blue) ## 7. Component Isolation - **Header**: Chart title at top center - **Main Chart**: 7x20 heatmap grid - **Footer**: Colorbar legend on right edge ## 8. Data Validation - All values match color intensity expectations - No missing values except for Type 5 (Lengths 0-4) and Type 7 (Lengths 14-19) - Type 6 shows highest overall accuracy (98.7-100% range)