## Chart: Accuracy vs. Number of Operations ### Overview The image presents two line charts comparing the accuracy of a system against the number of operations performed. The charts are differentiated by the number of pause tokens used: 5 in the left chart (a) and 10 in the right chart (b). Each chart displays three data series, representing different values of 'n' (n=1, n=2, n=4). The x-axis represents the number of operations, divided into "in-domain" (1-5 operations) and "out-of-domain" (6-10 operations) categories. The y-axis represents accuracy in percentage. ### Components/Axes * **Title (Left Chart):** (a) 5 pause tokens * **Title (Right Chart):** (b) 10 pause tokens * **X-axis Title:** # operations * **X-axis Categories:** in-domain (operations 1-5), out-of-domain (operations 6-10) * **X-axis Markers:** 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 * **Y-axis Title:** Accuracy (%) * **Y-axis Markers:** 0, 20, 40, 60, 80, 100 * **Legend (Top-Right of each chart):** * n=1 (coral color, solid line with 'x' markers) * n=2 (dark blue-gray color, dashed line with 'x' markers) * n=4 (light green color, dotted line with 'x' markers) ### Detailed Analysis **Left Chart (5 pause tokens):** * **n=1 (coral):** The accuracy decreases sharply from approximately 85% at 1 operation to about 10% at 6 operations. It then remains near 0% for operations 7-10. * (1, 85%), (2, 70%), (3, 40%), (4, 30%), (5, 15%), (6, 10%), (7, 2%), (8, 1%), (9, 0%), (10, 0%) * **n=2 (dark blue-gray):** The accuracy decreases from approximately 95% at 1 operation to about 5% at 6 operations. It then remains near 0% for operations 7-10. * (1, 95%), (2, 75%), (3, 60%), (4, 35%), (5, 20%), (6, 5%), (7, 2%), (8, 1%), (9, 0%), (10, 0%) * **n=4 (light green):** The accuracy decreases from approximately 90% at 1 operation to about 5% at 6 operations. It then remains near 0% for operations 7-10. * (1, 90%), (2, 80%), (3, 55%), (4, 40%), (5, 25%), (6, 5%), (7, 3%), (8, 2%), (9, 1%), (10, 1%) **Right Chart (10 pause tokens):** * **n=1 (coral):** The accuracy decreases sharply from approximately 85% at 1 operation to about 5% at 6 operations. It then remains near 0% for operations 7-10. * (1, 85%), (2, 60%), (3, 35%), (4, 25%), (5, 15%), (6, 5%), (7, 1%), (8, 0%), (9, 0%), (10, 0%) * **n=2 (dark blue-gray):** The accuracy decreases from approximately 95% at 1 operation to about 5% at 6 operations. It then remains near 0% for operations 7-10. * (1, 95%), (2, 75%), (3, 50%), (4, 30%), (5, 20%), (6, 5%), (7, 2%), (8, 1%), (9, 0%), (10, 0%) * **n=4 (light green):** The accuracy decreases from approximately 80% at 1 operation to about 5% at 6 operations. It then remains near 0% for operations 7-10. * (1, 80%), (2, 70%), (3, 55%), (4, 40%), (5, 20%), (6, 5%), (7, 3%), (8, 2%), (9, 1%), (10, 1%) ### Key Observations * All three data series (n=1, n=2, n=4) in both charts exhibit a similar trend: a rapid decrease in accuracy as the number of operations increases. * The accuracy drops significantly when transitioning from "in-domain" to "out-of-domain" operations (after 5 operations). * The initial accuracy (at 1 operation) is highest for n=2 in both charts. * The number of pause tokens (5 vs. 10) does not appear to drastically alter the overall trend or the relative performance of the different 'n' values. ### Interpretation The charts suggest that the system's accuracy is highly sensitive to the number of operations performed, particularly when moving to "out-of-domain" scenarios. The rapid decline in accuracy indicates that the system struggles to maintain performance as the complexity or novelty of the operations increases. The 'n' value seems to have a minor impact on the initial accuracy, but the overall trend of decreasing accuracy with more operations is consistent across all 'n' values. The number of pause tokens (5 vs. 10) has a minimal effect on the overall performance, suggesting that this parameter is not a primary driver of accuracy in this context. The data implies that the system needs improvement in handling a larger number of operations and generalizing to out-of-domain scenarios.

\n ## Line Chart: Accuracy vs. Number of Operations for Different 'n' Values ### Overview The image presents two line charts, side-by-side, illustrating the relationship between accuracy (in percentage) and the number of operations, for different values of 'n' (1, 2, and 4). Each chart is divided into two regions: "in-domain" and "out-of-domain", demarcated by a vertical dashed line. The charts compare performance under two different pause token conditions: (a) 5 pause tokens and (b) 10 pause tokens. ### Components/Axes * **X-axis:** "# operations" - ranging from 1 to 10, with integer markers. * **Y-axis:** "Accuracy (%)" - ranging from 0 to 100, with markers at 0, 20, 40, 60, 80, and 100. * **Legend:** Located in the top-right corner of each chart. * n=1 (represented by a red line with 'x' markers) * n=2 (represented by a purple line with '+' markers) * n=4 (represented by a green line with '*' markers) * **Vertical Dashed Line:** Separates the "in-domain" and "out-of-domain" regions. * **Chart Titles:** * (a) 5 pause tokens * (b) 10 pause tokens * **Domain Labels:** "in-domain" and "out-of-domain" are labeled below the x-axis. ### Detailed Analysis or Content Details **Chart (a) - 5 pause tokens:** * **n=1 (Red):** Starts at approximately 100% accuracy at operation 1. Decreases steadily within the in-domain (operations 1-4), reaching approximately 10% accuracy at operation 4. Continues to decrease in the out-of-domain (operations 5-10), leveling off around 5% accuracy at operation 10. * **n=2 (Purple):** Starts at approximately 98% accuracy at operation 1. Decreases more rapidly than n=1 within the in-domain, reaching approximately 20% accuracy at operation 4. Decreases further in the out-of-domain, leveling off around 5% accuracy at operation 10. * **n=4 (Green):** Starts at approximately 98% accuracy at operation 1. Decreases rapidly within the in-domain, reaching approximately 40% accuracy at operation 4. Decreases sharply in the out-of-domain, leveling off around 5% accuracy at operation 10. **Chart (b) - 10 pause tokens:** * **n=1 (Red):** Starts at approximately 100% accuracy at operation 1. Decreases steadily within the in-domain (operations 1-4), reaching approximately 20% accuracy at operation 4. Continues to decrease in the out-of-domain (operations 5-10), leveling off around 5% accuracy at operation 10. * **n=2 (Purple):** Starts at approximately 98% accuracy at operation 1. Decreases more rapidly than n=1 within the in-domain, reaching approximately 30% accuracy at operation 4. Decreases further in the out-of-domain, leveling off around 5% accuracy at operation 10. * **n=4 (Green):** Starts at approximately 98% accuracy at operation 1. Decreases rapidly within the in-domain, reaching approximately 50% accuracy at operation 4. Decreases sharply in the out-of-domain, leveling off around 5% accuracy at operation 10. ### Key Observations * Accuracy decreases as the number of operations increases for all values of 'n' in both charts. * The rate of accuracy decrease is more pronounced in the out-of-domain region compared to the in-domain region. * Higher values of 'n' (2 and 4) exhibit a faster decline in accuracy, especially within the in-domain region. * All lines converge to approximately 5% accuracy at operation 10, regardless of the value of 'n' or the number of pause tokens. * Increasing the number of pause tokens (from 5 to 10) generally leads to a slightly slower initial decline in accuracy, but the overall trend remains the same. ### Interpretation The data suggests that as the number of operations increases, the accuracy of the system degrades. This degradation is more significant when operating outside of the "in-domain" conditions. The parameter 'n' appears to influence the rate of accuracy decline, with larger values of 'n' leading to a faster decrease. The convergence of all lines to a similar low accuracy level at higher operation counts indicates a potential limit to the system's performance, regardless of the initial parameter settings. The difference between the two charts (5 vs. 10 pause tokens) suggests that increasing the pause duration can provide a slight buffer against initial accuracy loss, but does not fundamentally alter the overall trend of decreasing accuracy with increasing operations. This could indicate that the system is sensitive to the complexity of the operations performed, and that longer pauses may allow for some degree of recovery or stabilization, but ultimately cannot prevent the eventual decline in performance. The "in-domain" vs "out-of-domain" distinction highlights the importance of operating within the system's intended use case for optimal accuracy.

## Line Charts: Accuracy vs. Number of Operations with Varying Pause Tokens ### Overview The image contains two side-by-side line charts, labeled (a) and (b), which plot model accuracy (as a percentage) against the number of sequential operations performed. The charts compare performance for three different model configurations (n=1, n=2, n=4) and distinguish between "in-domain" and "out-of-domain" tasks. The primary variable between the two charts is the number of "pause tokens" used: 5 in chart (a) and 10 in chart (b). ### Components/Axes * **Chart Type:** Two line charts with markers. * **Y-Axis (Both Charts):** Label: "Accuracy (%)". Scale: 0 to 100, with major ticks at 0, 20, 40, 60, 80, 100. * **X-Axis (Both Charts):** Label: "# operations". Scale: 1 to 10, with integer ticks. * **Legend (Both Charts):** Located in the top-right corner of each plot area. * Orange line with 'x' markers: `n=1` * Dark blue/purple dashed line with 'x' markers: `n=2` * Green dotted line with 'x' markers: `n=4` * **Domain Separation:** A vertical dotted gray line at x=5.5 on both charts separates the x-axis into two regions, annotated below the axis: * Left of the line (operations 1-5): "in-domain" * Right of the line (operations 6-10): "out-of-domain" * **Subplot Titles/Captions:** * Chart (a) (Left): "(a) 5 pause tokens" * Chart (b) (Right): "(b) 10 pause tokens" ### Detailed Analysis **Chart (a): 5 pause tokens** * **Trend Verification:** All three data series show a steep, monotonic decline in accuracy as the number of operations increases. The decline is most severe in the "out-of-domain" region. * **Data Series n=1 (Orange, solid line):** * Starts at ~85% accuracy for 1 operation. * Drops to ~65% at 2 ops, ~45% at 3 ops, ~30% at 4 ops, ~18% at 5 ops. * In the out-of-domain region: ~5% at 6 ops, ~2% at 7 ops, and approaches 0% for 8-10 ops. * **Data Series n=2 (Dark blue, dashed line):** * Starts highest at ~95% for 1 operation. * Drops to ~80% at 2 ops, ~55% at 3 ops, ~40% at 4 ops, ~25% at 5 ops. * In the out-of-domain region: ~10% at 6 ops, ~5% at 7 ops, and near 0% for 8-10 ops. * **Data Series n=4 (Green, dotted line):** * Starts at ~90% for 1 operation. * Drops to ~80% at 2 ops, ~60% at 3 ops, ~45% at 4 ops, ~30% at 5 ops. * In the out-of-domain region: ~15% at 6 ops, ~5% at 7 ops, and near 0% for 8-10 ops. * **Relative Performance:** For 1-5 operations (in-domain), the order of performance is generally n=2 > n=4 > n=1. The gap between them narrows as operations increase. In the out-of-domain region, all models perform very poorly, with n=4 showing a slight, brief advantage at 6 operations. **Chart (b): 10 pause tokens** * **Trend Verification:** Identical overall trend to chart (a): a steep, monotonic decline in accuracy with increasing operations. * **Data Series n=1 (Orange, solid line):** * Starts at ~90% for 1 operation. * Drops to ~60% at 2 ops, ~38% at 3 ops, ~22% at 4 ops, ~12% at 5 ops. * In the out-of-domain region: ~3% at 6 ops, ~1% at 7 ops, and 0% for 8-10 ops. * **Data Series n=2 (Dark blue, dashed line):** * Starts at ~100% for 1 operation. * Drops to ~80% at 2 ops, ~55% at 3 ops, ~35% at 4 ops, ~20% at 5 ops. * In the out-of-domain region: ~8% at 6 ops, ~2% at 7 ops, and 0% for 8-10 ops. * **Data Series n=4 (Green, dotted line):** * Starts at ~95% for 1 operation. * Drops to ~80% at 2 ops, ~60% at 3 ops, ~40% at 4 ops, ~25% at 5 ops. * In the out-of-domain region: ~12% at 6 ops, ~3% at 7 ops, and 0% for 8-10 ops. * **Relative Performance:** The pattern is similar to chart (a). n=2 starts highest, followed closely by n=4, with n=1 starting lower. The performance hierarchy is maintained through the in-domain tasks, but all converge to near-zero accuracy in the out-of-domain region. ### Key Observations 1. **Universal Performance Degradation:** Accuracy for all models (n=1,2,4) and both pause token settings degrades rapidly and monotonically as the number of sequential operations increases. 2. **Sharp Domain Boundary:** There is a catastrophic drop in performance once the task moves from "in-domain" (operations 1-5) to "out-of-domain" (operations 6-10). Accuracy plummets to single digits or zero immediately after the boundary. 3. **Model Comparison (n-value):** Models with higher `n` (n=2, n=4) consistently outperform the n=1 model, especially for a lower number of operations. The n=2 model often shows the highest initial accuracy. 4. **Effect of Pause Tokens:** Comparing (a) and (b), increasing pause tokens from 5 to 10 appears to slightly improve the starting accuracy for n=1 and n=4 at 1 operation, but does not change the fundamental trend of rapid degradation or the out-of-domain collapse. ### Interpretation The data demonstrates a severe limitation in the models' ability to perform long, sequential reasoning chains. The steep negative slope indicates that each additional operation introduces significant error accumulation, leading to a rapid loss of accuracy. The stark "in-domain" vs. "out-of-domain" divide suggests the models are not performing robust, generalizable reasoning. Instead, they likely rely on pattern recognition or heuristics that work for shorter sequences (1-5 steps) seen during training but completely fail when extrapolated to longer sequences (6-10 steps), even if the operations are of the same type. This is a classic sign of poor generalization beyond the training distribution. The advantage of higher `n` (which may represent a model size, ensemble size, or number of reasoning steps considered) provides a buffer against error for in-domain tasks but is insufficient to overcome the fundamental generalization failure in the out-of-domain regime. The number of pause tokens has a marginal effect, indicating that simply giving the model more "thinking time" or intermediate steps does not solve the core problem of compositional generalization for long sequences. The charts highlight a critical challenge in building systems that can reliably perform multi-step reasoning.

## Line Graphs: Accuracy vs. Number of Operations with Pause Tokens ### Overview The image contains two line graphs (a and b) comparing model accuracy (%) against the number of operations (1–10) for in-domain and out-of-domain tasks. Each graph represents different numbers of pause tokens (n=1, n=2, n=4). Accuracy declines as operations increase, with distinct trends for in-domain and out-of-domain performance. ### Components/Axes - **X-axis**: "# operations" (1–10), split into "in-domain" (1–5.5) and "out-of-domain" (5.5–10) by a vertical dashed line. - **Y-axis**: "Accuracy (%)" (0–100). - **Legends**: - Top-right corner, with colors: - Red (`n=1`) - Blue (`n=2`) - Green (`n=4`) - **Graph Labels**: - (a) "5 pause tokens" - (b) "10 pause tokens" ### Detailed Analysis #### Graph (a): 5 Pause Tokens - **In-domain**: - **n=1 (red)**: Starts at ~85% (1 op), drops to ~60% (3 ops), ~35% (5 ops), ~20% (7 ops), ~5% (10 ops). - **n=2 (blue)**: Starts at ~90% (1 op), ~70% (3 ops), ~45% (5 ops), ~25% (7 ops), ~10% (10 ops). - **n=4 (green)**: Starts at ~80% (1 op), ~60% (3 ops), ~30% (5 ops), ~15% (7 ops), ~5% (10 ops). - **Out-of-domain**: - **n=1 (red)**: Starts at ~75% (1 op), ~50% (3 ops), ~25% (5 ops), ~10% (7 ops), ~2% (10 ops). - **n=2 (blue)**: Starts at ~85% (1 op), ~65% (3 ops), ~35% (5 ops), ~15% (7 ops), ~3% (10 ops). - **n=4 (green)**: Starts at ~70% (1 op), ~45% (3 ops), ~20% (5 ops), ~8% (7 ops), ~2% (10 ops). #### Graph (b): 10 Pause Tokens - **In-domain**: - **n=1 (red)**: Starts at ~90% (1 op), ~75% (3 ops), ~50% (5 ops), ~30% (7 ops), ~10% (10 ops). - **n=2 (blue)**: Starts at ~95% (1 op), ~80% (3 ops), ~60% (5 ops), ~40% (7 ops), ~15% (10 ops). - **n=4 (green)**: Starts at ~85% (1 op), ~65% (3 ops), ~40% (5 ops), ~25% (7 ops), ~5% (10 ops). - **Out-of-domain**: - **n=1 (red)**: Starts at ~80% (1 op), ~60% (3 ops), ~30% (5 ops), ~10% (7 ops), ~2% (10 ops). - **n=2 (blue)**: Starts at ~85% (1 op), ~65% (3 ops), ~35% (5 ops), ~15% (7 ops), ~3% (10 ops). - **n=4 (green)**: Starts at ~75% (1 op), ~50% (3 ops), ~25% (5 ops), ~10% (7 ops), ~2% (10 ops). ### Key Observations 1. **Pause Token Impact**: Higher `n` (pause tokens) correlates with higher accuracy, especially in in-domain tasks. 2. **In-Domain vs. Out-of-Domain**: In-domain accuracy is consistently higher than out-of-domain for all `n`. 3. **Decline Rate**: Accuracy drops faster for lower `n` (e.g., n=1 declines ~80% from 1 to 10 ops in in-domain). 4. **Threshold**: The split at 5.5 operations suggests a performance threshold where out-of-domain accuracy sharply declines. ### Interpretation The data demonstrates that increasing pause tokens (`n`) improves model robustness, particularly in in-domain scenarios. Out-of-domain performance is more sensitive to operational complexity, with accuracy decaying faster as operations increase. The vertical split at 5.5 operations may indicate a critical point where the model’s ability to generalize diminishes, highlighting the importance of pause tokens in maintaining stability during extended operations. The steeper decline for lower `n` suggests that insufficient pause tokens exacerbate overfitting or instability in dynamic tasks.