## Line Charts: Accuracy of Different Models on Sudoku, Sujiko, and Magic-Square ### Overview The image contains three line charts comparing the accuracy of four different models (Few-Shot, Logic-LM, PAL, and SymPro-LM) on three different puzzle types: Sudoku, Sujiko, and Magic-Square. The x-axis represents the board size, and the y-axis represents the accuracy in percentage. ### Components/Axes * **Titles:** * Chart 1: Sudoku * Chart 2: Sujiko * Chart 3: Magic-Square * **X-Axis:** Board Size * Sudoku: 4x4, 9x9, 16x16, 25x25 * Sujiko: 3x3, 4x4, 5x5 * Magic-Square: 3x3, 4x4, 5x5 * **Y-Axis:** Accuracy (%) * Scale: 0 to 100, with gridlines at intervals of 20. * **Legend (Top-Right):** * Orange line with circle markers: Few-Shot * Purple dashed line with square markers: Logic-LM * Blue dash-dotted line with triangle markers: PAL * Green dotted line with diamond markers: SymPro-LM ### Detailed Analysis #### Sudoku Chart * **Few-Shot (Orange):** Starts at approximately 60% accuracy for 4x4, drops to approximately 5% for 9x9, and remains at 0% for 16x16 and 25x25. * **Logic-LM (Purple):** Starts at approximately 60% accuracy for 4x4, drops to approximately 5% for 9x9, and remains at 0% for 16x16 and 25x25. * **PAL (Blue):** Starts at approximately 100% accuracy for 4x4, drops to approximately 60% for 25x25. * **SymPro-LM (Green):** Maintains 100% accuracy across all board sizes (4x4, 9x9, 16x16, 25x25). #### Sujiko Chart * **Few-Shot (Orange):** Starts at approximately 47% accuracy for 3x3, drops to approximately 20% for 4x4, and further to approximately 2% for 5x5. * **Logic-LM (Purple):** Starts at approximately 68% accuracy for 3x3, drops to approximately 53% for 4x4, and further to approximately 13% for 5x5. * **PAL (Blue):** Starts at approximately 100% accuracy for 3x3, drops to approximately 80% for 5x5. * **SymPro-LM (Green):** Maintains 100% accuracy across all board sizes (3x3, 4x4, 5x5). #### Magic-Square Chart * **Few-Shot (Orange):** Starts at approximately 20% accuracy for 3x3, drops to 0% for 4x4 and remains at 0% for 5x5. * **Logic-LM (Purple):** Starts at approximately 67% accuracy for 3x3, drops to approximately 25% for 4x4, and further to approximately 7% for 5x5. * **PAL (Blue):** Starts at approximately 100% accuracy for 3x3, drops to approximately 0% for 4x4 and remains at 0% for 5x5. * **SymPro-LM (Green):** Maintains 100% accuracy across all board sizes (3x3, 4x4, 5x5). ### Key Observations * SymPro-LM consistently achieves 100% accuracy across all puzzle types and board sizes. * Few-Shot and Logic-LM generally perform worse than PAL and SymPro-LM. * The accuracy of Few-Shot and Logic-LM decreases significantly as the board size increases for Sudoku. * PAL's performance varies significantly across puzzle types, performing well on Sudoku but poorly on Magic-Square for larger board sizes. ### Interpretation The data suggests that SymPro-LM is the most robust model for solving these puzzles, as its accuracy remains consistently high regardless of puzzle type or board size. PAL shows promise but is less consistent. Few-Shot and Logic-LM appear to struggle with larger board sizes, particularly in Sudoku. The relationship between board size and accuracy highlights the challenges these models face in scaling to more complex puzzles. The performance differences across puzzle types suggest that the models may be better suited for certain types of logical reasoning problems than others.

## Line Chart: Puzzle Solving Accuracy vs. Board Size ### Overview The image presents three line charts, each representing the accuracy of different puzzle-solving methods across varying board sizes. The puzzles are Sudoku, Sujiko, and Magic Square. The accuracy is measured in percentage (%), and the board size is specified as dimensions (e.g., 4x4, 9x9). Four methods are compared: Few-Shot, Logic-LM, PAL, and SymPro-LM. ### Components/Axes * **X-axis (all charts):** Board Size. The specific board sizes vary for each puzzle type. * Sudoku: 4x4, 9x9, 16x16, 25x25, 36x36 * Sujiko: 4x4, 5x5, 3x3 * Magic Square: 4x4, 5x5 * **Y-axis (all charts):** Accuracy (%). Scale ranges from 0% to 100%. * **Legend (top-right):** * Few-Shot (Orange) * Logic-LM (Purple) * PAL (Blue) * SymPro-LM (Green) * **Chart Titles (top-center):** * Sudoku * Sujiko * Magic Square ### Detailed Analysis or Content Details **Sudoku Chart:** * **Few-Shot (Orange):** Starts at approximately 40% accuracy for 4x4 boards, decreases sharply to approximately 10% for 9x9 boards, and continues to decline, reaching approximately 0% for 36x36 boards. The trend is strongly downward. * **Logic-LM (Purple):** Begins at approximately 60% for 4x4 boards, increases slightly to approximately 65% for 9x9 boards, then decreases to approximately 40% for 16x16 boards, and continues to decline to approximately 10% for 36x36 boards. The trend is initially slightly upward, then strongly downward. * **PAL (Blue):** Starts at approximately 85% for 4x4 boards, decreases to approximately 70% for 9x9 boards, then declines more rapidly to approximately 40% for 25x25 boards, and reaches approximately 10% for 36x36 boards. The trend is downward. * **SymPro-LM (Green):** Maintains a consistently high accuracy of approximately 100% across all board sizes (4x4 to 36x36). **Sujiko Chart:** * **Few-Shot (Orange):** Starts at approximately 30% for 4x4 boards, decreases to approximately 10% for 5x5 boards, and then increases to approximately 20% for 3x3 boards. The trend is initially downward, then upward. * **Logic-LM (Purple):** Begins at approximately 60% for 4x4 boards, increases to approximately 80% for 5x5 boards, and then decreases to approximately 15% for 3x3 boards. The trend is initially upward, then downward. * **PAL (Blue):** Starts at approximately 80% for 4x4 boards, increases to approximately 90% for 5x5 boards, and then decreases to approximately 60% for 3x3 boards. The trend is initially upward, then downward. * **SymPro-LM (Green):** Maintains a consistently high accuracy of approximately 100% across all board sizes (4x4, 5x5, 3x3). **Magic Square Chart:** * **Few-Shot (Orange):** Starts at approximately 40% for 4x4 boards, decreases to approximately 10% for 5x5 boards. The trend is downward. * **Logic-LM (Purple):** Begins at approximately 60% for 4x4 boards, decreases to approximately 10% for 5x5 boards. The trend is downward. * **PAL (Blue):** Starts at approximately 80% for 4x4 boards, decreases to approximately 40% for 5x5 boards. The trend is downward. * **SymPro-LM (Green):** Maintains a consistently high accuracy of approximately 100% across all board sizes (4x4, 5x5). ### Key Observations * SymPro-LM consistently achieves 100% accuracy across all puzzle types and board sizes. * Few-Shot consistently performs the worst, with accuracy decreasing as board size increases. * Logic-LM and PAL show varying performance depending on the puzzle type and board size. * For Sudoku and Sujiko, accuracy generally decreases with increasing board size for all methods except SymPro-LM. * Sujiko shows a slight increase in accuracy for Few-Shot, Logic-LM, and PAL when moving from 5x5 to 3x3 boards. ### Interpretation The data suggests that SymPro-LM is the most effective method for solving these puzzles, regardless of board size. This could be due to its superior algorithm or its ability to handle the complexity of larger boards. Few-Shot appears to struggle with increasing board size, indicating it may not generalize well to more complex instances. Logic-LM and PAL demonstrate a more nuanced performance, with accuracy varying depending on the puzzle type and board size. The slight increase in accuracy for some methods when moving from 5x5 to 3x3 Sujiko boards could be due to the specific characteristics of Sujiko puzzles, potentially making smaller boards easier to solve. The charts provide a clear comparison of the performance of different puzzle-solving methods, highlighting the strengths and weaknesses of each approach. The consistent performance of SymPro-LM suggests it is a robust and reliable solution for these types of puzzles.

## Line Charts: Performance of Different Methods on Logic Puzzles ### Overview The image displays three side-by-side line charts comparing the accuracy of four different methods (Few-Shot, Logic-LM, PAL, SymPro-LM) on three types of logic puzzles (Sudoku, Sujiko, Magic-Square) as the puzzle board size increases. The overall trend shows that the SymPro-LM method maintains perfect accuracy across all tested sizes, while the performance of other methods generally degrades with increased puzzle complexity. ### Components/Axes * **Chart Titles (Top Center):** "Sudoku", "Sujiko", "Magic-Square". * **Y-Axis (Left Side, Shared):** Labeled "Accuracy (%)". Scale runs from 0 to 100 in increments of 20. * **X-Axis (Bottom of Each Chart):** Labeled "Board Size". * **Sudoku Chart:** Categories are "4x4", "9x9", "16x16", "25x25". * **Sujiko Chart:** Categories are "3x3", "4x4", "5x5". * **Magic-Square Chart:** Categories are "3x3", "4x4", "5x5". * **Legend (Top-Right of Magic-Square Chart):** * **Few-Shot:** Orange line with circle markers. * **Logic-LM:** Purple dashed line with square markers. * **PAL:** Blue dash-dot line with triangle markers. * **SymPro-LM:** Green dotted line with diamond markers. ### Detailed Analysis **1. Sudoku Chart (Left Panel)** * **SymPro-LM (Green Diamond):** Maintains a constant 100% accuracy across all board sizes (4x4, 9x9, 16x16, 25x25). The line is perfectly horizontal at the top of the chart. * **PAL (Blue Triangle):** Starts at 100% for 4x4, 9x9, and 16x16 boards. Shows a significant drop to approximately 60% accuracy for the 25x25 board. * **Logic-LM (Purple Square):** Starts at approximately 60% for the 4x4 board. Accuracy drops sharply to near 0% for the 9x9 board and remains at approximately 0% for 16x16 and 25x25. * **Few-Shot (Orange Circle):** Follows a nearly identical path to Logic-LM. Starts at approximately 60% for 4x4, drops to 0% for 9x9, and remains at 0% for larger sizes. **2. Sujiko Chart (Middle Panel)** * **SymPro-LM (Green Diamond):** Maintains 100% accuracy across all board sizes (3x3, 4x4, 5x5). * **PAL (Blue Triangle):** Starts at 100% for 3x3 and 4x4 boards. Accuracy decreases to approximately 80% for the 5x5 board. * **Logic-LM (Purple Square):** Starts at approximately 65% for 3x3. Accuracy declines to approximately 55% for 4x4 and further to approximately 15% for 5x5. * **Few-Shot (Orange Circle):** Starts at approximately 45% for 3x3. Accuracy declines to approximately 20% for 4x4 and drops to 0% for 5x5. **3. Magic-Square Chart (Right Panel)** * **SymPro-LM (Green Diamond):** Maintains 100% accuracy across all board sizes (3x3, 4x4, 5x5). * **PAL (Blue Triangle):** Starts at 100% for the 3x3 board. Accuracy plummets to 0% for both the 4x4 and 5x5 boards. * **Logic-LM (Purple Square):** Starts at approximately 65% for 3x3. Accuracy declines to approximately 30% for 4x4 and further to approximately 5% for 5x5. * **Few-Shot (Orange Circle):** Starts at approximately 20% for 3x3. Accuracy drops to 0% for both the 4x4 and 5x5 boards. ### Key Observations 1. **Dominant Performance:** The SymPro-LM method achieves perfect (100%) accuracy on every puzzle type and board size tested, showing no degradation with increased complexity. 2. **Critical Failure Point for PAL:** The PAL method performs perfectly on smaller boards but experiences a catastrophic failure at the largest tested size for each puzzle: 25x25 for Sudoku, 5x5 for Sujiko, and 4x4 for Magic-Square. 3. **Rapid Degradation of Baselines:** Both the Few-Shot and Logic-LM methods show poor scalability. Their accuracy is moderate on the smallest puzzles but drops to near zero as the board size increases by just one step in most cases. 4. **Puzzle Difficulty Hierarchy:** For the non-perfect methods, the Magic-Square puzzle appears to be the most challenging, causing the steepest and earliest drops in accuracy (e.g., PAL fails at 4x4, while it holds until 5x5 in Sujiko). ### Interpretation The data strongly suggests that the **SymPro-LM** method possesses a fundamental architectural or algorithmic advantage for solving constraint-based logic puzzles, as it is completely unaffected by the scaling of problem size within the tested range. This implies it may be using a form of symbolic reasoning or program synthesis that generalizes perfectly. In contrast, the other methods (**PAL, Logic-LM, Few-Shot**) likely rely on pattern recognition or in-context learning that breaks down when the puzzle's combinatorial complexity exceeds a certain threshold. The sharp "cliff-edge" failure of PAL (e.g., from 100% to 0% in Magic-Square) is particularly notable, suggesting it hits a hard limit rather than a gradual decline. The charts collectively demonstrate a clear performance dichotomy: one method (SymPro-LM) is robust and scalable, while the others are fragile and limited to simpler instances. This has significant implications for applying large language models to formal reasoning tasks, highlighting the need for specialized techniques like symbolic programming integration to achieve reliable performance on complex problems.

## Line Charts: Algorithm Accuracy Across Puzzle Games ### Overview The image contains three line charts comparing the accuracy of four algorithms (Few-Shot, Logic-LM, PAL, SymPro-LM) across different puzzle board sizes for Sudoku, Sujiko, and Magic-Square games. Accuracy is measured on a 0-100% scale, with SymPro-LM consistently achieving perfect scores. ### Components/Axes - **X-Axes (Board Size)**: - Sudoku: 4x4, 9x9, 16x16, 25x25 - Sujiko: 3x3, 4x4, 5x5 - Magic-Square: 3x3, 4x4, 5x5 - **Y-Axes (Accuracy %)**: 0-100% scale - **Legends**: Located on the right of each chart, with color-coded labels: - Orange: Few-Shot - Purple: Logic-LM - Blue: PAL - Green: SymPro-LM ### Detailed Analysis #### Sudoku - **SymPro-LM**: Flat line at 100% across all board sizes. - **Logic-LM**: Starts at ~60% (4x4), drops sharply to ~5% (9x9), and plateaus near 0% for larger boards. - **Few-Shot**: Begins at ~55% (4x4), plummets to 0% by 9x9, remaining at 0% for larger boards. - **PAL**: Starts at ~95% (4x4), declines to ~60% (25x25). #### Sujiko - **SymPro-LM**: Flat line at 100% across all board sizes. - **Logic-LM**: Starts at ~65% (3x3), declines to ~50% (4x4), and drops to ~10% (5x5). - **Few-Shot**: Begins at ~45% (3x3), declines to ~20% (4x4), and drops to 0% (5x5). - **PAL**: Starts at ~95% (3x3), declines to ~80% (5x5). #### Magic-Square - **SymPro-LM**: Flat line at 100% across all board sizes. - **Logic-LM**: Starts at ~65% (3x3), declines to ~30% (4x4), and drops to ~5% (5x5). - **Few-Shot**: Begins at ~20% (3x3), declines to ~0% (4x4), and remains at 0% (5x5). - **PAL**: Starts at ~95% (3x3), drops sharply to 0% (4x4), and remains at 0% (5x5). ### Key Observations 1. **SymPro-LM Dominance**: Achieves 100% accuracy universally, indicating perfect performance across all games and board sizes. 2. **Logic-LM Degradation**: Shows strong initial performance but degrades significantly with increasing board complexity. 3. **PAL Instability**: Performs well initially but collapses entirely for larger boards in Magic-Square and Sujiko. 4. **Few-Shot Limitations**: Struggles across all games, with near-zero accuracy for boards larger than 4x4. ### Interpretation The data suggests SymPro-LM is the most robust algorithm, maintaining perfect accuracy regardless of puzzle complexity. Logic-LM and PAL exhibit initial competence but fail to scale, while Few-Shot demonstrates the poorest performance. This pattern implies that SymPro-LM's design inherently handles complexity better, whereas other algorithms may lack adaptive mechanisms for larger problem spaces. The sharp declines in Logic-LM and PAL for Magic-Square (5x5) highlight particular vulnerabilities to high-dimensional constraint satisfaction problems.