## Line Charts: PrOntoQA Ablation Study ### Overview The image presents a series of line charts from a PrOntoQA Ablation Study. The charts are arranged in a 3x3 grid, with each row representing a different percentage of training data (2%, 5%, and 90%) and each column representing a different type of input (Commonsense, Anticommonsense, and Noncommonsense). Each chart displays the "Score" (y-axis) versus the "Number of Epochs" (x-axis) for three different CAPT settings: null, order, and random. ### Components/Axes * **Title:** PrOntoQA Ablation Study * **X-axis:** Number of Epochs, with markers at 0, 1000, 2000, and 3000. * **Y-axis:** Score, ranging from 70 to 100 (depending on the specific chart). * **CAPT Setting Legend (Top-Right):** * Blue: CAPT=null * Green: CAPT=order * Orange: CAPT=random * **Row Labels:** * Commonsense - 2% Training (Top-Left) * Anticommonsense - 2% Training (Top-Middle) * Noncommonsense - 2% Training (Top-Right) * Commonsense - 5% Training (Middle-Left) * Anticommonsense - 5% Training (Middle-Middle) * Noncommonsense - 5% Training (Middle-Right) * Commonsense - 90% Training (Bottom-Left) * Anticommonsense - 90% Training (Bottom-Middle) * Noncommonsense - 90% Training (Bottom-Right) ### Detailed Analysis **Commonsense - 2% Training:** * **CAPT=null (Blue):** Starts around 80, rises sharply to approximately 98 by 250 epochs, then remains relatively stable around 98-100 until 3000 epochs. * **CAPT=order (Green):** Starts around 78, increases to approximately 83 by 250 epochs, then gradually increases to approximately 88 by 3000 epochs. * **CAPT=random (Orange):** Starts around 82, increases to approximately 86 by 250 epochs, then gradually increases to approximately 90 by 3000 epochs. **Anticommonsense - 2% Training:** * **CAPT=null (Blue):** Starts around 75, dips to approximately 72 by 500 epochs, then rises to approximately 75 by 3000 epochs. * **CAPT=order (Green):** Starts around 76, increases to approximately 78 by 500 epochs, then remains relatively stable around 75 until 3000 epochs. * **CAPT=random (Orange):** Starts around 82, increases to approximately 84 by 250 epochs, then decreases slightly to approximately 80 by 3000 epochs. **Noncommonsense - 2% Training:** * **CAPT=null (Blue):** Starts around 82, dips to approximately 79 by 1000 epochs, then rises to approximately 81 by 3000 epochs. * **CAPT=order (Green):** Starts around 72, increases to approximately 82 by 500 epochs, then increases to approximately 84 by 3000 epochs. * **CAPT=random (Orange):** Starts around 81, increases to approximately 83 by 500 epochs, then decreases slightly to approximately 81 by 3000 epochs. **Commonsense - 5% Training:** * **CAPT=null (Blue):** Starts around 73, rises sharply to approximately 98 by 250 epochs, then remains relatively stable around 98-100 until 3000 epochs. * **CAPT=order (Green):** Starts around 78, increases to approximately 90 by 500 epochs, then gradually increases to approximately 93 by 3000 epochs. * **CAPT=random (Orange):** Starts around 82, increases to approximately 92 by 1000 epochs, then gradually increases to approximately 94 by 3000 epochs. **Anticommonsense - 5% Training:** * **CAPT=null (Blue):** Starts around 73, dips to approximately 71 by 500 epochs, then rises to approximately 75 by 3000 epochs. * **CAPT=order (Green):** Starts around 82, increases to approximately 83 by 500 epochs, then remains relatively stable around 81 until 3000 epochs. * **CAPT=random (Orange):** Starts around 87, increases to approximately 88 by 250 epochs, then decreases slightly to approximately 80 by 3000 epochs. **Noncommonsense - 5% Training:** * **CAPT=null (Blue):** Starts around 82, dips to approximately 81 by 1000 epochs, then remains relatively stable around 80 until 3000 epochs. * **CAPT=order (Green):** Starts around 80, increases to approximately 86 by 500 epochs, then increases to approximately 88 by 3000 epochs. * **CAPT=random (Orange):** Starts around 83, increases to approximately 90 by 1000 epochs, then increases to approximately 92 by 3000 epochs. **Commonsense - 90% Training:** * **CAPT=null (Blue):** Starts around 99, remains relatively stable around 99-100 until 3000 epochs. * **CAPT=order (Green):** Starts around 81, increases to approximately 95 by 500 epochs, then gradually increases to approximately 98 by 3000 epochs. * **CAPT=random (Orange):** Starts around 81, increases to approximately 98 by 1000 epochs, then remains relatively stable around 99 until 3000 epochs. **Anticommonsense - 90% Training:** * **CAPT=null (Blue):** Starts around 73, dips to approximately 70 by 500 epochs, then rises to approximately 74 by 3000 epochs. * **CAPT=order (Green):** Starts around 80, increases to approximately 90 by 500 epochs, then remains relatively stable around 91 until 3000 epochs. * **CAPT=random (Orange):** Starts around 80, increases to approximately 90 by 500 epochs, then remains relatively stable around 92 until 3000 epochs. **Noncommonsense - 90% Training:** * **CAPT=null (Blue):** Starts around 83, increases to approximately 87 by 500 epochs, then decreases slightly to approximately 84 by 3000 epochs. * **CAPT=order (Green):** Starts around 80, increases to approximately 93 by 500 epochs, then remains relatively stable around 95 until 3000 epochs. * **CAPT=random (Orange):** Starts around 80, increases to approximately 95 by 500 epochs, then remains relatively stable around 96 until 3000 epochs. ### Key Observations * For "Commonsense" data, the "CAPT=null" setting (blue line) consistently achieves the highest scores, especially with higher percentages of training data (5% and 90%). * For "Anticommonsense" data, the "CAPT=null" setting (blue line) consistently performs the worst. * For "Noncommonsense" data, the "CAPT=random" setting (orange line) generally performs well, especially with higher percentages of training data (5% and 90%). * Increasing the percentage of training data generally improves the scores for all CAPT settings, but the effect is most pronounced for "Commonsense" data. * The "CAPT=null" setting seems to benefit the most from increased training data in the "Commonsense" category. ### Interpretation The data suggests that the "CAPT=null" setting is highly effective for "Commonsense" data, indicating that the model performs best when trained on straightforward, logical information. Conversely, the poor performance of "CAPT=null" on "Anticommonsense" data suggests that the model struggles with contradictory or illogical information when no CAPT is applied. The "CAPT=random" setting appears to be a good compromise for "Noncommonsense" data, providing a balance between performance and robustness. The ablation study demonstrates the importance of the CAPT setting and the type of training data used. The results highlight the model's sensitivity to the nature of the input and the need for appropriate CAPT strategies to handle different types of information. The significant improvement in scores with increased training data, particularly for "Commonsense" data with "CAPT=null", underscores the value of high-quality, relevant training data for achieving optimal model performance.

## Line Charts: PrOntoQA Ablation Study ### Overview The image presents a 3x3 grid of line charts, visualizing the results of an ablation study for the PrOntoQA model. Each chart represents a different training data composition (Commonsense, Anticommonsense, Noncommonsense) combined with different percentages of training data (2%, 5%, 90%). The charts plot 'Score' against 'Number of Epochs' for various 'CAPT Setting' configurations. ### Components/Axes * **X-axis:** Number of Epochs (ranging from 0 to 3000, with markers at 0, 1000, 2000, and 3000). * **Y-axis:** Score (ranging from approximately 70 to 100, with markers at 70, 75, 80, 85, 90, 95, and 100). * **Legend:** Located in the top-right corner of each chart, defining the lines: * CAPT=null (represented by a blue line with circle markers) * CAPT=ordinal (represented by a green line with circle markers) * CAPT=random (represented by an orange line with circle markers) * **Titles:** Each chart has a title indicating the data composition and training percentage (e.g., "Commonsense - 2% Training"). * **Overall Title:** "PrOntoQA Ablation Study" is positioned at the top-left of the entire image. ### Detailed Analysis or Content Details **Chart 1: Commonsense - 2% Training** * CAPT=null: Line starts at approximately 98, decreases to around 82 at 1000 epochs, then increases slightly to around 84 at 3000 epochs. * CAPT=ordinal: Line starts at approximately 95, decreases to around 80 at 1000 epochs, then remains relatively stable around 81-82 at 3000 epochs. * CAPT=random: Line starts at approximately 83, increases to around 92 at 1000 epochs, then decreases to around 88 at 3000 epochs. **Chart 2: Anticommonsense - 2% Training** * CAPT=null: Line starts at approximately 78, decreases to around 72 at 1000 epochs, then increases to around 75 at 3000 epochs. * CAPT=ordinal: Line starts at approximately 75, remains relatively stable around 75-76 throughout the epochs. * CAPT=random: Line starts at approximately 72, increases to around 78 at 1000 epochs, then decreases to around 74 at 3000 epochs. **Chart 3: Noncommonsense - 2% Training** * CAPT=null: Line starts at approximately 84, decreases to around 78 at 1000 epochs, then remains relatively stable around 78-80 at 3000 epochs. * CAPT=ordinal: Line starts at approximately 82, decreases to around 78 at 1000 epochs, then remains relatively stable around 78-79 at 3000 epochs. * CAPT=random: Line starts at approximately 85, decreases to around 80 at 1000 epochs, then increases to around 82 at 3000 epochs. **Chart 4: Commonsense - 5% Training** * CAPT=null: Line starts at approximately 98, decreases to around 83 at 1000 epochs, then increases to around 86 at 3000 epochs. * CAPT=ordinal: Line starts at approximately 94, increases to around 96 at 1000 epochs, then remains relatively stable around 95-96 at 3000 epochs. * CAPT=random: Line starts at approximately 83, increases to around 93 at 1000 epochs, then decreases to around 90 at 3000 epochs. **Chart 5: Anticommonsense - 5% Training** * CAPT=null: Line starts at approximately 82, decreases to around 75 at 1000 epochs, then increases to around 78 at 3000 epochs. * CAPT=ordinal: Line starts at approximately 75, increases to around 85 at 1000 epochs, then decreases to around 82 at 3000 epochs. * CAPT=random: Line starts at approximately 72, increases to around 82 at 1000 epochs, then remains relatively stable around 82-83 at 3000 epochs. **Chart 6: Noncommonsense - 5% Training** * CAPT=null: Line starts at approximately 87, decreases to around 82 at 1000 epochs, then remains relatively stable around 82-83 at 3000 epochs. * CAPT=ordinal: Line starts at approximately 85, decreases to around 80 at 1000 epochs, then remains relatively stable around 80-81 at 3000 epochs. * CAPT=random: Line starts at approximately 86, increases to around 88 at 1000 epochs, then decreases to around 85 at 3000 epochs. **Chart 7: Commonsense - 90% Training** * CAPT=null: Line starts at approximately 98, remains relatively stable around 98-99 throughout the epochs. * CAPT=ordinal: Line starts at approximately 96, increases to around 99 at 1000 epochs, then remains relatively stable around 99-100 at 3000 epochs. * CAPT=random: Line starts at approximately 84, increases to around 96 at 1000 epochs, then remains relatively stable around 96-97 at 3000 epochs. **Chart 8: Anticommonsense - 90% Training** * CAPT=null: Line starts at approximately 75, increases to around 85 at 1000 epochs, then remains relatively stable around 85-86 at 3000 epochs. * CAPT=ordinal: Line starts at approximately 72, increases to around 90 at 1000 epochs, then remains relatively stable around 90-91 at 3000 epochs. * CAPT=random: Line starts at approximately 70, increases to around 90 at 1000 epochs, then remains relatively stable around 90-91 at 3000 epochs. **Chart 9: Noncommonsense - 90% Training** * CAPT=null: Line starts at approximately 88, increases to around 93 at 1000 epochs, then remains relatively stable around 93-94 at 3000 epochs. * CAPT=ordinal: Line starts at approximately 86, increases to around 95 at 1000 epochs, then remains relatively stable around 95-96 at 3000 epochs. * CAPT=random: Line starts at approximately 85, increases to around 94 at 1000 epochs, then remains relatively stable around 94-95 at 3000 epochs. ### Key Observations * Generally, performance (Score) decreases initially with increasing epochs, then plateaus or slightly increases. * The 90% training data consistently yields the highest scores across all CAPT settings. * The 'CAPT=ordinal' and 'CAPT=random' settings often outperform 'CAPT=null', especially with higher training data percentages. * The 2% training data shows the most significant fluctuations in score across epochs. * Anticommonsense data consistently shows lower scores compared to Commonsense and Noncommonsense data. ### Interpretation The ablation study investigates the impact of different training data compositions and CAPT settings on the PrOntoQA model's performance. The results suggest that: 1. **Data Quantity Matters:** Increasing the training data percentage (from 2% to 90%) significantly improves the model's score, indicating that more training data leads to better generalization. 2. **CAPT Setting is Important:** The choice of CAPT setting influences performance. 'CAPT=ordinal' and 'CAPT=random' generally outperform 'CAPT=null', suggesting that incorporating some form of contextual information during training is beneficial. 3. **Data Type Affects Performance:** Commonsense and Noncommonsense data lead to higher scores than Anticommonsense data, implying that the model struggles with reasoning about contradictory or unusual scenarios. 4. **Initial Fluctuations:** The initial decrease in score with increasing epochs might be due to the model adjusting to the training data and overcoming initial overfitting. The subsequent plateau or slight increase suggests that the model is converging towards a stable solution. The study provides valuable insights into the design and training of the PrOntoQA model, highlighting the importance of data quantity, CAPT setting, and data type for achieving optimal performance. The observed trends suggest that the model benefits from contextual information and struggles with reasoning about uncommon scenarios.

## [Chart Set]: PrOntoQA Ablation Study - CAPT Setting Performance ### Overview The image displays a 3x3 grid of line charts from the "PrOntoQA Ablation Study." The charts compare the performance (Score) of three different "CAPT Settings" (null, order, random) over the course of training (Number of Epochs). The analysis is segmented by two variables: the type of reasoning task (Commonsense, Anticommonsense, Noncommonsense) and the percentage of training data used (2%, 5%, 90%). ### Components/Axes * **Overall Title:** "PrOntoQA Ablation Study" (Top-left corner of the entire figure). * **Chart Grid:** 3 rows (Training Data Percentage: 2%, 5%, 90%) x 3 columns (Task Type: Commonsense, Anticommonsense, Noncommonsense). * **Individual Chart Titles:** Each subplot has a title in the format `[Task Type] - [Training %] Training`. * **X-Axis (All Charts):** "Number of Epochs". The scale is linear, with major ticks at 0, 1000, 2000, and 3000. * **Y-Axis (All Charts):** "Score". The scale is linear but varies per chart to best display the data range (e.g., 70-100, 75-85, 80-95). * **Legend:** Located to the right of the top-right chart. It defines the three data series: * `CAPT=null` (Blue line with circle markers) * `CAPT=order` (Green line with square markers) * `CAPT=random` (Orange line with diamond markers) ### Detailed Analysis The following analysis breaks down each chart by row (training percentage) and column (task type). For each, the visual trend for each CAPT setting is described, followed by approximate score values at key epochs (0, ~250, ~500, ~1000, ~1500, ~3200). **Row 1: 2% Training Data** * **Commonsense - 2% Training (Top-Left):** * `CAPT=null` (Blue): Starts high (~97), quickly peaks near 100, and remains stable at ~100. * `CAPT=order` (Green): Starts low (~73), rises sharply to ~85 by epoch 250, then fluctuates slightly, ending near ~83. * `CAPT=random` (Orange): Starts around ~80, rises to ~87 by epoch 250, then gradually increases to ~88. * **Anticommonsense - 2% Training (Top-Center):** * `CAPT=null` (Blue): Starts at ~75, peaks at ~77, then declines to a low of ~71 before a slight recovery to ~72. * `CAPT=order` (Green): Starts at ~70, rises to ~79, then gradually declines to ~75. * `CAPT=random` (Orange): Starts at ~76, rises sharply to ~83, dips slightly, then climbs to a final high of ~83. * **Noncommonsense - 2% Training (Top-Right):** * `CAPT=null` (Blue): Starts at ~80, peaks at ~84, then declines to ~78. * `CAPT=order` (Green): Starts at ~70, rises steeply to ~83, then fluctuates, ending at ~83. * `CAPT=random` (Orange): Starts at ~84, dips to ~80, then recovers to ~81. **Row 2: 5% Training Data** * **Commonsense - 5% Training (Middle-Left):** * `CAPT=null` (Blue): Starts at ~97, quickly reaches and plateaus at ~100. * `CAPT=order` (Green): Starts at ~78, rises steadily to ~93. * `CAPT=random` (Orange): Starts at ~85, rises to ~95, then stabilizes around ~94. * **Anticommonsense - 5% Training (Middle-Center):** * `CAPT=null` (Blue): Starts at ~72, peaks at ~77, then declines to ~70 before a slow rise to ~75. * `CAPT=order` (Green): Starts at ~72, rises to ~78, then continues a steady climb to ~80. * `CAPT=random` (Orange): Starts at ~77, rises sharply to ~88, and remains stable at that level. * **Noncommonsense - 5% Training (Middle-Right):** * `CAPT=null` (Blue): Starts at ~83, peaks at ~85, then declines to ~80. * `CAPT=order` (Green): Starts at ~78, rises to ~85, dips slightly, then climbs to ~87. * `CAPT=random` (Orange): Starts at ~79, rises steeply to ~92, and remains stable. **Row 3: 90% Training Data** * **Commonsense - 90% Training (Bottom-Left):** * `CAPT=null` (Blue): Starts at ~96, quickly reaches and stays at ~100. * `CAPT=order` (Green): Starts at ~80, rises sharply to ~98, and stabilizes. * `CAPT=random` (Orange): Starts at ~79, rises sharply to ~99, and stabilizes. * **Anticommonsense - 90% Training (Bottom-Center):** * `CAPT=null` (Blue): Starts at ~68, peaks at ~76, then declines to ~70 before a slow rise to ~74. * `CAPT=order` (Green): Starts at ~78, rises to ~90, and continues a slow climb to ~91. * `CAPT=random` (Orange): Starts at ~77, rises sharply to ~91, and remains stable. * **Noncommonsense - 90% Training (Bottom-Right):** * `CAPT=null` (Blue): Starts at ~80, rises to ~87, then declines to ~84. * `CAPT=order` (Green): Starts at ~81, rises sharply to ~95, and remains stable. * `CAPT=random` (Orange): Starts at ~82, rises sharply to ~97, and remains stable. ### Key Observations 1. **CAPT=null (Blue) Performance:** This setting consistently shows the poorest or most volatile performance, especially on Anticommonsense and Noncommonsense tasks. It often peaks early and then degrades or stagnates. It performs best on Commonsense tasks, where it quickly reaches a ceiling. 2. **CAPT=random (Orange) Dominance:** This setting is the top performer in almost every chart, particularly for Anticommonsense and Noncommonsense tasks. It shows rapid initial learning and achieves the highest final scores. 3. **CAPT=order (Green) Performance:** This setting generally performs better than `null` but worse than `random`. Its performance improves significantly as the training data percentage increases (from 2% to 90%). 4. **Impact of Training Data Size:** All models show improved performance and stability with more training data (90% vs. 2%). The gap between `order`/`random` and `null` widens with more data on the harder tasks (Anticommonsense, Noncommonsense). 5. **Task Difficulty:** Commonsense tasks appear easiest, with scores clustering near 100. Anticommonsense tasks show the lowest overall scores and the most significant performance differences between CAPT settings. ### Interpretation This ablation study investigates the impact of different "CAPT" (likely a form of data augmentation or training curriculum) strategies on the PrOntoQA model's reasoning abilities. The data suggests several key insights: * **The `random` CAPT strategy is highly effective.** Its consistent superiority, especially on non-intuitive (Anticommonsense) and novel (Noncommonsense) reasoning tasks, indicates that introducing randomized elements during training helps the model generalize better and avoid overfitting to simplistic patterns. * **The `null` strategy (no CAPT) is insufficient.** Its poor and unstable performance, particularly with limited data, suggests the base training procedure is brittle. The early peak and subsequent decline on harder tasks may indicate catastrophic forgetting or an inability to escape local minima. * **Structured augmentation (`order`) has moderate benefits.** While better than nothing, a fixed-order strategy is less effective than a randomized one. This implies that the diversity and unpredictability of the training signal are more important than a rigid structure for building robust reasoning skills. * **Data efficiency is strategy-dependent.** With only 2% of training data, the choice of CAPT strategy is critical, as seen in the large performance gaps. With 90% data, the `order` and `random` strategies converge to high performance, but `null` still lags, showing that better training methodologies can compensate for data scarcity to a significant degree. In summary, the charts provide strong evidence that incorporating randomized CAPT strategies during training is crucial for developing robust and generalizable reasoning capabilities in the PrOntoQA model, particularly when dealing with limited data or tasks that require non-commonsense logic.

## Heatmap: ProOntoQA Ablation Study ### Overview The heatmap illustrates the performance of ProOntoQA under different training settings and ablation studies. Each row represents a different training setting, and each column represents a different ablation study. The color intensity indicates the score, with darker shades representing higher scores. ### Components/Axes - **Rows**: Represent different training settings (Commonsense - 2%, Anticommense - 2%, Noncommense - 2%, Commonsense - 5%, Anticommense - 5%, Noncommense - 5%, Commonsense - 90%, Anticommense - 90%, Noncommense - 90%). - **Columns**: Represent different ablation studies (CAPT Setting, CAPT=null, CAPT=order, CAPT=random). - **X-Axis**: Represents the number of epochs. - **Y-Axis**: Represents the score. ### Detailed Analysis or ### Content Details - **Commonsense - 2% Training**: The highest scores are observed in the CAPT Setting and CAPT=order, with scores peaking around 95. - **Anticommense - 2% Training**: The scores are generally lower, with the highest score around 85 in the CAPT Setting. - **Noncommense - 2% Training**: Similar to Anticommense, the highest score is around 85 in the CAPT Setting. - **Commonsense - 5% Training**: The scores are slightly higher than in the 2% training setting, with the highest score around 90 in the CAPT Setting. - **Anticommense - 5% Training**: The scores are similar to the 2% training setting, with the highest score around 85 in the CAPT Setting. - **Noncommense - 5% Training**: The scores are slightly higher than in the 2% training setting, with the highest score around 85 in the CAPT Setting. - **Commonsense - 90% Training**: The scores are consistently high, with the highest score around 95 in the CAPT Setting. - **Anticommense - 90% Training**: The scores are similar to the 2% and 5% training settings, with the highest score around 85 in the CAPT Setting. - **Noncommense - 90% Training**: The scores are slightly higher than in the 2% and 5% training settings, with the highest score around 85 in the CAPT Setting. ### Key Observations - The CAPT Setting consistently yields the highest scores across all training settings and ablation studies. - The scores are highest when the model is trained with 90% of the data. - The scores are lowest when the model is trained with 2% of the data and the CAPT Setting is used. ### Interpretation The heatmap suggests that the CAPT Setting is the most effective training method for ProOntoQA, regardless of the amount of data used or the ablation studies conducted. The highest scores are achieved when the model is trained with 90% of the data, indicating that a larger dataset is beneficial for the model's performance. The ablation studies show that the presence of common sense and the specific training settings (CAPT Setting, CAPT=null, CAPT=order, CAPT=random) do not significantly impact the model's performance.

## Line Chart Grid: PrOntoQA Ablation Study Performance Across Training Percentages ### Overview The image displays a 3x3 grid of line charts comparing model performance across three datasets (Commonsense, Anticommonsense, Noncommonsense) at three training percentages (2%, 5%, 90%). Each chart tracks scores over 3,000 training epochs, with three performance baselines represented by colored lines (blue: CAPT=null, green: CAPT=order, orange: CAPT=random). ### Components/Axes - **X-axis**: Number of Epochs (0–3,000 in increments of 1,000) - **Y-axis**: Score (70–100) - **Legend**: Located in top-right corner of each chart, with color-coded labels: - Blue: CAPT=null - Green: CAPT=order - Orange: CAPT=random - **Chart Titles**: Positioned in top-left of each subplot (e.g., "Commonsense - 2% Training") ### Detailed Analysis #### Commonsense - 2% Training - **Blue (null)**: Starts at ~95, fluctuates slightly, stabilizes near 98–100 - **Green (order)**: Begins at ~80, rises steadily to ~88 by 3,000 epochs - **Orange (random)**: Starts at ~80, peaks at ~87, then plateaus #### Anticommonsense - 2% Training - **Blue (null)**: Drops from ~95 to ~70 by 1,000 epochs, then recovers to ~75 - **Green (order)**: Starts at ~75, rises to ~82, then plateaus - **Orange (random)**: Begins at ~85, peaks at ~88, then declines slightly #### Noncommonsense - 2% Training - **Blue (null)**: Starts at ~85, dips to ~75, then recovers to ~82 - **Green (order)**: Begins at ~70, rises to ~85, then stabilizes - **Orange (random)**: Starts at ~75, peaks at ~88, then declines slightly #### Commonsense - 5% Training - **Blue (null)**: Starts at ~98, fluctuates minimally, stabilizes near 100 - **Green (order)**: Begins at ~85, rises to ~92, then plateaus - **Orange (random)**: Starts at ~85, peaks at ~93, then declines slightly #### Anticommonsense - 5% Training - **Blue (null)**: Drops from ~95 to ~75, then recovers to ~80 - **Green (order)**: Starts at ~80, rises to ~86, then plateaus - **Orange (random)**: Begins at ~85, peaks at ~89, then stabilizes #### Noncommonsense - 5% Training - **Blue (null)**: Starts at ~85, dips to ~78, then recovers to ~83 - **Green (order)**: Begins at ~75, rises to ~87, then stabilizes - **Orange (random)**: Starts at ~80, peaks at ~90, then declines slightly #### Commonsense - 90% Training - **Blue (null)**: Starts at ~95, fluctuates minimally, stabilizes near 100 - **Green (order)**: Begins at ~90, rises to ~98, then plateaus - **Orange (random)**: Starts at ~85, peaks at ~97, then stabilizes #### Anticommonsense - 90% Training - **Blue (null)**: Drops from ~95 to ~70, then recovers to ~80 - **Green (order)**: Starts at ~85, rises to ~92, then plateaus - **Orange (random)**: Begins at ~85, peaks at ~93, then stabilizes #### Noncommonsense - 90% Training - **Blue (null)**: Starts at ~85, dips to ~75, then recovers to ~82 - **Green (order)**: Begins at ~80, rises to ~95, then stabilizes - **Orange (random)**: Starts at ~85, peaks at ~96, then stabilizes ### Key Observations 1. **Training Percentage Impact**: Higher training percentages (90%) consistently yield higher scores across all datasets and CAPT settings. 2. **CAPT Setting Performance**: - **CAPT=null** performs best in Commonsense (90% training) but underperforms in Anticommonsense/Noncommonsense. - **CAPT=random** often outperforms CAPT=order in Anticommonsense and Noncommonsense datasets. - **CAPT=order** shows stable but moderate performance across all datasets. 3. **Epochs Correlation**: All lines generally trend upward with increasing epochs, though some exhibit mid-training dips (e.g., blue lines in Anticommonsense charts). ### Interpretation The data suggests that: - **Training Scale Matters**: 90% training achieves near-perfect scores in Commonsense, while lower percentages struggle with generalization. - **CAPT Strategy Tradeoffs**: - CAPT=null excels in high-resource scenarios (90% training) but fails in low-resource settings. - CAPT=random adapts better to challenging datasets (Anticommonsense/Noncommonsense) but shows instability in early epochs. - CAPT=order provides consistent but suboptimal performance across all settings. - **Dataset Complexity**: Anticommonsense and Noncommonsense datasets require more sophisticated CAPT strategies to achieve high scores, indicating greater semantic complexity. This analysis highlights the importance of CAPT configuration selection based on both dataset type and available training data.