## Line Charts: PrOntoQA Ablation Study ### Overview The image presents a series of line charts from a PrOntoQA Ablation Study. The charts are arranged in a 3x3 grid, with each row representing a different percentage of training data (2%, 5%, and 90%) and each column representing a different type of input (Commonsense, Anticommonsense, and Noncommonsense). Each chart displays the "Score" (y-axis) versus the "Number of Epochs" (x-axis) for three different CAPT settings: null, order, and random. ### Components/Axes * **Title:** PrOntoQA Ablation Study * **X-axis:** Number of Epochs, with markers at 0, 1000, 2000, and 3000. * **Y-axis:** Score, ranging from 70 to 100 (depending on the specific chart). * **CAPT Setting Legend (Top-Right):** * Blue: CAPT=null * Green: CAPT=order * Orange: CAPT=random * **Row Labels:** * Commonsense - 2% Training (Top-Left) * Anticommonsense - 2% Training (Top-Middle) * Noncommonsense - 2% Training (Top-Right) * Commonsense - 5% Training (Middle-Left) * Anticommonsense - 5% Training (Middle-Middle) * Noncommonsense - 5% Training (Middle-Right) * Commonsense - 90% Training (Bottom-Left) * Anticommonsense - 90% Training (Bottom-Middle) * Noncommonsense - 90% Training (Bottom-Right) ### Detailed Analysis **Commonsense - 2% Training:** * **CAPT=null (Blue):** Starts around 80, rises sharply to approximately 98 by 250 epochs, then remains relatively stable around 98-100 until 3000 epochs. * **CAPT=order (Green):** Starts around 78, increases to approximately 83 by 250 epochs, then gradually increases to approximately 88 by 3000 epochs. * **CAPT=random (Orange):** Starts around 82, increases to approximately 86 by 250 epochs, then gradually increases to approximately 90 by 3000 epochs. **Anticommonsense - 2% Training:** * **CAPT=null (Blue):** Starts around 75, dips to approximately 72 by 500 epochs, then rises to approximately 75 by 3000 epochs. * **CAPT=order (Green):** Starts around 76, increases to approximately 78 by 500 epochs, then remains relatively stable around 75 until 3000 epochs. * **CAPT=random (Orange):** Starts around 82, increases to approximately 84 by 250 epochs, then decreases slightly to approximately 80 by 3000 epochs. **Noncommonsense - 2% Training:** * **CAPT=null (Blue):** Starts around 82, dips to approximately 79 by 1000 epochs, then rises to approximately 81 by 3000 epochs. * **CAPT=order (Green):** Starts around 72, increases to approximately 82 by 500 epochs, then increases to approximately 84 by 3000 epochs. * **CAPT=random (Orange):** Starts around 81, increases to approximately 83 by 500 epochs, then decreases slightly to approximately 81 by 3000 epochs. **Commonsense - 5% Training:** * **CAPT=null (Blue):** Starts around 73, rises sharply to approximately 98 by 250 epochs, then remains relatively stable around 98-100 until 3000 epochs. * **CAPT=order (Green):** Starts around 78, increases to approximately 90 by 500 epochs, then gradually increases to approximately 93 by 3000 epochs. * **CAPT=random (Orange):** Starts around 82, increases to approximately 92 by 1000 epochs, then gradually increases to approximately 94 by 3000 epochs. **Anticommonsense - 5% Training:** * **CAPT=null (Blue):** Starts around 73, dips to approximately 71 by 500 epochs, then rises to approximately 75 by 3000 epochs. * **CAPT=order (Green):** Starts around 82, increases to approximately 83 by 500 epochs, then remains relatively stable around 81 until 3000 epochs. * **CAPT=random (Orange):** Starts around 87, increases to approximately 88 by 250 epochs, then decreases slightly to approximately 80 by 3000 epochs. **Noncommonsense - 5% Training:** * **CAPT=null (Blue):** Starts around 82, dips to approximately 81 by 1000 epochs, then remains relatively stable around 80 until 3000 epochs. * **CAPT=order (Green):** Starts around 80, increases to approximately 86 by 500 epochs, then increases to approximately 88 by 3000 epochs. * **CAPT=random (Orange):** Starts around 83, increases to approximately 90 by 1000 epochs, then increases to approximately 92 by 3000 epochs. **Commonsense - 90% Training:** * **CAPT=null (Blue):** Starts around 99, remains relatively stable around 99-100 until 3000 epochs. * **CAPT=order (Green):** Starts around 81, increases to approximately 95 by 500 epochs, then gradually increases to approximately 98 by 3000 epochs. * **CAPT=random (Orange):** Starts around 81, increases to approximately 98 by 1000 epochs, then remains relatively stable around 99 until 3000 epochs. **Anticommonsense - 90% Training:** * **CAPT=null (Blue):** Starts around 73, dips to approximately 70 by 500 epochs, then rises to approximately 74 by 3000 epochs. * **CAPT=order (Green):** Starts around 80, increases to approximately 90 by 500 epochs, then remains relatively stable around 91 until 3000 epochs. * **CAPT=random (Orange):** Starts around 80, increases to approximately 90 by 500 epochs, then remains relatively stable around 92 until 3000 epochs. **Noncommonsense - 90% Training:** * **CAPT=null (Blue):** Starts around 83, increases to approximately 87 by 500 epochs, then decreases slightly to approximately 84 by 3000 epochs. * **CAPT=order (Green):** Starts around 80, increases to approximately 93 by 500 epochs, then remains relatively stable around 95 until 3000 epochs. * **CAPT=random (Orange):** Starts around 80, increases to approximately 95 by 500 epochs, then remains relatively stable around 96 until 3000 epochs. ### Key Observations * For "Commonsense" data, the "CAPT=null" setting (blue line) consistently achieves the highest scores, especially with higher percentages of training data (5% and 90%). * For "Anticommonsense" data, the "CAPT=null" setting (blue line) consistently performs the worst. * For "Noncommonsense" data, the "CAPT=random" setting (orange line) generally performs well, especially with higher percentages of training data (5% and 90%). * Increasing the percentage of training data generally improves the scores for all CAPT settings, but the effect is most pronounced for "Commonsense" data. * The "CAPT=null" setting seems to benefit the most from increased training data in the "Commonsense" category. ### Interpretation The data suggests that the "CAPT=null" setting is highly effective for "Commonsense" data, indicating that the model performs best when trained on straightforward, logical information. Conversely, the poor performance of "CAPT=null" on "Anticommonsense" data suggests that the model struggles with contradictory or illogical information when no CAPT is applied. The "CAPT=random" setting appears to be a good compromise for "Noncommonsense" data, providing a balance between performance and robustness. The ablation study demonstrates the importance of the CAPT setting and the type of training data used. The results highlight the model's sensitivity to the nature of the input and the need for appropriate CAPT strategies to handle different types of information. The significant improvement in scores with increased training data, particularly for "Commonsense" data with "CAPT=null", underscores the value of high-quality, relevant training data for achieving optimal model performance.