## Line Chart: Evaluation on Task ### Overview The image displays a line chart titled "Evaluation on Task," comparing the performance of multiple continual learning methods across a sequence of eight tasks (T1 to T8). The chart plots "Accuracy %" on the y-axis against "Training Sequence Per Task" on the x-axis. Each method is represented by a distinct colored line, showing how its accuracy changes as it is sequentially trained on new tasks. The legend at the top provides the method names, their average accuracy across tasks, and a standard deviation value in parentheses. ### Components/Axes * **Chart Title:** "Evaluation on Task" (centered at the top). * **Y-Axis:** Labeled "Accuracy %". Scale ranges from 0 to 100 in increments of 20. * **X-Axis:** Labeled "Training Sequence Per Task". It is divided into eight discrete sections, labeled T1 through T8 from left to right. Each section represents the evaluation point after training on that specific task. * **Legend:** Positioned at the very top of the chart, above the plot area. It lists 10 methods with their associated line styles, colors, markers, and summary statistics (average accuracy and standard deviation). * `finetuning`: 23.11 (43.74) - Black dotted line with circle markers. * `PackNet`: 64.88 (0.00) - Green solid line with star markers. * `EWC`: 42.01 (19.84) - Yellow solid line with no distinct marker. * `LwF`: 30.59 (32.76) - Light blue solid line with no distinct marker. * `mean-IMM`: 31.43 (33.63) - Brown solid line with no distinct marker. * `joint*`: 65.82 (n/a) - Gray right-pointing triangle marker (appears as isolated points, not a connected line). * `SI`: 43.40 (18.23) - Orange solid line with no distinct marker. * `MAS`: 45.72 (13.08) - Red solid line with no distinct marker. * `EBLL`: 33.82 (29.26) - Dark blue solid line with no distinct marker. * `mode-IMM`: 34.45 (2.76) - Dark red/brown solid line with no distinct marker. ### Detailed Analysis The chart shows the accuracy trajectory for each method across the task sequence. Below is a task-by-task breakdown of the visible trends and approximate data points. * **T1 (Initial Task):** All methods begin with high accuracy, clustered between approximately 75% and 85%. The `joint*` baseline (gray triangle) is at the top (~85%). `PackNet` (green) starts around 80%. Most other methods start near 80% but show an immediate, sharp decline within the T1 evaluation window. * **T2:** Performance for most methods has dropped significantly from T1. `PackNet` remains high, near 80%. `joint*` is around 55%. The other methods have fallen into a range of roughly 20% to 55%, with `finetuning` (black dotted) showing one of the steepest drops to below 20%. * **T3:** The downward trend continues for most. `PackNet` holds steady near 80%. `joint*` is around 50%. The cluster of other methods now lies between approximately 10% and 45%. * **T4:** A notable pattern emerges. `PackNet` maintains its high accuracy (~80%). `joint*` is near 60%. Several methods (e.g., `LwF`-light blue, `EBLL`-dark blue) show a temporary recovery or smaller drop compared to T3, clustering around 40-60%. `finetuning` remains very low (<20%). * **T5:** Similar to T4, but the cluster of recovering methods is slightly lower (30-55%). `PackNet` is still near 80%. `joint*` is around 50%. * **T6:** Performance for most non-`PackNet` methods converges into a tighter, lower band between roughly 20% and 40%. `PackNet` dips slightly but remains above 70%. `joint*` is near 40%. * **T7:** This task shows a dramatic, sharp spike in accuracy for several methods, most prominently `LwF` (light blue) and `EBLL` (dark blue), which jump to over 80%. `MAS` (red) and `SI` (orange) also spike to around 70%. `PackNet` remains high (~80%). `joint*` is near 85%. `finetuning` shows a smaller spike to ~40%. * **T8:** Data is sparse. Only a few isolated points are visible: `joint*` near 95%, `PackNet` near 80%, and `finetuning` near 20%. The lines for other methods do not extend to T8, suggesting they may have been evaluated only up to T7 or their data points are not plotted. ### Key Observations 1. **PackNet's Stability:** The `PackNet` method (green line) demonstrates remarkable stability, maintaining high accuracy (near 80%) across all tasks T1-T7 with minimal degradation. This is reflected in its low standard deviation (0.00). 2. **Catastrophic Forgetting:** Most other methods exhibit classic catastrophic forgetting. Their accuracy drops sharply after the initial task (T1) and generally trends downward through T6, indicating they forget previously learned tasks as new ones are learned. 3. **The T7 Anomaly:** Task T7 causes a significant, unexpected performance increase for several methods (`LwF`, `EBLL`, `MAS`, `SI`). This suggests T7 might be an easier task, or there is a positive transfer effect from previously learned tasks for these specific algorithms. 4. **`joint*` Baseline:** The `joint*` method (gray triangles), which likely represents an upper-bound joint training on all tasks seen so far, generally performs well but shows variability, dropping in mid-sequence tasks (T3, T6) before recovering. 5. **`finetuning` Collapse:** Simple `finetuning` (black dotted line) performs the worst, showing the most severe and rapid forgetting, with accuracy often below 20% after the first task. ### Interpretation This chart is a comparative evaluation of continual learning algorithms designed to mitigate catastrophic forgetting. The data clearly demonstrates the core challenge: as a model sequentially learns new tasks (T1→T8), its performance on earlier tasks typically degrades. * **Method Efficacy:** `PackNet` appears highly effective in this specific evaluation setup, as it prevents forgetting almost entirely (near-zero variance in accuracy). This suggests it successfully isolates or protects parameters for previous tasks. In contrast, `finetuning` serves as a baseline showing the severe problem these methods aim to solve. * **Task Dependency:** The performance is not uniformly declining. The spike at T7 indicates that task characteristics heavily influence outcomes. Some tasks may reinforce previous knowledge (positive transfer) for certain algorithms, while others cause interference. * **Algorithm Behavior:** Methods like `LwF` and `EBLL` show high volatility—suffering from severe forgetting but also capable of large recoveries (T7). Methods like `MAS` and `SI` show more moderate, consistent forgetting curves. The high standard deviations for most methods (e.g., `finetuning`: 43.74) confirm this high variability in performance across the task sequence. * **Practical Implication:** The choice of continual learning algorithm involves a trade-off. `PackNet` offers stability but may have other costs (e.g., computational, memory). Other methods offer average performance that is highly task-dependent, requiring careful consideration of the expected task sequence. The chart argues that there is no universal best solution; performance is contingent on both the algorithm and the nature of the tasks being learned.