## Line Charts: Performance Comparison of Different LLM Configurations ### Overview The image presents a series of five line charts comparing the performance of different Large Language Model (LLM) configurations across various metrics. The x-axis represents the data size (in thousands), and the y-axis represents the performance score for each metric. The charts compare "Single-LLM", "Multi-LLMone-stage", "Single-LLMmulti-task", "α-UMIw/o reuse", and "α-UMIw/ reuse". ### Components/Axes * **X-axis (all charts):** Data size in thousands, labeled with values 12.1k, 31.3k, 47.0k, and 62.7k. * **Y-axis (Plan ACC):** Performance score ranging from 80.0 to 87.5. * **Y-axis (Act. EM):** Performance score ranging from 50 to 60. * **Y-axis (Hallu.):** Performance score ranging from 0 to 8. * **Y-axis (Aug. F1):** Performance score ranging from 40 to 50. * **Y-axis (R-L):** Performance score ranging from 25 to 45. * **Chart Titles:** (a) Plan ACC, (b) Act. EM, (c) Hallu., (d) Aug. F1, (e) R-L. * **Legend (bottom-right):** * Blue line with circle markers: Single-LLM * Dark red dashed line with square markers: Multi-LLMone-stage * Green dashed line with plus markers: Single-LLMmulti-task * Light red dashed line with no markers: α-UMIw/o reuse * Red line with triangle markers: α-UMIw/ reuse ### Detailed Analysis #### (a) Plan ACC (Planning Accuracy) * **Single-LLM (Blue):** Starts at approximately 79 at 12.1k, increases to ~84 at 31.3k, then remains relatively stable around 84 until 62.7k. * **Multi-LLMone-stage (Dark Red):** Starts at ~83 at 12.1k, increases to ~87 at 31.3k, then decreases slightly to ~87 at 62.7k. * **Single-LLMmulti-task (Green):** Starts at ~83 at 12.1k, increases to ~86 at 31.3k, then remains relatively stable around 86 until 62.7k. * **α-UMIw/o reuse (Light Red):** Starts at ~83 at 12.1k, increases to ~87 at 31.3k, then remains relatively stable around 87 until 62.7k. * **α-UMIw/ reuse (Red):** Starts at ~84 at 12.1k, increases to ~87 at 31.3k, then increases to ~88 at 62.7k. #### (b) Act. EM (Action Exact Match) * **Single-LLM (Blue):** Starts at ~51 at 12.1k, increases to ~54 at 31.3k, then decreases to ~52 at 47.0k, and increases to ~54 at 62.7k. * **Multi-LLMone-stage (Dark Red):** Starts at ~47 at 12.1k, increases to ~52 at 31.3k, then decreases to ~52 at 47.0k, and decreases to ~48 at 62.7k. * **Single-LLMmulti-task (Green):** Starts at ~50 at 12.1k, increases to ~50 at 31.3k, then remains relatively stable around 50 until 62.7k. * **α-UMIw/o reuse (Light Red):** Starts at ~50 at 12.1k, increases to ~57 at 31.3k, then decreases to ~57 at 47.0k, and decreases to ~57 at 62.7k. * **α-UMIw/ reuse (Red):** Starts at ~54 at 12.1k, increases to ~58 at 31.3k, then increases to ~58 at 47.0k, and increases to ~59 at 62.7k. #### (c) Hallu. (Hallucination) * **Single-LLM (Blue):** Starts at ~2 at 12.1k, increases to ~3 at 31.3k, then decreases to ~1 at 47.0k, and increases to ~2 at 62.7k. * **Multi-LLMone-stage (Dark Red):** Starts at ~5 at 12.1k, decreases to ~1 at 31.3k, then increases to ~2 at 47.0k, and increases to ~6 at 62.7k. * **Single-LLMmulti-task (Green):** Starts at ~4 at 12.1k, increases to ~8 at 31.3k, then decreases to ~5 at 47.0k, and decreases to ~3 at 62.7k. * **α-UMIw/o reuse (Light Red):** Starts at ~4 at 12.1k, decreases to ~1 at 31.3k, then remains relatively stable around 1 until 62.7k. * **α-UMIw/ reuse (Red):** Starts at ~1 at 12.1k, increases to ~1 at 31.3k, then remains relatively stable around 1 until 62.7k. #### (d) Aug. F1 (Augmented F1 Score) * **Single-LLM (Blue):** Starts at ~36 at 12.1k, increases to ~46 at 31.3k, then remains relatively stable around 46 until 62.7k. * **Multi-LLMone-stage (Dark Red):** Starts at ~40 at 12.1k, increases to ~42 at 31.3k, then remains relatively stable around 42 until 62.7k. * **Single-LLMmulti-task (Green):** Starts at ~44 at 12.1k, decreases to ~42 at 31.3k, then remains relatively stable around 42 until 62.7k. * **α-UMIw/o reuse (Light Red):** Starts at ~43 at 12.1k, increases to ~43 at 31.3k, then remains relatively stable around 43 until 62.7k. * **α-UMIw/ reuse (Red):** Starts at ~47 at 12.1k, increases to ~51 at 31.3k, then remains relatively stable around 51 until 62.7k. #### (e) R-L (Reward Learning) * **Single-LLM (Blue):** Starts at ~36 at 12.1k, increases to ~43 at 31.3k, then increases to ~44 at 47.0k, and increases to ~45 at 62.7k. * **Multi-LLMone-stage (Dark Red):** Starts at ~40 at 12.1k, increases to ~42 at 31.3k, then increases to ~42 at 47.0k, and increases to ~44 at 62.7k. * **Single-LLMmulti-task (Green):** Starts at ~25 at 12.1k, increases to ~33 at 31.3k, then increases to ~37 at 47.0k, and decreases to ~30 at 62.7k. * **α-UMIw/o reuse (Light Red):** Starts at ~33 at 12.1k, increases to ~42 at 31.3k, then increases to ~42 at 47.0k, and increases to ~43 at 62.7k. * **α-UMIw/ reuse (Red):** Starts at ~40 at 12.1k, increases to ~43 at 31.3k, then increases to ~43 at 47.0k, and increases to ~45 at 62.7k. ### Key Observations * **α-UMIw/ reuse (Red):** Generally performs well across all metrics, often achieving the highest scores, especially as the data size increases. * **Single-LLMmulti-task (Green):** Shows variable performance, sometimes performing well (e.g., Plan ACC) and sometimes underperforming (e.g., R-L). * **Hallucination (c):** The α-UMIw/ reuse (Red) consistently shows the lowest hallucination rates. ### Interpretation The charts provide a comparative analysis of different LLM configurations, highlighting the impact of various architectural choices and training strategies on performance across different metrics. The "α-UMIw/ reuse" configuration appears to be a strong performer, particularly in terms of planning accuracy, action exact match, and minimizing hallucination. The performance variations across metrics suggest that different configurations are better suited for specific tasks or evaluation criteria. The increase in data size generally leads to improved performance for most configurations, indicating the importance of data scale in LLM training. The "Hallu." chart is particularly important, as it indicates the model's tendency to generate nonsensical or factually incorrect information. Lower scores on this metric are desirable.

\n ## Line Charts: Performance Metrics

## Multi-Panel Line Chart: Model Performance vs. Training Data Size ### Overview The image displays a set of five line charts arranged in a 2x3 grid (with the bottom-right space occupied by a legend). Each chart plots the performance of five different language model architectures against increasing training data sizes. The charts measure distinct evaluation metrics: Plan Accuracy (ACC), Action Exact Match (Act. EM), Hallucination rate (Hallu.), Augmented F1 score (Aug. F1), and ROUGE-L score (R-L). ### Components/Axes * **Common X-Axis (All Charts):** Labeled with training data sizes: `12.1k`, `31.3k`, `47.0k`, `62.7k`. This represents the amount of training data used. * **Y-Axes (Per Chart):** * (a) Plan ACC: Range ~80.0 to ~88.0. * (b) Act. EM: Range ~48 to ~60. * (c) Hallu.: Range ~0 to ~8. Lower values are better. * (d) Aug. F1: Range ~40 to ~53. * (e) R-L: Range ~25 to ~45. * **Legend (Bottom-Right Corner):** Identifies five model series: 1. `Single-LLM` (Blue solid line, circle markers) 2. `Multi-LLM_one-stage` (Brown dashed line, square markers) 3. `Single-LLM_multi-task` (Green dashed line, diamond markers) 4. `α-UMI_w/o reuse` (Light red/pink dashed line, triangle markers) 5. `α-UMI_w/ reuse` (Red solid line, star markers) ### Detailed Analysis **Chart (a) Plan ACC:** * **Trend:** All models show an increase in accuracy as training data grows from 12.1k to 31.3k, after which trends diverge. * **Data Points (Approximate):** * `α-UMI_w/ reuse` (Red): Starts ~83.0, rises to ~87.0, ends highest at ~88.5. * `α-UMI_w/o reuse` (Pink): Follows a similar but slightly lower path than its counterpart, ending ~87.5. * `Multi-LLM_one-stage` (Brown): Starts ~82.5, peaks at ~87.5, ends ~87.8. * `Single-LLM_multi-task` (Green): Starts ~83.5, peaks at ~86.5, then declines to ~85.5. * `Single-LLM` (Blue): Starts lowest at ~79.5, rises to ~84.0, then slightly declines to ~83.0. **Chart (b) Act. EM:** * **Trend:** Performance is more volatile. The `α-UMI_w/ reuse` model shows a strong upward trend. * **Data Points (Approximate):** * `α-UMI_w/ reuse` (Red): Starts ~54.5, dips slightly, then rises sharply to ~59.5. * `α-UMI_w/o reuse` (Pink): Starts ~50.5, rises to ~56.5, then declines to ~56.0. * `Single-LLM` (Blue): Fluctuates between ~52.0 and ~54.5. * `Multi-LLM_one-stage` (Brown): Starts low ~48.5, peaks at ~53.0, then drops to ~48.0. * `Single-LLM_multi-task` (Green): Fluctuates between ~49.5 and ~51.0, ends at ~53.5. **Chart (c) Hallu. (Hallucination Rate):** * **Trend:** Lower is better. The `α-UMI` models show consistently low and decreasing hallucination rates. * **Data Points (Approximate):** * `α-UMI_w/ reuse` (Red): Starts ~1.2, decreases steadily to ~0.5. * `α-UMI_w/o reuse` (Pink): Starts ~1.8, decreases to ~1.0. * `Single-LLM` (Blue): Fluctuates between ~1.5 and ~2.5. * `Multi-LLM_one-stage` (Brown): Starts ~4.0, peaks at ~5.5, ends ~6.0. * `Single-LLM_multi-task` (Green): Starts highest at ~4.5, spikes to ~7.8, then declines to ~3.0. **Chart (d) Aug. F1:** * **Trend:** General upward trend for most models, with `α-UMI_w/ reuse` leading. * **Data Points (Approximate):** * `α-UMI_w/ reuse` (Red): Starts ~48.0, rises to ~50.5, ends highest at ~53.0. * `α-UMI_w/o reuse` (Pink): Follows closely below the red line, ending ~49.5. * `Single-LLM` (Blue): Rises from ~44.0 to ~47.5. * `Multi-LLM_one-stage` (Brown): Starts ~40.5, rises to ~45.0, then drops to ~41.0. * `Single-LLM_multi-task` (Green): Starts ~44.5, dips, then rises to ~46.0. **Chart (e) R-L (ROUGE-L):** * **Trend:** Most models improve with more data. `Single-LLM` and `α-UMI` models perform strongly. * **Data Points (Approximate):** * `Single-LLM` (Blue): Starts ~36.0, rises sharply to ~42.5, ends highest at ~44.5. * `α-UMI_w/ reuse` (Red): Starts ~39.5, rises steadily to ~44.0. * `α-UMI_w/o reuse` (Pink): Starts ~34.0, rises to ~40.0, then plateaus. * `Multi-LLM_one-stage` (Brown): Starts ~33.5, rises to ~42.0, then drops to ~37.0. * `Single-LLM_multi-task` (Green): Starts lowest at ~25.0, rises to ~37.0, then drops sharply to ~29.5. ### Key Observations 1. **Consistent Leader:** The `α-UMI_w/ reuse` model (red line) is the top or near-top performer across all five metrics, showing robust and scalable performance. 2. **Hallucination Control:** Both `α-UMI` variants demonstrate significantly lower and more stable hallucination rates compared to the other three models, especially as data increases. 3. **Multi-Task Instability:** The `Single-LLM_multi-task` model (green) shows high volatility, particularly in Hallucination and R-L scores, where it exhibits sharp peaks and drops. 4. **Data Scaling Benefit:** Most models show clear performance gains when increasing training data from 12.1k to 31.3k. The benefit beyond 31.3k is model- and metric-dependent. 5. **Single-LLM Baseline:** The basic `Single-LLM` (blue) is often outperformed but shows strong, steady improvement in the R-L metric. ### Interpretation This set of charts provides a comparative analysis of model architectures for a task likely involving planning and execution (given metrics like Plan ACC and Act. EM). The data suggests that the proposed `α-UMI` framework, particularly with reuse (`w/ reuse`), offers a superior balance of high accuracy, low hallucination, and strong generalization (as seen in F1 and R-L scores). Its consistent performance across metrics indicates it is a more reliable and scalable approach. The volatility of the `Multi-LLM_one-stage` and `Single-LLM_multi-task` models suggests they may be less stable or more sensitive to the specific training data distribution. The fact that `α-UMI_w/ reuse` often outperforms its no-reuse counterpart (`w/o reuse`) implies that the reuse mechanism within the framework provides a tangible benefit, possibly through more efficient knowledge consolidation or reduced catastrophic forgetting. Overall, the visualization argues for the effectiveness of the `α-UMI` architecture, highlighting its advantages in both performance and stability as the volume of training data increases.

## Line Graphs: Model Performance Across Metrics ### Overview The image contains six line graphs (labeled a-f) comparing the performance of different language model configurations across five metrics: Plan ACC, Act EM, Hallu., Aug. F1, and R-L. Each graph plots performance against dataset size (x-axis: 12.1k–62.7k) with distinct colored lines representing model variants. The legend on the right maps colors to model types. --- ### Components/Axes - **X-Axes**: Dataset size (12.1k, 31.3k, 47.0k, 62.7k) across all graphs. - **Y-Axes**: - (a) Plan ACC: 80–87.5 - (b) Act EM: 2–60 - (c) Hallu.: 2–8 - (d) Aug. F1: 25–50 - (e) R-L: 25–45 - **Legend** (right): - Blue: Single-LLM - Dark red: Multi-LLMone-stage - Green: Single-LLMmulti-task - Light pink: α-UMi w/o reuse - Red: α-UMi w/ reuse --- ### Detailed Analysis #### (a) Plan ACC - **Trends**: - Red (α-UMi w/ reuse) starts at ~85, peaks at 87.5 (47.0k), then drops to ~86. - Blue (Single-LLM) starts at 80, rises to 82.5 (31.3k), then declines to 82.5. - Green (Single-LLMmulti-task) fluctuates between 82.5–85. - **Values**: - At 12.1k: Red ~85, Blue ~80, Green ~82.5. - At 62.7k: Red ~86, Blue ~82.5, Green ~85. #### (b) Act EM - **Trends**: - Red (α-UMi w/ reuse) peaks at 57.5 (31.3k), then drops to ~55. - Blue (Single-LLM) rises to 55 (31.3k), then declines to ~50. - Dark red (Multi-LLMone-stage) fluctuates between 45–55. - **Values**: - At 12.1k: Red ~55, Blue ~50, Dark red ~45. - At 62.7k: Red ~55, Blue ~50, Dark red ~40. #### (c) Hallu. - **Trends**: - Green (Single-LLMmulti-task) starts at 6, drops to 4 (47.0k), then rises to 5. - Red (α-UMi w/ reuse) peaks at 3 (31.3k), then drops to 2. - Light pink (α-UMi w/o reuse) fluctuates between 2–4. - **Values**: - At 12.1k: Green ~6, Red ~2, Light pink ~3. - At 62.7k: Green ~5, Red ~2, Light pink ~3. #### (d) Aug. F1 - **Trends**: - Red (α-UMi w/ reuse) peaks at 50 (31.3k), then drops to ~45. - Blue (Single-LLM) rises to 45 (31.3k), then declines to ~42.5. - Dark red (Multi-LLMone-stage) fluctuates between 35–45. - **Values**: - At 12.1k: Red ~45, Blue ~40, Dark red ~35. - At 62.7k: Red ~45, Blue ~42.5, Dark red ~35. #### (e) R-L - **Trends**: - Red (α-UMi w/ reuse) peaks at 45 (31.3k), then drops to ~40. - Blue (Single-LLM) rises to 40 (31.3k), then declines to ~35. - Green (Single-LLMmulti-task) fluctuates between 30–35. - **Values**: - At 12.1k: Red ~40, Blue ~35, Green ~30. - At 62.7k: Red ~40, Blue ~35, Green ~25. --- ### Key Observations 1. **α-UMi w/ reuse** (red) consistently outperforms other models in Plan ACC, Aug. F1, and R-L. 2. **Single-LLMmulti-task** (green) shows the worst performance in Hallu. and R-L, with a sharp drop at 62.7k. 3. **Multi-LLMone-stage** (dark red) exhibits instability, particularly in Act EM (40 at 62.7k vs. 55 at 31.3k). 4. **α-UMi w/o reuse** (light pink) underperforms its "w/ reuse" counterpart across all metrics. --- ### Interpretation - **Model Efficiency**: α-UMi w/ reuse demonstrates superior performance, suggesting reuse mechanisms enhance accuracy. Single-LLMmulti-task struggles with hallucination (Hallu.) and reasoning (R-L), indicating potential overfitting or task-specific limitations. - **Dataset Size Impact**: Performance generally improves with larger datasets (e.g., Plan ACC peaks at 47.0k), but plateaus or declines at 62.7k, hinting at diminishing returns or data quality issues. - **Outliers**: The green line in (c) Hallu. peaks at 6 (12.1k), suggesting initial overconfidence in smaller datasets. The dark red line in (b) Act EM drops sharply at 62.7k, possibly due to model instability at scale. This analysis highlights trade-offs between model complexity, reuse strategies, and dataset size in language model performance.