## Chart: Hits@1 vs. latency on CWQ ### Overview This is a scatter plot showing the relationship between Hits@1 on CWQ (percentage) and per-query latency (seconds, median) for different language model families. The plot distinguishes between Embedding models, Pure LLM models, and LLMs+KG models using different shapes and colors. ### Components/Axes * **Title:** Hits@1 vs. latency on CWQ * **X-axis:** Hits@1 on CWQ (%) * Scale ranges from approximately 15% to 75% with tick marks at 20, 30, 40, 50, 60, and 70. * **Y-axis:** Per-query latency 10'x (seconds, median) * Scale ranges from -0.25 to 1.50 with tick marks at -0.25, 0.00, 0.25, 0.50, 0.75, 1.00, 1.25, and 1.50. * **Legend (top-left):** * Embedding: Blue circle * Pure LLM: Blue square * LLMs+KG: Blue triangle ### Detailed Analysis The data points are scattered across the plot, with some clustering in certain regions. * **Embedding Models:** * KV-Mem: Located at approximately (18%, -0.15 seconds). * NSM: Located at approximately (48%, -0.15 seconds). * Both are represented by blue circles. * **Pure LLM Models:** * ChatGPT (1 call): Located at approximately (45%, 0.3 seconds). * StructGPT: Located at approximately (58%, 0.5 seconds). * GPT-4 (1 call): Located at approximately (55%, 0.6 seconds). * UniKGOA: Located at approximately (52%, 0.5 seconds). * All are represented by blue squares. * **LLMs+KG Models:** * PathHD: Located at approximately (70%, 0.3 seconds). * EjDeLiS: Located at approximately (72%, 0.8 seconds). * GoG: Located at approximately (70%, 0.9 seconds). * KG-Agent: Located at approximately (70%, 1.0 seconds). * Think-on-Graph: Located at approximately (65%, 1.0 seconds). * RoG: Located at approximately (60%, 1.3 seconds). * All are represented by blue triangles. ### Key Observations * There appears to be a general trend where higher Hits@1 on CWQ is associated with higher per-query latency. * Embedding models (KV-Mem, NSM) have the lowest latency but also the lowest Hits@1. * LLMs+KG models generally have higher Hits@1 but also higher latency. * Pure LLM models are clustered in the middle range for both metrics. ### Interpretation The plot suggests a trade-off between accuracy (Hits@1) and speed (latency) for different language model families. Embedding models are fast but less accurate, while LLMs+KG models are more accurate but slower. Pure LLM models offer a balance between the two. The specific models mentioned (ChatGPT, GPT-4, etc.) provide benchmarks for performance within these categories. The data indicates that incorporating knowledge graphs (KG) into language models tends to improve accuracy at the cost of increased latency. The outlier RoG has the highest latency and a high Hits@1 score.

\n ## Scatter Plot: Hits@1 vs. Latency on CWQ ### Overview This scatter plot visualizes the relationship between Hits@1 (percentage) on the CWQ dataset and per-query latency (in seconds, median) for various models. The models are categorized into three families: Embedding, Pure LLM, and LLMs+KG. Each point represents a model, with its position determined by its performance on the two metrics. The shape of the marker indicates the model family. ### Components/Axes * **X-axis:** Hits@1 on CWQ (%) - Ranges from approximately 20% to 75%. * **Y-axis:** Per-query latency 10^x (seconds, median) - Ranges from approximately -0.3 to 1.5. The axis is on a logarithmic scale. * **Legend:** Located in the top-left corner. * **Embedding (Blue Circles):** Represents models utilizing embedding techniques. * **Pure LLM (Blue Squares):** Represents models that are purely Large Language Models. * **LLMs+KG (Red Triangles):** Represents models combining Large Language Models with Knowledge Graphs. * **Data Points:** Each point represents a specific model. The points are colored according to their family (as defined in the legend). ### Detailed Analysis Here's a breakdown of the data points, categorized by family and with approximate values: **Embedding (Blue Circles):** * **KV-Mem:** Approximately (22%, -0.25). * **NSM:** Approximately (42%, -0.3). **Pure LLM (Blue Squares):** * **ChatGPT (1 call):** Approximately (47%, 0.35). * **GPT-4 (1 call):** Approximately (50%, 0.5). * **UniKQA:** Approximately (48%, 0.45). * **StructGPT:** Approximately (51%, 0.55). **LLMs+KG (Red Triangles):** * **RoQ:** Approximately (62%, 1.3). * **KG-Agent:** Approximately (68%, 0.9). * **fiDELIS:** Approximately (69%, 0.8). * **Think-on-Graph:** Approximately (65%, 1.0). * **PathHD:** Approximately (72%, 0.3). **Trends:** * **Embedding Models:** Generally exhibit low latency and moderate Hits@1 scores. * **Pure LLM Models:** Show a moderate trade-off between latency and Hits@1. * **LLMs+KG Models:** Tend to have higher latency but also achieve higher Hits@1 scores. There is a positive correlation between latency and Hits@1 within this family. ### Key Observations * There's a clear separation between the three model families. * LLMs+KG models consistently outperform the other two families in terms of Hits@1, but at the cost of increased latency. * KV-Mem and NSM have the lowest latency, but also the lowest Hits@1 scores. * PathHD is an outlier within the LLMs+KG family, exhibiting relatively low latency compared to its Hits@1 score. * RoQ has the highest latency. ### Interpretation The data suggests a trade-off between accuracy (Hits@1) and speed (latency) in question answering systems. Embedding-based models prioritize speed, while LLMs+KG models prioritize accuracy. Pure LLM models offer a balance between the two. The use of Knowledge Graphs appears to significantly improve accuracy, but introduces computational overhead, resulting in higher latency. The positioning of PathHD suggests it may be a more efficient LLM+KG model, achieving a relatively high Hits@1 score with lower latency than its counterparts. This could be due to optimizations in its architecture or implementation. The logarithmic scale on the Y-axis emphasizes the impact of latency, particularly for models with higher latency values. The large difference in latency between models like KV-Mem and RoQ is visually amplified by this scaling. The scatter plot provides valuable insights for selecting the appropriate model for a given application, depending on the relative importance of accuracy and speed.

\n ## Scatter Plot: Hits@1 vs. latency on CWQ ### Overview The image is a scatter plot comparing various AI models and systems on two performance metrics: accuracy (Hits@1 on the CWQ benchmark) and speed (per-query latency). The plot categorizes models into three families, distinguished by color and marker shape, showing a general trade-off between higher accuracy and increased latency. ### Components/Axes * **Title:** "Hits@1 vs. latency on CWQ" * **X-Axis:** "Hits@1 on CWQ (%)". Scale ranges from 0 to 80, with major ticks at 0, 20, 40, 60, 80. * **Y-Axis:** "Per-query latency (seconds, median)". Scale ranges from -0.25 to 1.50, with major ticks at -0.25, 0.00, 0.25, 0.50, 0.75, 1.00, 1.25, 1.50. * **Legend (Top-Left Corner):** * **Family:** (Header) * **Fine-tuning:** Green square marker. * **Pure LLM:** Blue circle marker. * **LLMs+KG:** Orange triangle marker. ### Detailed Analysis The plot contains 10 distinct data points, each labeled with a model/system name. Their approximate coordinates (x=Hits@1 %, y=Latency seconds) are as follows: **Fine-tuning Family (Green Squares):** 1. **XVSMon:** Positioned at the far left and bottom. Approximate coordinates: (10%, -0.15s). This indicates very low accuracy and the lowest latency on the chart. 2. **GPT4-T:** Positioned near the center-bottom. Approximate coordinates: (40%, -0.10s). Shows moderate accuracy with very low latency. **Pure LLM Family (Blue Circles):** 3. **ChatQA-2:** Positioned left of center. Approximate coordinates: (35%, 0.25s). 4. **UnkQA:** Positioned near the center. Approximate coordinates: (45%, 0.35s). 5. **GPT-4 (1-shot):** Positioned right of center. Approximate coordinates: (50%, 0.40s). This cluster shows a trend of increasing latency with modest gains in accuracy. **LLMs+KG Family (Orange Triangles):** 6. **PaLM2:** Positioned on the right side, lower than its group. Approximate coordinates: (70%, 0.20s). An outlier within its family, showing high accuracy with relatively low latency. 7. **Think-and-Execute:** Positioned in the upper-right quadrant. Approximate coordinates: (65%, 0.85s). 8. **KG-GPT:** Positioned in the upper-right quadrant. Approximate coordinates: (75%, 0.90s). 9. **KG-LLaMA:** Positioned in the upper-right quadrant. Approximate coordinates: (72%, 1.00s). 10. **RoG:** Positioned at the top of the chart. Approximate coordinates: (60%, 1.40s). This model has the highest latency by a significant margin. ### Key Observations 1. **Clear Family Clustering:** The three model families form distinct clusters. "Fine-tuning" models are in the low-latency, low-to-moderate accuracy region (bottom-left). "Pure LLM" models form a central cluster with moderate latency and accuracy. "LLMs+KG" models dominate the high-accuracy region (right side) but with a wide spread in latency. 2. **Accuracy-Latency Trade-off:** There is a general positive correlation between Hits@1 accuracy and latency. Moving from left to right (increasing accuracy), the data points generally move upward (increasing latency). 3. **Notable Outliers:** * **PaLM2 (LLMs+KG):** Achieves high accuracy (~70%) with latency (~0.20s) comparable to the "Pure LLM" cluster, making it highly efficient. * **RoG (LLMs+KG):** Has the highest latency (~1.40s) but only moderate accuracy (~60%), suggesting a potential inefficiency. * **XVSMon (Fine-tuning):** Has the lowest accuracy and latency, potentially representing a very fast but less capable baseline. ### Interpretation This chart visualizes the performance landscape of different approaches to complex question answering (on the CWQ benchmark). The data suggests a fundamental trade-off: achieving higher accuracy (Hits@1) typically requires more computational time (latency). * **Fine-tuning** approaches (green) prioritize speed, offering the lowest latencies but at a significant cost to accuracy. They are suitable for scenarios where response time is critical and some error is acceptable. * **Pure LLM** approaches (blue) represent a middle ground, offering a balance between reasonable accuracy and moderate speed. * **LLMs augmented with Knowledge Graphs (LLMs+KG)** (orange) generally achieve the highest accuracy, demonstrating the value of structured knowledge for complex reasoning. However, this comes at the cost of higher and more variable latency, likely due to the overhead of KG retrieval and integration. The wide latency spread within this group (from PaLM2's ~0.20s to RoG's ~1.40s) indicates significant differences in the efficiency of their KG integration mechanisms. The standout model is **PaLM2**, which breaks the general trend by delivering high accuracy with low latency, suggesting a particularly efficient architecture or integration method. Conversely, **RoG** appears to be the least efficient, incurring a very high latency penalty for its level of accuracy. This analysis would be crucial for a practitioner selecting a model based on their specific constraints for accuracy versus response time.

## Scatter Plot: Hits@1 vs. Latency on CWQ ### Overview This scatter plot compares the performance of various models on the CWQ dataset, plotting **Hits@1 (accuracy)** against **latency (10^x seconds)**. Models are categorized into three families: **Embedding**, **Pure LLM**, and **LLMs+KG**, with distinct symbols and colors. The y-axis uses a logarithmic scale (10^x) to represent latency. --- ### Components/Axes - **X-axis**: Hits@1 on CWQ (%) - Range: 0–80% - Labels: Discrete ticks at 20, 30, 40, 50, 60, 70. - **Y-axis**: Per-query latency (10^x seconds, median) - Range: -0.25 to 1.50 (log scale) - Labels: Discrete ticks at -0.25, 0.0, 0.25, 0.5, 0.75, 1.0, 1.25, 1.50. - **Legend**: - **Embedding**: Green circles (●) - **Pure LLM**: Blue squares (■) - **LLMs+KG**: Orange triangles (▲) --- ### Detailed Analysis #### Data Points and Trends 1. **Embedding (Green Circles)**: - **KV-Mem**: (20%, -0.25) - **NSM**: (50%, 0.0) - **Trend**: Low Hits@1 and low latency. 2. **Pure LLM (Blue Squares)**: - **ChatGPT (1 call)**: (50%, 0.5) - **UniKGQA**: (45%, 0.5) - **StructGPT**: (55%, 0.5) - **GPT-4 (1 call)**: (60%, 0.75) - **Trend**: Moderate Hits@1 (45–60%) with consistent latency (~0.5–0.75). 3. **LLMs+KG (Orange Triangles)**: - **Think-On-Graph**: (65%, 1.0) - **KG-Agent**: (70%, 1.25) - **GoG**: (75%, 1.5) - **FIDELIS**: (70%, 1.25) - **RoG**: (75%, 1.5) - **Trend**: High Hits@1 (65–75%) but significantly higher latency (1.0–1.5). #### Spatial Grounding - **Legend**: Top-left corner, clearly labeled with symbols and colors. - **Data Points**: - **Bottom-left**: Embedding models (KV-Mem, NSM). - **Center**: Pure LLM models (ChatGPT, UniKGQA, StructGPT). - **Top-right**: LLMs+KG models (Think-On-Graph, KG-Agent, GoG, FIDELIS, RoG). --- ### Key Observations 1. **Trade-off Between Accuracy and Latency**: - LLMs+KG models dominate the top-right quadrant, achieving the highest Hits@1 but with the highest latency. - Embedding models (KV-Mem, NSM) cluster in the bottom-left, showing poor performance in both metrics. 2. **Outliers**: - **NSM**: (50%, 0.0) – Unusually low latency for its Hits@1 (50%), suggesting optimization or unique architecture. - **KG-Agent** and **RoG**: Highest Hits@1 (70–75%) but extreme latency (1.25–1.5), indicating computational intensity. 3. **Logarithmic Scale Impact**: - Latency values (10^x) amplify differences in the upper range (e.g., 1.5 vs. 1.0 = 10x increase). --- ### Interpretation - **Performance Hierarchy**: - **LLMs+KG** prioritize accuracy over speed, suitable for applications where precision is critical (e.g., medical diagnosis). - **Pure LLMs** balance moderate accuracy and latency, ideal for general-purpose use. - **Embedding models** are fast but less accurate, potentially useful for real-time systems with relaxed accuracy requirements. - **Anomalies**: - **NSM**’s low latency at 50% Hits@1 suggests it may use lightweight mechanisms (e.g., caching) or simplified queries. - **KG-Agent** and **RoG**’s high latency could stem from complex knowledge graph traversal or large-scale inference. - **Practical Implications**: - Developers must weigh accuracy needs against computational costs. For example, GPT-4 (60% Hits@1, 0.75 latency) offers a middle ground, while RoG (75% Hits@1, 1.5 latency) is optimal for high-stakes scenarios. - **Unanswered Questions**: - Why do some LLMs+KG models (e.g., Think-On-Graph) achieve higher Hits@1 than others despite similar latency? - How do query complexity or dataset characteristics influence these trade-offs? --- ### Conclusion The chart highlights a clear Pareto frontier: no model dominates all others in both metrics. The choice of model depends on the application’s tolerance for latency versus the need for accuracy. Future work could explore hybrid approaches to mitigate the latency-accuracy trade-off.