## Line Chart: NDCG@10 vs. Embedding Dimensions for Different Models ### Overview The image is a line chart comparing the performance of different language models (LLM2Vec-7B and LLM2Vec-1B, with and without SMEC) based on their NDCG@10 scores across varying embedding dimensions. The x-axis represents embedding dimensions, while the y-axis represents the NDCG@10 score. The chart also indicates lossless dimension compression ratios for two of the models. ### Components/Axes * **Title:** None explicitly present in the image. * **X-axis:** * **Label:** Embedding dimensions * **Scale:** 128, 256, 512, 1024, 1536, 3584 * **Y-axis:** * **Label:** NDCG@10 * **Scale:** 0.4, 0.5, 0.6, 0.7, 0.8, 0.9 * **Legend:** Located in the bottom-right corner of the chart. * **LLM2Vec-7B:** Solid blue line with triangle markers. * **LLM2Vec-7B (w/ SMEC):** Solid maroon line with triangle markers. * **LLM2Vec-1B:** Dotted light blue line with circle markers. * **LLM2Vec-1B (w/ SMEC):** Dotted dark blue line with circle markers. ### Detailed Analysis * **LLM2Vec-7B (Solid Blue Line, Triangle Markers):** * Trend: Slopes upward. * Data Points: * 128: 0.568 * 256: 0.648 * 512: 0.707 * 1024: 0.757 * 1536: 0.790 * 3584: 0.802 * **LLM2Vec-7B (w/ SMEC) (Solid Maroon Line, Triangle Markers):** * Trend: Slopes upward. * Data Points: * 128: 0.772 * 256: 0.803 * 512: 0.832 * 1024: 0.844 * 1536: 0.852 * 3584: 0.862 * **LLM2Vec-1B (Dotted Light Blue Line, Circle Markers):** * Trend: Slopes upward. * Data Points: * 128: 0.492 * 256: 0.576 * 512: 0.635 * 1024: 0.684 * 1536: 0.715 * 3584: Not explicitly labeled, but appears to be around 0.73. * **LLM2Vec-1B (w/ SMEC) (Dotted Dark Blue Line, Circle Markers):** * Trend: Slopes upward. * Data Points: * 128: 0.718 * 256: 0.743 * 512: 0.770 * 1024: 0.784 * 1536: 0.793 * 3584: 0.802 * **Annotations:** * "~ 14x lossless dimension compression": Points to the LLM2Vec-7B (w/ SMEC) line. * "~ 12 x lossless dimension compression": Points to the LLM2Vec-1B line. ### Key Observations * The LLM2Vec-7B models (both with and without SMEC) consistently outperform the LLM2Vec-1B models across all embedding dimensions. * For both the 7B and 1B models, using SMEC generally improves performance. * The performance gains from increasing embedding dimensions appear to diminish as the dimensions increase, especially for the LLM2Vec-7B (w/ SMEC) model. * The LLM2Vec-7B (w/ SMEC) model achieves the highest NDCG@10 scores. ### Interpretation The chart demonstrates the impact of model size (7B vs. 1B), the use of SMEC, and embedding dimensions on the performance of language models, as measured by NDCG@10. The LLM2Vec-7B model with SMEC shows the best performance, suggesting that both a larger model size and the SMEC technique contribute to improved ranking quality. The annotations regarding lossless dimension compression suggest that these models can achieve good performance even with reduced dimensionality, potentially saving computational resources. The diminishing returns observed with increasing embedding dimensions indicate that there may be an optimal point beyond which further increases in dimensionality provide minimal performance gains.

## Line Chart: NDCG@10 vs. Embedding Dimensions ### Overview This line chart illustrates the relationship between embedding dimensions and NDCG@10 scores for four different models: LLM2Vec-7B, LLM2Vec-7B with SMEC, LLM2Vec-1B, and LLM2Vec-1B with SMEC. The chart demonstrates how performance (NDCG@10) changes as the dimensionality of the embeddings increases. Two compression ratios are indicated: ~14x and ~12x. ### Components/Axes * **X-axis:** Embedding dimensions, ranging from 128 to 3584. Markers are placed at 128, 256, 512, 1024, 1536, and 3584. * **Y-axis:** NDCG@10, ranging from 0.4 to 0.9. * **Legend:** Located in the bottom-right corner. * LLM2Vec-7B (Solid Blue Line) * LLM2Vec-7B(w/ SMEC) (Solid Red Line) * LLM2Vec-1B (Dashed Blue Line) * LLM2Vec-1B(w/ SMEC) (Dashed Green Line) * **Annotations:** * “~14x lossless dimension compression” – positioned above the LLM2Vec-7B and LLM2Vec-7B(w/ SMEC) lines. * “~12x lossless dimension compression” – positioned below the LLM2Vec-1B and LLM2Vec-1B(w/ SMEC) lines. ### Detailed Analysis Here's a breakdown of the data series and their values: * **LLM2Vec-7B (Solid Blue Line):** This line slopes upward, indicating increasing NDCG@10 with increasing embedding dimensions. * 128: 0.492 * 256: 0.576 * 512: 0.648 * 1024: 0.743 * 1536: 0.77 * 3584: 0.803 * **LLM2Vec-7B(w/ SMEC) (Solid Red Line):** This line also slopes upward, consistently outperforming LLM2Vec-7B. * 128: 0.718 * 256: 0.803 * 512: 0.832 * 1024: 0.844 * 1536: 0.852 * 3584: 0.862 * **LLM2Vec-1B (Dashed Blue Line):** This line slopes upward, but starts at a lower NDCG@10 and generally performs worse than LLM2Vec-7B. * 128: 0.568 * 256: 0.648 * 512: 0.635 * 1024: 0.684 * 1536: 0.715 * 3584: 0.757 * **LLM2Vec-1B(w/ SMEC) (Dashed Green Line):** This line slopes upward, and consistently outperforms LLM2Vec-1B. * 128: 0.707 * 256: 0.73 * 512: 0.77 * 1024: 0.784 * 1536: 0.793 * 3584: 0.802 ### Key Observations * The models with SMEC consistently outperform their counterparts without SMEC across all embedding dimensions. * LLM2Vec-7B generally outperforms LLM2Vec-1B, even without SMEC. * The rate of improvement in NDCG@10 diminishes as embedding dimensions increase, particularly for the models with SMEC. * The annotations indicate that the models achieve significant lossless dimension compression (approximately 14x for LLM2Vec-7B and 12x for LLM2Vec-1B). ### Interpretation The data suggests that increasing embedding dimensions generally improves retrieval performance (as measured by NDCG@10). However, the gains from increasing dimensionality become smaller at higher dimensions. The consistent performance improvement from using SMEC indicates that this technique effectively enhances the quality of the embeddings, leading to better retrieval results. The annotations highlight the efficiency of the models, demonstrating that they can achieve good performance with significantly reduced dimensionality, which is beneficial for storage and computational costs. The difference in compression ratios between the 7B and 1B models could be due to differences in model architecture or training data. The flattening of the curves at higher dimensions suggests a point of diminishing returns, where further increasing dimensionality does not yield substantial improvements in performance.

## Line Chart: Performance of LLM2Vec Models with and without SMEC Across Embedding Dimensions ### Overview This image is a line chart comparing the performance (measured by NDCG@10) of four different model configurations as the embedding dimension increases. The chart demonstrates the impact of a technique called "SMEC" on two base models (LLM2Vec-7B and LLM2Vec-1B) and highlights significant "lossless dimension compression" capabilities. ### Components/Axes * **X-Axis (Horizontal):** Labeled "Embedding dimensions". It has discrete, non-linearly spaced tick marks at the values: 128, 256, 512, 1024, 1536, and 3584. * **Y-Axis (Vertical):** Labeled "NDCG@10". It is a linear scale ranging from 0.4 to 0.9, with major tick marks at 0.1 intervals (0.4, 0.5, 0.6, 0.7, 0.8, 0.9). * **Legend:** Located in the bottom-right corner. It defines four data series: 1. `LLM2Vec-7B`: Represented by a solid light blue line with upward-pointing triangle markers. 2. `LLM2Vec-7B(w/ SMEC)`: Represented by a solid dark red/brown line with upward-pointing triangle markers. 3. `LLM2Vec-1B`: Represented by a dotted light blue line with circle markers. 4. `LLM2Vec-1B(w/ SMEC)`: Represented by a dotted dark blue line with circle markers. * **Annotations:** * Top-left area: Text "~ 14× lossless dimension compression" in dark red/brown, with a dashed green arrow pointing from the `LLM2Vec-7B(w/ SMEC)` line at dimension 128 to the `LLM2Vec-7B` line at dimension 1536. * Center-right area: Text "~ 12 × lossless dimension compression" in dark red/brown, with a dashed green arrow pointing from the `LLM2Vec-1B(w/ SMEC)` line at dimension 128 to the `LLM2Vec-1B` line at dimension 1536. ### Detailed Analysis **Data Series and Trends:** 1. **LLM2Vec-7B (Light Blue, Solid Line, Triangles):** * **Trend:** Slopes steeply upward, showing significant performance gains as dimensions increase, with the rate of improvement slowing at higher dimensions. * **Data Points:** * Dimension 128: NDCG@10 ≈ 0.568 * Dimension 256: NDCG@10 ≈ 0.648 * Dimension 512: NDCG@10 ≈ 0.707 * Dimension 1024: NDCG@10 ≈ 0.757 * Dimension 1536: NDCG@10 ≈ 0.79 * Dimension 3584: NDCG@10 ≈ 0.802 2. **LLM2Vec-7B(w/ SMEC) (Dark Red/Brown, Solid Line, Triangles):** * **Trend:** Slopes gently upward. It starts at a much higher performance level than the base model and maintains a consistent lead across all dimensions. * **Data Points:** * Dimension 128: NDCG@10 ≈ 0.772 * Dimension 256: NDCG@10 ≈ 0.803 * Dimension 512: NDCG@10 ≈ 0.832 * Dimension 1024: NDCG@10 ≈ 0.844 * Dimension 1536: NDCG@10 ≈ 0.852 * Dimension 3584: NDCG@10 ≈ 0.862 3. **LLM2Vec-1B (Light Blue, Dotted Line, Circles):** * **Trend:** Slopes upward, but its performance is consistently lower than the 7B models. The curve is less steep than the base 7B model. * **Data Points:** * Dimension 128: NDCG@10 ≈ 0.492 * Dimension 256: NDCG@10 ≈ 0.576 * Dimension 512: NDCG@10 ≈ 0.635 * Dimension 1024: NDCG@10 ≈ 0.684 * Dimension 1536: NDCG@10 ≈ 0.715 4. **LLM2Vec-1B(w/ SMEC) (Dark Blue, Dotted Line, Circles):** * **Trend:** Slopes gently upward. Similar to the 7B SMEC variant, it starts at a much higher performance than its base model and maintains a lead. * **Data Points:** * Dimension 128: NDCG@10 ≈ 0.718 * Dimension 256: NDCG@10 ≈ 0.743 * Dimension 512: NDCG@10 ≈ 0.77 * Dimension 1024: NDCG@10 ≈ 0.784 * Dimension 1536: NDCG@10 ≈ 0.793 ### Key Observations 1. **SMEC Provides a Major Boost:** For both the 1B and 7B models, applying SMEC results in a substantial performance increase at every embedding dimension. The SMEC variants (dark lines) are always above their corresponding base models (light lines). 2. **Performance at Low Dimensions:** The most dramatic relative improvement is at the lowest dimension (128). For example, LLM2Vec-7B(w/ SMEC) at 128 dimensions (0.772) outperforms the base LLM2Vec-7B at 1536 dimensions (0.79) by a small margin, achieving comparable performance with ~12x fewer dimensions. 3. **Diminishing Returns:** All curves show diminishing returns; the performance gain from doubling the dimension decreases as the dimension grows larger. 4. **Model Size Comparison:** The 7B models consistently outperform their 1B counterparts, both with and without SMEC, indicating that larger base model capacity leads to better retrieval performance. 5. **Compression Claims:** The annotations explicitly state that SMEC enables "~14×" and "~12×" lossless dimension compression for the 7B and 1B models, respectively. This is visually supported by the dashed green arrows connecting a high-performing low-dimension SMEC point to a similarly performing high-dimension base model point. ### Interpretation This chart presents a compelling case for the effectiveness of the SMEC technique in the context of dense retrieval models (LLM2Vec). The core message is one of **efficiency without sacrifice**. * **What the data suggests:** SMEC allows models to achieve high retrieval performance (high NDCG@10) using drastically fewer embedding dimensions. This is "lossless" in the sense that a compressed model (e.g., 7B with SMEC at 128 dims) can match or exceed the performance of a much larger uncompressed model (e.g., 7B without SMEC at 1536 dims). * **How elements relate:** The x-axis (dimensions) is a proxy for memory and computational cost. The y-axis (NDCG@10) is a proxy for quality. The SMEC lines demonstrate a superior Pareto frontier—better quality at every cost point. The gap between the SMEC and non-SMEC lines for a given model size quantifies the "free" performance gain from the technique. * **Notable Implications:** The practical implication is significant. Deploying a retrieval system with SMEC could reduce storage requirements for vector databases by an order of magnitude (12-14x) and lower the computational cost of similarity search, all while maintaining or improving result quality. This makes high-performance retrieval more feasible for resource-constrained environments. The consistent results across two different model scales (1B and 7B) suggest the technique is robust.

## Line Chart: NDCG@10 Performance vs. Embedding Dimensions ### Overview The chart compares the performance of four language model variants (LLM2Vec-7B, LLM2Vec-7B with SMEC, LLM2Vec-1B, and LLM2Vec-1B with SMEC) across different embedding dimensions (128 to 3584). Performance is measured as NDCG@10, a metric for ranking quality, plotted against embedding dimensions. The chart includes annotations for lossless dimension compression ratios (~12x and ~14x) and visualizes trends in model efficiency and effectiveness. ### Components/Axes - **X-axis**: "Embedding dimensions" (logarithmic scale: 128, 256, 512, 1024, 1536, 3584). - **Y-axis**: "NDCG@10" (normalized Discounted Cumulative Gain at 10 results, range: 0.4–0.9). - **Legend**: Located in the bottom-right corner, mapping colors/markers to model variants: - **Blue line with triangles**: LLM2Vec-7B - **Maroon line with triangles**: LLM2Vec-7B(w/ SMEC) - **Dashed cyan line with circles**: LLM2Vec-1B - **Dotted blue line with squares**: LLM2Vec-1B(w/ SMEC) - **Annotations**: - Green dashed lines labeled "~12x lossless dimension compression" and "~14x lossless dimension compression." - Arrows pointing to specific data points (e.g., 0.832 at 512 dimensions for LLM2Vec-7B(w/ SMEC)). ### Detailed Analysis 1. **LLM2Vec-7B(w/ SMEC)** (Maroon line): - Starts at **0.772** (128 dimensions) and increases steadily to **0.862** (3584 dimensions). - Shows the highest NDCG@10 across all dimensions, with a ~14x lossless compression annotation at 512 dimensions. 2. **LLM2Vec-7B** (Blue line): - Begins at **0.718** (128 dimensions) and rises to **0.802** (3584 dimensions). - Outperforms LLM2Vec-1B variants but lags behind its SMEC-enhanced counterpart. 3. **LLM2Vec-1B(w/ SMEC)** (Dotted blue line): - Starts at **0.492** (128 dimensions) and improves to **0.802** (3584 dimensions). - Demonstrates the largest relative gain (~12x compression) at 512 dimensions. 4. **LLM2Vec-1B** (Dashed cyan line): - Begins at **0.568** (128 dimensions) and reaches **0.715** (3584 dimensions). - Shows minimal improvement compared to its SMEC-enhanced version. ### Key Observations - **SMEC Enhancement**: All SMEC variants (7B and 1B) outperform their base models, with the 1B SMEC variant showing the most dramatic improvement (~12x compression). - **Model Size Impact**: Larger models (7B) consistently achieve higher NDCG@10 than smaller models (1B), even without SMEC. - **Diminishing Returns**: Performance gains plateau as embedding dimensions increase, particularly beyond 1024 dimensions. ### Interpretation The data highlights the interplay between model size, embedding dimensions, and efficiency techniques like SMEC. While larger models (7B) inherently perform better, SMEC significantly boosts the efficiency of smaller models (1B), enabling them to approach the performance of larger models with reduced computational overhead. The ~12x and ~14x compression annotations suggest that SMEC allows for substantial dimensionality reduction without sacrificing ranking quality. This implies that SMEC is a critical optimization for deploying resource-constrained models in real-world applications.