## Chart: Risk-Coverage Curve (Dialect Task) ### Overview The image is a risk-coverage curve comparing two methods: BiLSTM + Conformal Prediction (CP) and Modal MLNN Reasoner (Fixed Point) on a dialect task. The y-axis represents Selective Risk (1 - Accuracy on non-abstained), and the x-axis represents coverage. The BiLSTM curve shows a rapid decrease in risk as coverage increases, eventually plateauing near zero risk. The Modal MLNN Reasoner is represented by a single point. ### Components/Axes * **Title:** Risk-Coverage Curve (Dialect Task) * **X-axis:** Coverage, ranging from 0.0 to 1.0 in increments of 0.2. * **Y-axis:** Selective Risk (1 - Accuracy on non-abstained), ranging from 0.00 to 1.00 in increments of 0.25. * **Legend:** Located in the bottom-right corner. * Blue line with markers: BiLSTM + Conformal Prediction (CP) * Red circle: Modal MLNN Reasoner (Fixed Point) ### Detailed Analysis * **BiLSTM + Conformal Prediction (CP):** * Color: Blue * Trend: The curve starts at approximately (0, 0.9) and rapidly decreases to near 0.0 as coverage increases to approximately 0.2. From 0.2 to 1.0, the curve remains relatively flat, hovering around 0.0. * Data Points (approximate): * (0.0, 0.9) * (0.1, 0.1) * (0.2, 0.02) * (0.4, 0.01) * (0.6, 0.005) * (0.8, 0.005) * (1.0, 0.005) * **Modal MLNN Reasoner (Fixed Point):** * Color: Red * Location: Appears to be at approximately (0.0, 0.0). * Data Point (approximate): * (0.0, 0.0) ### Key Observations * The BiLSTM + Conformal Prediction method achieves very low selective risk with relatively low coverage. * The Modal MLNN Reasoner has a fixed point at (0.0, 0.0), indicating perfect accuracy on the non-abstained data, but at zero coverage. * The BiLSTM curve shows a steep initial drop, indicating that even a small amount of coverage significantly reduces the selective risk. ### Interpretation The risk-coverage curve demonstrates the trade-off between coverage and selective risk for the two methods. The BiLSTM + Conformal Prediction method is highly effective, achieving near-zero risk with moderate coverage. The Modal MLNN Reasoner, while having perfect accuracy at its fixed point, offers no coverage. This suggests that the BiLSTM + CP method is superior for this dialect task, as it provides a good balance between accuracy and the ability to make predictions on a wider range of inputs. The steep initial drop in the BiLSTM curve highlights the value of even a small amount of coverage in reducing risk.

## Line Chart: Risk-Coverage Curve (Dialect Task) ### Overview This image displays a "Risk-Coverage Curve" for a "Dialect Task." It compares the performance of two distinct machine learning approaches: a "BiLSTM + Conformal Prediction (CP)" model, represented by a continuous blue line with diamond markers, and a "Modal MLNN Reasoner (Fixed Point)," represented by a single red circular marker. The chart illustrates the trade-off between the coverage of the model (the proportion of data points the model is allowed to predict on) and the selective risk (1 minus the accuracy on those non-abstained predictions). ### Components/Axes * **Title:** "Risk-Coverage Curve (Dialect Task)" centered at the top. * **Y-Axis:** Labeled "Selective Risk (1 - Accuracy on non-abstained)". The scale ranges from 0.00 to 1.00 in increments of 0.25. * **X-Axis:** Unlabeled, but implied to represent "Coverage" (the proportion of the dataset predicted). The scale ranges from 0.0 to 1.0 in increments of 0.2. * **Legend:** Located in the bottom-right quadrant of the chart area. * **Blue Line with Diamond Markers:** Labeled "BiLSTM + Conformal Prediction (CP)". * **Red Circle Marker:** Labeled "Modal MLNN Reasoner (Fixed Point)". * **Grid:** Light gray dashed grid lines are present throughout the plot area. ### Detailed Analysis * **BiLSTM + Conformal Prediction (CP) Trend:** * The curve originates at the origin (0,0). * It exhibits a very sharp, near-vertical upward slope between X=0.0 and X=0.05. By X=0.05, the Selective Risk has already climbed to approximately 0.85–0.90. * After X=0.1, the curve flattens significantly, showing a very gradual, slight upward trend as it approaches X=1.0. * At X=1.0, the Selective Risk is approximately 0.95–0.98. * **Modal MLNN Reasoner (Fixed Point) Data Point:** * This is a single data point located in the bottom-left region of the chart. * **Coordinates:** The X-value is approximately 0.03–0.04. The Y-value is approximately 0.02–0.03. ### Key Observations * **Performance Disparity:** The "Modal MLNN Reasoner" significantly outperforms the "BiLSTM + CP" model in this specific metric. At a very low coverage (approx. 0.03), the MLNN Reasoner achieves a very low risk (approx. 0.02), whereas the BiLSTM model at a similar coverage level (approx. 0.03) has a much higher risk (approx. 0.75). * **Model Behavior:** The BiLSTM + CP curve suggests that as the model is forced to cover more of the dataset (moving right on the X-axis), its risk increases rapidly and remains high, indicating poor performance when forced to make predictions on the entire dataset. * **Fixed Point Nature:** The "Modal MLNN Reasoner" is plotted as a single point, confirming the label "Fixed Point," implying it does not have a tunable coverage parameter to generate a full curve like the Conformal Prediction method. ### Interpretation This chart demonstrates a classic selective classification trade-off. "Selective Risk" is defined as (1 - Accuracy), meaning lower values are better. The data suggests that the **Modal MLNN Reasoner** is a highly specialized or highly confident model that operates effectively only on a small subset of the data (low coverage), where it achieves very high accuracy (low risk). Conversely, the **BiLSTM + Conformal Prediction** model is designed to be flexible, allowing for varying levels of coverage. However, the steepness of its curve indicates that even at low coverage levels, the BiLSTM model struggles to maintain high accuracy compared to the MLNN Reasoner. As coverage increases toward 1.0 (predicting on all data), the BiLSTM's risk remains very high, suggesting the model is not well-calibrated or inherently inaccurate for this specific dialect task when forced to predict on the full distribution.

## Line Chart: Risk-Coverage Curve (Dialect Task) ### Overview The chart compares two approaches for risk-coverage performance in a dialect task: "BiLSTM + Conformal Prediction (CP)" and "Modal MLNN Reasoner (Fixed Point)". The y-axis represents "Selective Risk (1 - Accuracy on non-abstained)" (0.00–1.00), while the x-axis represents "Risk-Coverage" (0.0–1.0). The blue line (BiLSTM + CP) shows a steep initial increase followed by a plateau, while the red data point (Modal MLNN) remains at the origin. ### Components/Axes - **Y-Axis**: "Selective Risk (1 - Accuracy on non-abstained)" (0.00–1.00, increments of 0.25) - **X-Axis**: "Risk-Coverage" (0.0–1.0, increments of 0.2) - **Legend**: - Blue line: "BiLSTM + Conformal Prediction (CP)" - Red circle: "Modal MLNN Reasoner (Fixed Point)" - **Placement**: Legend is positioned in the bottom-right corner. ### Detailed Analysis - **BiLSTM + CP (Blue Line)**: - Starts at ~0.75 on the y-axis at x=0.0. - Rises sharply to 1.00 by x=0.2. - Remains flat at 1.00 for x=0.2–1.0. - **Modal MLNN Reasoner (Red Circle)**: - Single data point at (x=0.0, y=0.00). ### Key Observations 1. The BiLSTM + CP approach achieves near-perfect risk coverage (y=1.00) after x=0.2. 2. The Modal MLNN Reasoner shows no risk coverage (y=0.00) at x=0.0. 3. The blue line’s steep ascent suggests rapid improvement in performance with increasing risk-coverage. ### Interpretation The chart demonstrates that the BiLSTM + Conformal Prediction method outperforms the Modal MLNN Reasoner in balancing risk and coverage for dialect tasks. The abrupt plateau at y=1.00 for BiLSTM + CP implies it achieves full accuracy on non-abstained data once risk-coverage exceeds 20% (x=0.2). In contrast, the Modal MLNN Reasoner fails to cover any risk at the measured point, highlighting a critical limitation. This suggests BiLSTM + CP is more robust for dialect tasks requiring high accuracy under constrained risk.