## Flowchart: AMIGO — Benchmark Framework Overview
### Overview
The diagram illustrates a four-stage framework for evaluating a Visual Language Model (VLM) through iterative questioning, validation, and verification. It emphasizes protocol adherence, candidate set management, and final guess accuracy.
### Components/Axes
1. **Benchmark Model (Evaluated VLM)**
- **Conversation History**:
- T1: "Does it have a separate belt?" → Yes
- T2: "Does it have front buttons?" → No
- T3: "Does it have a wrap-style bodice?" → Yes
- T4: "Does it have tiered skirt?" → Yes
- **New Question**: "Does your favorite dress have a V-neckline?"
- **Final Guess**: "Dress #3"
2. **Question-Violation Detector (LLM-Based)**
- **Check Criteria**:
- Follows protocol?
- Valid & non-redundant?
- **Decision**:
- ✅ "Question OK" (green)
- ❌ "Violation → Reject" (red)
3. **User / Oracle Simulator (VLM Agent)**
- **Actions**:
- View target image
- Zoom-in tool enabled
- Answer questions with:
- ✅ Yes (green)
- ❌ No (red)
- ❓ Unsure (yellow)
- **Visual**: Person interacting with a laptop and dresses.
4. **Verification Module**
- **Tasks**:
- Maintain & audit feasible candidate set
- Check image-answer pairs
- Update candidate set
- Verify final guess (#3)
- **Outcome**:
- ✅ "PASS" (green checkmark with laurel wreath)
### Flowchart Connections
- **Orange Arrows**:
- Benchmark Model → Detector → Oracle Simulator → Benchmark (loop)
- Detector → Skip (if violation)
- **Red Arrow**: Final guess incorrect → loop back
- **Green Arrow**: Final guess correct → "PASS"
### Key Observations
- **Protocol Enforcement**: The detector rejects non-compliant questions, ensuring data quality.
- **Iterative Testing**: The loop between Benchmark Model and Detector/Oracle Simulator suggests continuous refinement.
- **Candidate Set Management**: The verification module dynamically updates candidates based on image-answer pairs.
- **Final Guess Validation**: The red box around "Dress #3" in the verification module highlights its critical role.
### Interpretation
This framework evaluates a VLM’s ability to answer visual questions while maintaining protocol compliance. The detector acts as a gatekeeper, filtering invalid queries to prevent redundant or biased data. The Oracle Simulator mimics human interaction, testing the model’s reasoning under constraints (e.g., zoom tools). The verification module ensures answers align with visual evidence, with the final guess (#3) serving as the model’s output. The "PASS" outcome indicates successful alignment between the model’s reasoning and ground truth. The loop structure implies a focus on robustness, as incorrect guesses trigger re-evaluation.
**Note**: No numerical data or trends are present; the diagram emphasizes process flow and decision logic.
