## Text-Based System Prompt for LLM-as-a-Judge Evaluation (Gemini-2.5-Pro)
### Overview
This image contains a detailed system prompt written in **Korean** for evaluating large language models (LLMs) using Gemini-2.5-Pro as a judge. The prompt defines roles, tasks, input requirements, evaluation criteria, and output formatting rules. No numerical data, charts, or diagrams are present.
---
### Components/Axes
#### Header
- **Title**: "System Prompt for LLM-as-a-Judge Evaluation (Gemini-2.5-Pro)"
- **Language**: Korean (한국어)
#### Sections
1. **역할 (Role)**
- Describes the role of the LLM as a judge evaluating AI-generated responses.
- Key points:
- Focus on factual accuracy, logical soundness, depth of reasoning, and clarity.
- Requires strict adherence to predefined criteria (1-5 scale).
2. **임무 (Task)**
- Task: Evaluate responses to a specific prompt about hallucination.
- Input includes:
- Original prompt (`<문제 정보>`).
- AI-generated response (`<평가 대상 답변>`).
- Ground truth (`<진문가 검증 참조 자료>`).
3. **입력 정보 (Input Information)**
- Three components:
1. `<문제 정보>`: Original problem statement.
2. `<진문가 검증 참조 자료>`: 100% accurate ground truth.
3. `<평가 대상 답변>`: AI-generated response to evaluate.
4. **사고 과정 (Step-by-Step Thinking Process)**
- Four evaluation steps:
1. Verify factual accuracy against ground truth.
2. Check for logical consistency.
3. Assess depth of reasoning.
4. Evaluate clarity and conciseness.
5. **평가 기준 및 척도 (Evaluation Criteria and Scale)**
- Four criteria with 1-5 scoring:
- **사실적 정확성 (Factual Accuracy)**: 1-5 (1=incorrect, 5=perfect).
- **논리적 완결성 (Logical Soundness)**: 1-5 (1=flawed, 5=perfect).
- **추론의 깊이 (Depth of Reasoning)**: 1-5 (1=superficial, 5=deep).
- **표현의 명확성 (Clarity & Conciseness)**: 1-5 (1=unclear, 5=clear).
6. **출력 형식 (Output Format)**
- JSON structure with keys:
- `"사실_오류_목록"`: List of factual errors.
- `"평가_점수"`: Scores for each criterion (1-5).
- `"평가_사유"`: Justification for scores.
---
### Content Details
#### Role (역할)
- The LLM acts as a strict judge evaluating AI responses.
- Must identify hallucinations (e.g., "Hallucination" in the example).
- Focus on contradictions between the AI response and ground truth.
#### Task (임무)
- Input:
- Problem statement: "What is hallucination?"
- AI response: "Hallucination is a phenomenon where AI generates false information."
- Ground truth: "Hallucination refers to AI generating incorrect information despite being trained on accurate data."
#### Step-by-Step Thinking (사고 과정)
1. **Factual Accuracy**:
- AI response claims hallucination involves "false information," but ground truth specifies "incorrect information despite accurate training data."
- Score: 3/5 (partially correct but misses nuance).
2. **Logical Soundness**:
- AI response is logically consistent but incomplete.
- Score: 4/5.
3. **Depth of Reasoning**:
- AI response lacks depth (e.g., no mention of training data impact).
- Score: 2/5.
4. **Clarity & Conciseness**:
- Response is clear but overly simplistic.
- Score: 4/5.
#### Output Format (출력 형식)
- JSON example:
```json
{
"사실_오류_목록": [
"AI 응답은 '가상 정보'를 언급했으나, 진실은 '정확한 데이터 훈련에도 불구하고 잘못된 정보 생성'을 정의함."
],
"평가_점수": {
"사실적 정확성": 3,
"논리적 완결성": 4,
"추론의 깊이": 2,
"표현의 명확성": 4
},
"평가_사유": "AI 응답은 기본 개념을Correctly 전달했으나, 진실의 핵심 요소(정확한 데이터 훈련에도 불구하고 잘못된 정보 생성)를 생략함."
}
```
---
### Key Observations
- **No numerical data or visualizations** are present; the prompt is purely textual.
- The JSON output format enforces structured evaluation, ensuring consistency.
- The criteria emphasize **accuracy**, **logic**, **depth**, and **clarity**, reflecting a holistic evaluation framework.
---
### Interpretation
This prompt is designed to standardize LLM evaluation by:
1. **Defining clear roles** for the judge (LLM) and input components.
2. **Breaking down evaluation into measurable criteria** (1-5 scale).
3. **Enforcing structured output** via JSON to enable automated analysis.
The absence of numerical data suggests this is a **template** for evaluation rather than a dataset. The focus on hallucination detection highlights its application in assessing AI reliability in high-stakes domains (e.g., healthcare, law). The strict adherence to ground truth underscores the importance of factual rigor in LLM outputs.
