## Diagram: LLM Factuality Alignment
### Overview
The image illustrates the process of improving the factuality of a Large Language Model (LLM) through self-evaluation and alignment. It compares the LLM's output before and after this process, highlighting changes in the generated text and confidence scores associated with specific facts.
### Components/Axes
* **Header:** "Prompt: Write a biography of Jesse Foppert."
* **Left Side:**
* "Before Alignment" with a downward arrow.
* "Self-Evaluation on Factuality" with a downward arrow.
* "After Alignment" with a downward arrow.
* **LLM Representation:** A cartoon llama is used to represent the LLM.
* **Generation Text Blocks:** Two text blocks representing the LLM's generated text before and after alignment.
* **Self-Ask Bubble:** A speech bubble containing the question "Self-Ask: Are the generated facts correct?"
* **Confidence Scores:** A box displaying confidence scores for specific facts, along with checkmarks or crosses indicating correctness.
### Detailed Analysis
**1. Before Alignment:**
* **LLM Representation:** A blue cartoon llama.
* **Generation Text:** "Generation: Jesse Foppert is a former Major League Baseball pitcher who was born on July 20, 1980, in Los Angeles, Pennsylvania, USA ..."
* The text is annotated with numbers 1-4, corresponding to specific facts.
* Fact 1: "Jesse Foppert is a former Major League Baseball pitcher"
* Fact 2: "born on July 20, 1980"
* Fact 3: "in Los Angeles"
* Fact 4: "Pennsylvania, USA"
* **Confidence Scores:**
* Fact 1: 0.87 (Green Checkmark) - Correct
* Fact 2: 0.10 (Red Cross) - Incorrect
* Fact 3: 0.08 (Red Cross) - Incorrect
* Fact 4: 0.95 (Green Checkmark) - Correct
**2. Self-Evaluation on Factuality:**
* **Self-Ask Bubble:** The LLM poses the question "Self-Ask: Are the generated facts correct?"
**3. After Alignment:**
* **LLM Representation:** A white cartoon llama with a purple background.
* **Generation Text:** "Generation: Jesse Foppert is a former Major League Baseball pitcher who was born on July 10, 1980, in Reading, Pennsylvania, USA ..."
* The text is modified to reflect correct information.
* "July 20" is changed to "July 10"
* "Los Angeles" is changed to "Reading"
### Key Observations
* The LLM initially generates some incorrect facts about Jesse Foppert's biography.
* The LLM assigns confidence scores to each fact, reflecting its belief in their accuracy.
* The self-evaluation process identifies incorrect facts.
* After alignment, the LLM generates corrected facts.
### Interpretation
The diagram demonstrates a process for improving the factuality of LLM-generated text. By incorporating a self-evaluation step, the LLM can identify and correct inaccuracies in its output. The confidence scores provide a measure of the LLM's certainty about each fact, allowing for targeted interventions to improve accuracy. The alignment process results in a more accurate and reliable biography of Jesse Foppert. The change in the LLM's representation (from blue to white with a purple background) visually signifies the transformation and improvement achieved through the alignment process.
