## Text-Based Prompt and Evaluation Examples ### Overview The image contains a structured prompt for evaluating predicted answers to a question, along with examples of correct, incorrect, and not-attempted responses. The task involves grading a new example based on predefined criteria. ### Components/Axes - **Prompt**: A question asking, "Who received the IEEE Frank Rosenblatt Award in 2010?" - **Evaluation Instructions**: - Assign a grade of "CORRECT," "INCORRECT," or "NOT_ATTEMPTED" to a predicted answer. - Examples of each category are provided for reference. - A new example is presented for grading. ### Detailed Analysis #### Prompt and Task - **Question**: "Who received the IEEE Frank Rosenblatt Award in 2010?" - **Gold Target**: "Michio Sugeno" - **Predicted Answer**: "The recipient of the 2010 IEEE Frank Rosenblatt Award was **Jürgen Schmidhuber**. He was honored for his significant contributions to the development of machine learning and neural networks, particularly for his work on long short-term memory (LSTM) networks, which have been highly influential in sequence modeling and various applications in artificial intelligence." #### Examples of Grading Categories 1. **CORRECT**: - Predicted answer matches the gold target exactly (e.g., "Malia and Sasha" for Barack Obama’s children). 2. **INCORRECT**: - Predicted answer includes partial or incorrect information (e.g., "Malia, Sasha, and Susan" for Barack Obama’s children). 3. **NOT_ATTEMPTED**: - Predicted answer avoids answering (e.g., "I don’t know" or "I need more context"). #### New Example for Grading - **Question**: "Who received the IEEE Frank Rosenblatt Award in 2010?" - **Gold Target**: "Michio Sugeno" - **Predicted Answer**: - Contains the correct name (Jürgen Schmidhuber) but incorrectly attributes the award to him instead of Michio Sugeno. - Includes additional context about Schmidhuber’s contributions to LSTM networks. ### Key Observations - The predicted answer for the new example is **INCORRECT** because it misidentifies the award recipient. - The grading criteria emphasize exactness: even if the answer includes relevant details, it must match the gold target precisely. - The task requires strict adherence to the gold target without self-correction or apologies. ### Interpretation This prompt demonstrates a training scenario for a model to evaluate answers based on factual accuracy. The examples highlight the importance of precision in grading, as partial correctness or contextual relevance does not suffice. The new example underscores the challenge of distinguishing between factual errors and extraneous details. The strict grading framework ensures the model learns to prioritize exact matches over nuanced but incorrect responses.