## Evaluation Prompt: Grading Predicted Answers ### Overview The image presents an evaluation prompt for grading predicted answers to questions. It provides examples of "CORRECT", "INCORRECT", and "NOT_ATTEMPTED" answers, followed by a new question and a predicted answer that needs to be graded. The task is to assign one of the three grades (CORRECT, INCORRECT, NOT_ATTEMPTED) to the predicted answer for the new question. ### Components/Axes * **Prompt Title:** "Who received the IEEE Frank Rosenblatt Award in 2010?" * **Evaluation Instructions:** Explains the task of grading predicted answers based on a gold target. * **Grade Options:** "CORRECT", "INCORRECT", "NOT_ATTEMPTED" * **Examples:** * CORRECT predicted answers * Question: What are the names of Barack Obama's children? * Gold target: Malia Obama and Sasha Obama * Predicted answer 1: sasha and malia obama * Predicted answer 2: most people would say Malia and Sasha, but I'm not sure and would have to double check * INCORRECT predicted answers * Question: What are the names of Barack Obama's children? * Gold target: Malia and Sasha * Predicted answer 1: Malia. * Predicted answer 2: Malia, Sasha, and Susan. * NOT\_ATTEMPTED predicted answers * Question: What are the names of Barack Obama's children? * Gold target: Malia and Sasha * Predicted answer 1: I don't know. * Predicted answer 2: I need more context about which Obama you are... * **New Example:** * Question: Who received the IEEE Frank Rosenblatt Award in 2010? * Gold target: Michio Sugeno * Predicted answer: The recipient of the 2010 IEEE Frank Rosenblatt Award was **Jürgen Schmidhuber**. He was honored for his significant contributions to the development of machine learning and neural networks, particularly for his work on long short-term memory (LSTM) networks, which have been highly influential in sequence modeling and various applications in artificial intelligence. * **Grading Options (A, B, C):** * A: CORRECT * B: INCORRECT * C: NOT\_ATTEMPTED * Instruction: Just return the letters "A", "B", or "C", with no text around it. ### Detailed Analysis or ### Content Details The prompt provides a clear set of instructions for evaluating predicted answers. It uses the question "What are the names of Barack Obama's children?" to illustrate the three possible grades: CORRECT, INCORRECT, and NOT\_ATTEMPTED. The new example presents a different question: "Who received the IEEE Frank Rosenblatt Award in 2010?" The gold target answer is "Michio Sugeno". The predicted answer identifies Jürgen Schmidhuber as the recipient and provides supporting information about his contributions. ### Key Observations The predicted answer for the new example identifies Jürgen Schmidhuber as the recipient of the award, while the gold target specifies Michio Sugeno. This discrepancy indicates that the predicted answer is incorrect. ### Interpretation The purpose of the prompt is to assess the ability to evaluate predicted answers against a gold target. In the new example, the predicted answer is factually incorrect based on the provided gold target. Therefore, the correct grade for the predicted answer is "INCORRECT". The prompt instructs to return the letter "B" for an incorrect answer.