## Chart/Diagram Type: Error Analysis in VirtualHome and BEHAVIOR Trajectories ### Overview The image presents a comparative analysis of runtime errors encountered in two simulated environments: VirtualHome and BEHAVIOR. Each environment showcases four distinct error types: "Wrong Order," "Missing Step," "Affordance Error," and "Additional Step." For each error type, the image provides details such as the model used, task name, task ID, and specific conditions or states that triggered the error. ### Components/Axes **Header:** * "VirtualHome: Trajectory – Runtime Error" (Top section) * "BEHAVIOR: Trajectory – Runtime Error" (Bottom section) **Error Categories (for both VirtualHome and BEHAVIOR):** * Wrong Order * Missing Step * Affordance Error * Additional Step **Information within each error category:** * Model: Name of the AI model used. * Task Name: Description of the task being performed. * Task ID: Unique identifier for the task. * Action/Steps: List of actions performed or attempted. * Precondition: Conditions that should be met before an action. * Historical State: State of the environment at a previous time. * Affordance: Description of why an action could not be performed. * Current State: The current state of the environment. * Expected State: The expected state of the environment. ### Detailed Analysis or ### Content Details **VirtualHome Errors:** * **Wrong Order:** * Model: Gemini 1.5 Flash * Task Name: Drink * Task ID: scene\_1\_171\_2 * Actions: PUTBACK(cup.100, sink.231), DRINK(cup.1000) * Precondition: holds(cup.1000) = False * Historical State: holds(cup.1000) = False * **Missing Step:** * Model: Gemini 1.5 Flash * Task Name: Wash hands * Task ID: scene\_1\_813\_2 * Actions: WALK(bathroom.1), RINSE(hands\_both.1000) * Precondition: next\_to(sink.42) = False, holds(soap.100) = False * Historical State: next\_to(sink.42) = False, holds(soap.100) = False * **Affordance Error:** * Model: Mixtral 8x22b MOE * Task Name: Work * Task ID: scene\_1\_670\_2 * Actions: WALK(home\_office.319), FIND(mouse.413), TYPE(mouse.413) * Affordance: mouse.413 can't be typed * **Additional Step:** * Model: Mistral Large * Task Name: Set up table * Task ID: scene\_1\_93\_1 * Action: STANDUP() * Current State: stand\_up(character.45) = True * Expected State: stand\_up(character.45) = False **BEHAVIOR Errors:** * **Wrong Order:** * Model: GPT-4o * Task Name: Chopping Vegetables * Actions: RIGHT\_GRASP(carving\_knife\_69), LEFT\_GRASP(tomato\_61), SLICE(tomato\_61) * Precondition: not\_in\_hand(tomato\_61) = False * Historical State: not\_in\_hand(tomato\_61) = True * **Missing Step:** * Model: GPT-4o * Task Name: Cleaning bathtubs * Actions: stained(bathtub\_35), RIGHT\_GRASP(scrub\_brush\_0), CLEAN(bathtub\_35) * Precondition: soaked(scrub\_brush\_0) = False * Historical State: soaked(scrub\_brush\_0) = False * **Affordance Error:** * Model: Claude-3 Sonnet * Task Name: Bottling fruit * Actions: sliced(strawberry\_0), RIGHT\_TRANSFER\_CONTENTS\_INSIDE(strawberry\_0) * Affordance: strawberry\_0 is sliced and not interactable. Should interact with strawberry\_0\_part0 and strawberry\_0\_part1 * **Additional Step:** * Model: Claude-3 Opus * Task Name: Cleaning up the kitchen * Action: OPEN(top\_cabinet\_27), OPEN(top\_cabinet\_27) * Current State: open(top\_cabinet\_27) = True * Expected State: open(top\_cabinet\_27) = False ### Key Observations * Both VirtualHome and BEHAVIOR environments exhibit similar types of errors (Wrong Order, Missing Step, Affordance Error, Additional Step). * Different AI models are used in each environment (e.g., Gemini, Mixtral, GPT, Claude). * The errors are related to common household tasks (e.g., drinking, washing hands, chopping vegetables, cleaning). * Preconditions, historical states, current states, and expected states are used to diagnose the errors. ### Interpretation The data suggests that AI agents, when performing tasks in simulated environments, can encounter various types of errors related to task execution. These errors can stem from incorrect action sequences (Wrong Order), omission of necessary steps (Missing Step), inability to interact with objects in the environment (Affordance Error), or execution of unnecessary actions (Additional Step). The use of different AI models in VirtualHome and BEHAVIOR highlights the variability in performance and error patterns across different AI architectures. The specific details provided for each error type (e.g., preconditions, historical states) offer valuable insights for debugging and improving the AI agents' task execution capabilities. The errors observed in common household tasks indicate that even seemingly simple tasks can pose challenges for AI agents, requiring careful consideration of action sequences, object affordances, and environmental states. The discrepancies between current and expected states further emphasize the need for robust error detection and recovery mechanisms in AI systems.