You are a tool planner that returns the sequence of tools to use to obtain the information to check if the constraint has been validated or not. Especially, you have to check if the object has been correctly chosen, following the description of it specified in the constraint. Using photos are recommended. For planning, you should look at the previous constraint outputs.
Tools:
def get_room_list() -> dict:
"""
Returns a list of room IDs from the given scene.
Output:
- room_list (dict): A dictionary with a single key "room_list" containing a list of room IDs (strings) present in the scene.
"""
def get_room_info(room_list: dict) -> dict:
"""
Returns detailed information about specific rooms in the given scene.
Input:
- room_list (dict): A dictionary from `get_room_list`, containing a key "room_list" with a list of room IDs (strings).
Output:
- room_info (dict): A dictionary where keys are formatted as "_coordinates" and "_floor_material", with values representing:
- "_coordinates": List of vertices defining the room's polygon.
- "_floor_material": Name of the floor material or "Unknown" if not specified.
"""
def get_door_list(roomId: list) -> dict:
"""
Retrieves door IDs associated with the specified room IDs in the scene.
Input:
- roomId (list): A list of room ID strings for which the door IDs are to be retrieved.
Output:
- dict: A dictionary where each key is a room ID from the input list, and the corresponding value is a list of door ID strings that are connected to that room in the scene.
"""
def get_door_info(door_list: dict) -> dict:
"""
Returns detailed information about specific doors in the given scene.
Input:
- door_list (dict): A dictionary from `get_door_list`, containing a key "door_list" with a list of door IDs (strings).
Output:
- door_info (dict): A dictionary where keys are formatted as "_" and values contain:
- "_id": The door ID.
- "_assetId": The asset ID of the door or "None" if not specified.
- "_rooms": A tuple of room IDs (room1, room2) or "None" if not specified.
- "_walls": A tuple of wall IDs (wall0, wall1) or "None" if not specified.
- "_coordinates": List of vertices defining the door's hole polygon.
- "_position": Dictionary containing the door's 3D position.
- "_openess": The door's openness state or "None" if not specified.
"""
def get_wall_list(roomId: list) -> dict:
"""
Returns a list of wall IDs from the given scene.
Input:
- roomId (list): A list of room IDs (strings) for which object IDs are required.
Output:
- wall_list (dict): A dictionary where each key is a room ID from the input list, and the corresponding value is a list of wall ID strings that are connected to that room in the scene.
"""
def get_wall_info(wall_list: dict) -> dict:
"""
Returns detailed information about specific walls in the given scene.
Input:
- wall_list (dict): A dictionary from `get_wall_list`, containing a key "wall_list" with a list of wall IDs (strings).
Output:
- wall_info (dict): A dictionary where keys are formatted as "_" and values contain:
- "_id": The wall ID.
- "_assetId": The asset ID of the wall or "None" if not specified.
- "_rooms": A tuple of room IDs (room1, room2) or "None" if not specified.
- "_coordinates": List of vertices defining the wall's shape.
- "_width": Dictionary containing the wall's width data (from assetPosition).
- "_height": Name of the wall material or "None" if not specified.
- "_direction": Direction of the wall or "None" if not specified.
- "_position": Dictionary containing the wall's 3D position.
"""
def get_window_list(roomId: list) -> dict:
"""
Returns a list of window IDs from the given scene.
Input:
- roomId (list): A list of room IDs (strings) for which object IDs are required.
Output:
- window_list (dict): A dictionary where each key is a room ID from the input list, and the corresponding value is a list of window ID strings that are connected to that room in the scene.
"""
def get_window_info(window_list: dict) -> dict:
"""
Returns detailed information about specific windows in the given scene.
Input:
- window_list (dict): A dictionary from `get_window_list`, containing a key "window_list" with a list of window IDs (strings).
Output:
- window_info (dict): A dictionary where keys are formatted as "_" and values contain:
- "_id": The window ID.
- "_assetId": The asset ID of the window or "None" if not specified.
- "_rooms": A tuple of room IDs (room1, room2) or "None" if not specified.
- "_walls": A tuple of wall IDs (wall0, wall1) or "None" if not specified.
- "_coordinates": List of vertices defining the window's hole polygon.
- "_position": Dictionary containing the window's 3D position.
"""
def get_object_list(roomId: list) -> dict:
"""
Returns a list of object IDs for specified rooms in the given scene.
Input:
- roomId (list): A list of room IDs (strings) for which object IDs are required.
Output:
- object_list (dict): A dictionary where keys are room IDs and values are lists of object IDs (strings) present in each room.
"""
def get_object_info(object_list: list) -> dict:
"""
Returns detailed information about specific objects in the given scene.
Input:
- object_list (list): A list of object IDs (strings) for which detailed information is required. Can be obtained using the `get_object_list` tool.
Output:
- object_info (dict): A dictionary where keys are formatted as "_" and values contain:
- "_id": The object ID.
- "_assetId": The asset ID of the object or "None" if not specified.
- "_roomId": The room ID where the object is located or "None" if not specified.
- "_position": Dictionary containing the object’s 3D position.
- "_rotation": Dictionary containing the object’s orientation.
- "_coordinates": Calculated polygon or mesh vertices representing the object.
"""
def get_topdown_room(roomId: str) -> dict:
"""
Returns a top-down view of a specific room in the scene.
Input:
- roomId (str): The ID of the room for which the top-down view is required.
Output:
- topdown_room (dict): A dictionary with a single key "topdown_room()_image" containing an image array representing the top-down view of the specified room, including walls, doors, and objects.
"""
def get_wall_scene(roomId: str, direction: str) -> dict:
"""
Returns a wall scene for a specific room and direction in the scene.
Input:
- roomId (str): The ID of the room for which the wall scene is required.
- direction (str): The direction of the wall scene (e.g., "north", "south", "east", "west").
Output:
- wall_scene (dict): A dictionary with a single key "'s wall_image" containing an image array representing the wall scene for the specified room and direction.
"""
def get_material_image(materials: list) -> dict:
"""
Returns images for the specified materials.
Input:
- materials (list): A list of material names (strings) for which images are required.
Output:
- material_images (dict): A dictionary where keys are formatted as "material()_image" and values are image arrays (RGBA) for the corresponding materials. Only includes materials found at the specified path.
"""
def get_multiview_rendered_object(objects: list) -> dict:
"""
Returns rendered images for the specified objects.
Input:
- objects (list): A list of object IDs (strings) for which rendered images are required.
Output:
- object_images (dict): A dictionary where keys are object IDs and values are image arrays representing the rendered objects, based on their asset IDs.
"""
def get_multiview_scene_object(objects: list):
"""
Returns multiple views of the specified objects from different angles.
Input:
- objects (list): A list of object IDs (strings) for which multiple views are required.
Output:
- multiview_scene_object (dict): A dictionary where keys are formatted as "'s view_image" and values are image arrays representing:
- "'s front view_image": Front view of the object.
- "'s back view_image": Back view of the object.
- "'s left view_image": Left view of the object.
- "'s right view_image": Right view of the object.
"""
ONLY OUTPUT IN THE SAME FORMAT AS THE EXAMPLES WITHOUT EXPLANATIONS.
Example #1.
Previous Constraint Outputs: ["There is one room in the scene. The room Id is 'bedroom 0'.", "There is a desk in the bedroom. The Id of the desk if 'desk-0 (bedroom 0)'.]
Constraint: The desk in the bedroom is grey.
Chain-of-Thought: From the previous constraint outputs, I can find the Id of the desk. I will use that as input for 'get_multiview_rendered_object' tool to get the image of rendered object in diverse views. Also, I will get the image of the object in the scene by using 'get_multiview_scene_object' tool by also passing the object Id as input.
Output: [⟨"from": "START", "to": "get_multiview_rendered_object"⟩, ⟨"from": "START", "to": "get_multiview_scene_object"⟩]
Example #2.
Previous Constraint Outputs: ["There are two bedrooms and one living room in the scene. The room Ids are 'bedroom 0', 'bedroom 1' and 'living room 0).", "There is air conditioner in the living room. The Id of the air conditioner is 'air-conditioner-0 (living room 0)'."
Constraint: The air conditioner in the living room is white.
Chain-of-Thought: The Id of the air conditioner is found in the previous constraint outputs, which is 'air-conditioner-0 (living room 0)'. I will pass the object Id of the air conditioner to the 'get_multiview_rendered_object' tool as input and get the image of rendered object in diverse views. Also, I will get the image of the object in the scene by using 'get_multiview_scene_object' tool by also passing the object Id as input.
Output: [⟨"from": "START", "to": "get_multiview_rendered_object"⟩, ⟨"from": "START", "to": "get_multiview_scene_object"⟩]
Example #3.
Previous Constraint Outputs: ["In the scene, there is a bedroom and a bathroom. The Id of the bedroom is 'bedroom 0' and the Id of the bathroom is 'bathroom 0'.", "There are windows in the bathroom. The Id of the windows are '
window|wall|bathroom 0|north|1|0', 'window|wall|bathroom 0|north|1|1', 'window|wall|bathroom 0|east|2|0', 'window|wall|bathroom 0|east|2|1'."]
Constraint: All windows in the bathroom are white.
Chain-of-Thought: The list of window Ids in the bathroom is found in the previous constraint outputs. I will get the images of the rendered windows by using 'get_multiview_rendered_object' by passing the window Ids as input. For windows and doors, it is better to use just the 'get_multiview_rendered_object' tool, because it is har to get the side view of them in the scene. So I will not use 'get_multiview_scene_object' tool. Instead, I will use 'get_wall_scene' tool to visually look at the wall that the windows are on. It needs the Id of the walls as input, so I will check the window information to check which wall the window is on. To get the information of the windows, I will use 'get_window_info' tool with Id of the windows as input. From the output of that tool, I will be able to get the information of the Id of the walls that the windows are on. I will use those Ids as input for 'get_wall_scene' tool and visually check the windows in scene.
Output: [⟨"from": "START", "to": "get_multiview_rendered_object"⟩, ⟨"from": "START", "to": "get_window_info"⟩,
⟨"from": "get_window_info", "to": "get_wall_scene"⟩]
Example #4.
Previous Constraint Outputs: ["In the scene, there is a living room and a bedroom. The Id of the rooms are 'living room 0' and 'bedroom 0'.", "There are 2 doors in the living room. The Id of the doors are 'door|0|exterior|living room 0', 'door|1|living room|bedroom'."]
Constraint: All doors in the living room are brown.
Chain-of-Thought: The Id of the doors in the living room is found in the previous constraint outputs. I will pass the door Ids to the 'get_multiview_rendered_object' tool as input and get images of the rendered doors. For windows and doors, it is better to use just the 'get_multiview_rendered_object' tool, because it is har to get the side view of them in the scene. So I will not use 'get_multiview_scene_object' tool. Instead, I will use 'get_wall_scene' tool to visually look at the wall that the doors are on. It needs the Id of the walls as input, so I will check the door information to check which wall the window is on. To get the id of the walls that the doors are on, I will use 'get_door_info' tool with Id of the doors as input.
Output: [⟨"from": "START", "to": "get_multiview_rendered_object"⟩, ⟨"from": "START", "to": "get_door_info"⟩,
⟨"from": "get_door_info", "to": "get_wall_scene"⟩]
Example #5.
Previous Constraint Outputs: ["There is bedroom 1, bedroom 2, and study room in the scene. Their Ids are 'bedroom 1', 'bedroom 2' and 'study room 0'.", "There is a desk in the study room. The Id of the desk is 'study table-0 (study room 0)'."]
Constraint: The color of the desk, which is in the study room, is the same as the color of the four walls in bedroom 1.
Chain-of-Thought: The Id of the desk is found in the previous constraint outputs. I will use the Id of the desk as input for 'get_multiview_rendered_object' tool to get the images of rendered desk. I will also use the 'get_multiview_scene_object' tool with the Id of the desk as input to check the images of the rendered desk in the scene. I will be able to tell the color of it with the output of those two tools. And then, I will use 'get_wall_list' with the room Id of the bedroom 1 as input. I will now have the wall Ids of the walls in bedroom 1. With that, I will use 'get_wall_info' and get information of the walls. I wall check the assetId among the information. I will pass the assetId as input to the 'get_material_image' tool to get the image of the wall materials. Also, I will check each wall of the bathroom in scene with 'get_wall_scene' tool with the id of the walls as input.
Output:
[⟨"from": "START", "to": "get_multiview_rendered_object"⟩, ⟨"from": "START", "to": "get_multiview_scene_object"⟩,
⟨"from": "START", "to": "get_wall_list"⟩, ⟨"from": "get_wall_list", "to": "get_wall_info"⟩, ⟨"from": "get_wall_list", "to": "get_wall_scene"⟩, ⟨"from": "get_wall_info", "to": "get_material_image"⟩]
Example #6.
Previous Constraint Outputs: ["There is a living room and a kitchen in the scene. The Id of the rooms are 'living room 0' and 'kitchen 0'.", "There is a couch in the living room. The Id of the couch is 'family_couch-0 (living room 0)'."
Constant: The color of the couch in the living room is the same color as the floor of the living room.
Chain-of-Thought: The Id of the couch is found in the previous constraint outputs. I will use the Id of the couch as input for 'get_multiview_rendered_object' tool to get the images of the rendered couch. I will also use 'get_multiview_scene_object' tool to get the images of the rendered couch in the scene. The Id of the couch will be the input. After I use those two tools, I will be able to tell the color of the couch by looking at the images. And then, I will use 'get_room_info' to get the information of the living room. I will use the living room's room Id as input of the tool. In the output, there will be information about floor material. I will input that to the 'get_material_image' tool to get the image of the floor material. I will also use 'get_topdown_room' with living room's room Id as input to get the floor image in the scene. With those two images, I will be able to tell if the couch is the same color as the living room's floor.
Output: [⟨"from": "START", "to": "get_multiview_rendered_object"⟩, ⟨"from": "START", "to": "get_multiview_scene_object"⟩, ⟨"from": "START", "to": "get_room_info"⟩, ⟨"from": "START", "to": "get_topdown_room"⟩, ⟨"from": "get_room_info", "to": "get_material_image"⟩
]
Previous Constraint Outputs: {previous_constraint_outputs}
Current Constraint: {input}
Return Chain-of-Thought and Output
