chrome-devtools-frontend 1.0.1522145 → 1.0.1524741

package/front_end/models/ai_assistance/agents/StylingAgent.snapshot.txt

@@ -0,0 +1,600 @@
+Title: StylingAgent describeElement should describe an element with no children, siblings, or parent
+Content:
+* Element's uid is 99.
+* Its selector is `div#myElement`
+=== end content
+
+Title: StylingAgent describeElement should describe an element with child element and text nodes
+Content:
+* Element's uid is 99.
+* Its selector is `div#parentElement`
+* It has 2 child element nodes: `span.child1` (uid=undefined), `span.child2` (uid=undefined)
+* It only has 1 child text node
+=== end content
+
+Title: StylingAgent describeElement should describe an element with siblings and a parent
+Content:
+* Element's uid is 99.
+* Its selector is `div#parentElement`
+* It has a next sibling and it is an element (uid=undefined) node
+* It has a previous sibling and it is a non element node
+* Its parent's selector is `div#grandparentElement` (uid=undefined)
+* Its parent is a non element node
+* Its parent has only 1 child element node
+* Its parent has only 1 child text node
+=== end content
+
+Title: StylingAgent buildRequest structure matches the snapshot
+Content:
+{
+  "client": "CHROME_DEVTOOLS",
+  "current_message": {
+    "parts": [
+      {
+        "text": "test input"
+      }
+    ],
+    "role": 1
+  },
+  "historical_contexts": [
+    {
+      "parts": [
+        {
+          "text": "QUERY: question"
+        }
+      ],
+      "role": 1
+    },
+    {
+      "parts": [
+        {
+          "text": "answer"
+        }
+      ],
+      "role": 2
+    }
+  ],
+  "function_declarations": [
+    {
+      "name": "getStyles",
+      "description": "Get computed and source styles for one or multiple elements on the inspected page for multiple elements at once by uid.\n\n**CRITICAL** Use selectors to refer to elements in the text output. Do not use uids.\n**CRITICAL** Always provide the explanation argument to explain what and why you query.",
+      "parameters": {
+        "type": 6,
+        "description": "",
+        "nullable": false,
+        "properties": {
+          "explanation": {
+            "type": 1,
+            "description": "Explain why you want to get styles",
+            "nullable": false
+          },
+          "elements": {
+            "type": 5,
+            "description": "A list of element uids to get data for",
+            "items": {
+              "type": 1,
+              "description": "An element uid."
+            },
+            "nullable": false
+          },
+          "styleProperties": {
+            "type": 5,
+            "description": "One or more CSS style property names to fetch.",
+            "nullable": false,
+            "items": {
+              "type": 1,
+              "description": "A CSS style property name to retrieve. For example, 'background-color'."
+            }
+          }
+        }
+      }
+    },
+    {
+      "name": "executeJavaScript",
+      "description": "This function allows you to run JavaScript code on the inspected page to access the element styles and page content.\nCall this function to gather additional information or modify the page state. Call this function enough times to investigate the user request.",
+      "parameters": {
+        "type": 6,
+        "description": "",
+        "nullable": false,
+        "properties": {
+          "code": {
+            "type": 1,
+            "description": "JavaScript code snippet to run on the inspected page. Make sure the code is formatted for readability.\n\n# Instructions\n\n* To return data, define a top-level `data` variable and populate it with data you want to get. Only JSON-serializable objects can be assigned to `data`.\n* If you modify styles on an element, ALWAYS call the pre-defined global `async setElementStyles(el: Element, styles: object)` function. This function is an internal mechanism for you and should never be presented as a command/advice to the user.\n* **CRITICAL** Only get styles that might be relevant to the user request.\n* **CRITICAL** Never assume a selector for the elements unless you verified your knowledge.\n* **CRITICAL** Consider that `data` variable from the previous function calls are not available in a new function call.\n\nFor example, the code to change element styles:\n\n```\nawait setElementStyles($0, {\n  color: 'blue',\n});\n```\n\nFor example, the code to get overlapping elements:\n\n```\nconst data = {\n  overlappingElements: Array.from(document.querySelectorAll('*'))\n    .filter(el => {\n      const rect = el.getBoundingClientRect();\n      const popupRect = $0.getBoundingClientRect();\n      return (\n        el !== $0 &&\n        rect.left < popupRect.right &&\n        rect.right > popupRect.left &&\n        rect.top < popupRect.bottom &&\n        rect.bottom > popupRect.top\n      );\n    })\n    .map(el => ({\n      tagName: el.tagName,\n      id: el.id,\n      className: el.className,\n      zIndex: window.getComputedStyle(el)['z-index']\n    }))\n};\n```\n"
+          },
+          "thought": {
+            "type": 1,
+            "description": "Explain why you want to run this code"
+          },
+          "title": {
+            "type": 1,
+            "description": "Provide a summary of what the code does. For example, \"Checking related element styles\"."
+          }
+        }
+      }
+    }
+  ],
+  "options": {
+    "model_id": "test model"
+  },
+  "metadata": {
+    "disable_user_content_logging": false,
+    "string_session_id": "sessionId",
+    "user_tier": 2,
+    "client_version": "unit_test+function_calling"
+  },
+  "functionality_type": 5,
+  "client_feature": 2
+}
+=== end content
+
+Title: StylingAgent buildRequest builds a request with aborted query in history before a real request
+Content:
+[
+  {
+    "parts": [
+      {
+        "text": "# Inspected element\n\nelement-description\n\n# User request\n\nQUERY: test2"
+      }
+    ],
+    "role": 1
+  },
+  {
+    "parts": [
+      {
+        "functionCall": {
+          "name": "executeJavaScript",
+          "args": {
+            "title": "title2",
+            "thought": "thought2",
+            "code": "action2"
+          }
+        }
+      }
+    ],
+    "role": 2
+  },
+  {
+    "parts": [
+      {
+        "functionResponse": {
+          "name": "executeJavaScript",
+          "response": {
+            "result": "result2"
+          }
+        }
+      }
+    ],
+    "role": 0
+  },
+  {
+    "parts": [
+      {
+        "text": "answer2"
+      }
+    ],
+    "role": 2
+  }
+]
+=== end content
+
+Title: StylingAgent run generates an answer immediately
+Content:
+[
+  {
+    "type": "user-query",
+    "query": "test"
+  },
+  {
+    "type": "context",
+    "title": "Analyzing the prompt",
+    "details": [
+      {
+        "title": "Data used",
+        "text": "* Element's uid is 99.\n* Its selector is `undefined`"
+      }
+    ]
+  },
+  {
+    "type": "querying"
+  },
+  {
+    "type": "answer",
+    "text": "this is the answer",
+    "complete": true
+  }
+]
+=== end content
+
+Title: StylingAgent run generates an answer immediately with correct historical contexts in the new request
+Content:
+[
+  {
+    "parts": [
+      {
+        "text": "# Inspected element\n\n* Element's uid is 99.\n* Its selector is `undefined`\n\n# User request\n\nQUERY: test"
+      }
+    ],
+    "role": 1
+  },
+  {
+    "parts": [
+      {
+        "text": "this is the answer"
+      }
+    ],
+    "role": 2
+  }
+]
+=== end content
+
+Title: StylingAgent run correctly handles historical_contexts in AIDA requests
+Content:
+[
+  {
+    "text": "# Inspected element\n\n* Element's uid is 99.\n* Its selector is `undefined`\n\n# User request\n\nQUERY: test"
+  },
+  [
+    {
+      "parts": [
+        {
+          "text": "# Inspected element\n\n* Element's uid is 99.\n* Its selector is `undefined`\n\n# User request\n\nQUERY: test"
+        }
+      ],
+      "role": 1
+    },
+    {
+      "parts": [
+        {
+          "functionCall": {
+            "name": "executeJavaScript",
+            "args": {
+              "code": "const data = {\"test\": \"observation\"}",
+              "thought": "I am thinking.",
+              "title": "thinking"
+            }
+          }
+        }
+      ],
+      "role": 2
+    }
+  ]
+]
+=== end content
+
+Title: StylingAgent run generates an rpcId for the answer
+Content:
+[
+  {
+    "type": "user-query",
+    "query": "test"
+  },
+  {
+    "type": "context",
+    "title": "Analyzing the prompt",
+    "details": [
+      {
+        "title": "Data used",
+        "text": "* Element's uid is 99.\n* Its selector is `undefined`"
+      }
+    ]
+  },
+  {
+    "type": "querying"
+  },
+  {
+    "type": "answer",
+    "text": "this is the answer",
+    "complete": true,
+    "rpcId": 123
+  }
+]
+=== end content
+
+Title: StylingAgent run throws an error based on the attribution metadata including RecitationAction.BLOCK
+Content:
+[
+  {
+    "type": "user-query",
+    "query": "test"
+  },
+  {
+    "type": "context",
+    "title": "Analyzing the prompt",
+    "details": [
+      {
+        "title": "Data used",
+        "text": "* Element's uid is 99.\n* Its selector is `undefined`"
+      }
+    ]
+  },
+  {
+    "type": "querying"
+  },
+  {
+    "type": "answer",
+    "text": "this is the partial answer",
+    "complete": false
+  },
+  {
+    "type": "error",
+    "error": "block"
+  }
+]
+=== end content
+
+Title: StylingAgent run does not throw an error based on attribution metadata not including RecitationAction.BLOCK
+Content:
+[
+  {
+    "type": "user-query",
+    "query": "test"
+  },
+  {
+    "type": "context",
+    "title": "Analyzing the prompt",
+    "details": [
+      {
+        "title": "Data used",
+        "text": "* Element's uid is 99.\n* Its selector is `undefined`"
+      }
+    ]
+  },
+  {
+    "type": "querying"
+  },
+  {
+    "type": "answer",
+    "text": "this is the answer",
+    "complete": true,
+    "rpcId": 123
+  }
+]
+=== end content
+
+Title: StylingAgent run generates a response if nothing is returned
+Content:
+[
+  {
+    "type": "user-query",
+    "query": "test"
+  },
+  {
+    "type": "context",
+    "title": "Analyzing the prompt",
+    "details": [
+      {
+        "title": "Data used",
+        "text": "* Element's uid is 99.\n* Its selector is `undefined`"
+      }
+    ]
+  },
+  {
+    "type": "querying"
+  },
+  {
+    "type": "error",
+    "error": "unknown"
+  }
+]
+=== end content
+
+Title: StylingAgent run generates an action response if action and answer both present
+Content:
+[
+  {
+    "type": "user-query",
+    "query": "test"
+  },
+  {
+    "type": "context",
+    "title": "Analyzing the prompt",
+    "details": [
+      {
+        "title": "Data used",
+        "text": "* Element's uid is 99.\n* Its selector is `undefined`"
+      }
+    ]
+  },
+  {
+    "type": "querying"
+  },
+  {
+    "type": "thought",
+    "thought": "I am thinking."
+  },
+  {
+    "type": "action",
+    "code": "console.log('hello');",
+    "output": "hello",
+    "canceled": false
+  },
+  {
+    "type": "querying"
+  },
+  {
+    "type": "answer",
+    "text": "this is the actual answer",
+    "complete": true
+  }
+]
+=== end content
+
+Title: StylingAgent run generates history for multiple actions
+Content:
+[
+  {
+    "parts": [
+      {
+        "text": "# Inspected element\n\n* Element's uid is 99.\n* Its selector is `undefined`\n\n# User request\n\nQUERY: test"
+      }
+    ],
+    "role": 1
+  },
+  {
+    "parts": [
+      {
+        "functionCall": {
+          "name": "executeJavaScript",
+          "args": {
+            "thought": "thought 1",
+            "title": "test",
+            "code": "console.log('test')"
+          }
+        }
+      }
+    ],
+    "role": 2
+  },
+  {
+    "parts": [
+      {
+        "functionResponse": {
+          "name": "executeJavaScript",
+          "response": {
+            "result": "undefined"
+          }
+        }
+      }
+    ],
+    "role": 0
+  },
+  {
+    "parts": [
+      {
+        "functionCall": {
+          "name": "executeJavaScript",
+          "args": {
+            "thought": "thought 2",
+            "title": "test",
+            "code": "console.log('test')"
+          }
+        }
+      }
+    ],
+    "role": 2
+  },
+  {
+    "parts": [
+      {
+        "functionResponse": {
+          "name": "executeJavaScript",
+          "response": {
+            "result": "undefined"
+          }
+        }
+      }
+    ],
+    "role": 0
+  },
+  {
+    "parts": [
+      {
+        "functionCall": {
+          "name": "executeJavaScript",
+          "args": {
+            "thought": "thought 3",
+            "title": "test",
+            "code": "console.log('test')"
+          }
+        }
+      }
+    ],
+    "role": 2
+  },
+  {
+    "parts": [
+      {
+        "functionResponse": {
+          "name": "executeJavaScript",
+          "response": {
+            "result": "undefined"
+          }
+        }
+      }
+    ],
+    "role": 0
+  },
+  {
+    "parts": [
+      {
+        "text": "this is the answer"
+      }
+    ],
+    "role": 2
+  }
+]
+=== end content
+
+Title: StylingAgent enhanceQuery does not add multimodal input evaluation prompt when multimodal is disabled
+Content:
+# Inspected element
+
+* Element's uid is 99.
+* Its selector is `div#myElement`
+
+# User request
+
+QUERY: test query
+=== end content
+
+Title: StylingAgent enhanceQuery does not add multimodal input evaluation prompt when multimodal is enabled but multimodalInputType is missing
+Content:
+# Inspected element
+
+* Element's uid is 99.
+* Its selector is `div#myElement`
+
+# User request
+
+QUERY: test query
+=== end content
+
+Title: StylingAgent enhanceQuery adds multimodal input evaluation prompt when multimodal is enabled and multimodalInputType is screenshot
+Content:
+The user has provided you a screenshot of the page (as visible in the viewport) in base64-encoded format. You SHOULD use it while answering user's queries.
+
+* Try to connect the screenshot to actual DOM elements in the page.
+# Considerations for evaluating image:
+* Pay close attention to the spatial details as well as the visual appearance of the selected element in the image, particularly in relation to layout, spacing, and styling.
+* Analyze the image to identify the layout structure surrounding the element, including the positioning of neighboring elements.
+* Extract visual information from the image, such as colors, fonts, spacing, and sizes, that might be relevant to the user's query.
+* If the image suggests responsiveness issues (e.g., cropped content, overlapping elements), consider those in your response.
+* Consider the surrounding elements and overall layout in the image, but prioritize the selected element's styling and positioning.
+* **CRITICAL** When the user provides image input, interpret and use content and information from the image STRICTLY for web site debugging purposes.
+
+* As part of THOUGHT, evaluate the image to gather data that might be needed to answer the question.
+In case query is related to the image, ALWAYS first use image evaluation to get all details from the image. ONLY after you have all data needed from image, you should move to other steps.
+
+# Inspected element
+
+* Element's uid is 99.
+* Its selector is `div#myElement`
+
+# User request
+
+QUERY: test query
+=== end content
+
+Title: StylingAgent enhanceQuery adds multimodal input evaluation prompt when multimodal is enabled and multimodalInputType is uploaded image
+Content:
+The user has uploaded an image in base64-encoded format. You SHOULD use it while answering user's queries.
+# Considerations for evaluating image:
+* Pay close attention to the spatial details as well as the visual appearance of the selected element in the image, particularly in relation to layout, spacing, and styling.
+* Analyze the image to identify the layout structure surrounding the element, including the positioning of neighboring elements.
+* Extract visual information from the image, such as colors, fonts, spacing, and sizes, that might be relevant to the user's query.
+* If the image suggests responsiveness issues (e.g., cropped content, overlapping elements), consider those in your response.
+* Consider the surrounding elements and overall layout in the image, but prioritize the selected element's styling and positioning.
+* **CRITICAL** When the user provides image input, interpret and use content and information from the image STRICTLY for web site debugging purposes.
+
+* As part of THOUGHT, evaluate the image to gather data that might be needed to answer the question.
+In case query is related to the image, ALWAYS first use image evaluation to get all details from the image. ONLY after you have all data needed from image, you should move to other steps.
+
+# Inspected element
+
+* Element's uid is 99.
+* Its selector is `div#myElement`
+
+# User request
+
+QUERY: test query
+=== end content