pi-chrome 0.15.35 → 0.15.37
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +8 -0
- package/README.md +35 -11
- package/extensions/chrome-profile-bridge/browser-extension/manifest.json +1 -1
- package/extensions/chrome-profile-bridge/browser-extension/service_worker.js +87 -105
- package/extensions/chrome-profile-bridge/browser-extension/snapshot_injected.js +677 -0
- package/extensions/chrome-profile-bridge/index.ts +221 -7
- package/package.json +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -2,6 +2,14 @@
|
|
|
2
2
|
|
|
3
3
|
All notable user-facing changes to `pi-chrome`.
|
|
4
4
|
|
|
5
|
+
## 0.15.36 — 2026-06-03
|
|
6
|
+
|
|
7
|
+
- **Richer page observation.** `chrome_snapshot` now returns a concise, agent-friendly observation — structural layout/context, page hints, visible actions, form fields, a page map, query matches, and a diff of changes since the previous snapshot — instead of a raw JSON dump. New `mode` (`auto`/`interactive`/`forms`/`pageMap`/`text`/`changes`/`full`), `query`, and `maxTextChars` parameters let the agent zoom in instead of dumping the whole page.
|
|
8
|
+
- **New `chrome_find` tool.** Find elements, regions, or text by natural-language query (`'merge button'`, `'email error'`) and get ranked matches with stable uids and coordinates. Thin wrapper around `chrome_snapshot({ query })`.
|
|
9
|
+
- **New `chrome_inspect` tool.** Inspect one snapshot uid/selector deeply: nearby text, nearby actions, form context, ancestors, and a suggested click target. Falls back to a focused snapshot if the loaded extension predates `page.inspect`.
|
|
10
|
+
- **`includeSnapshot` now embeds the formatted snapshot.** `chrome_click`/`chrome_type`/`chrome_fill`/`chrome_key` with `includeSnapshot=true` append the fresh concise snapshot to the tool text so the agent can verify in one round trip.
|
|
11
|
+
- **Snapshot logic moved to a packaged `snapshot_injected.js`.** The MAIN-world snapshot/inspect implementation ships as an eval-free packaged script, shared `window.__PI_CHROME_STATE__` (same `el-` uid scheme) with the existing input helpers.
|
|
12
|
+
|
|
5
13
|
## 0.15.34 — 2026-06-01
|
|
6
14
|
|
|
7
15
|
- **Every tab Pi uses now joins the session group.** Previously only `chrome_tab new`/`group` created/used the `Pi Session: <name-or-id>` group; tabs Pi drove via `page.*` actions (navigate, click, type, snapshot, screenshot, etc.) on the existing/active tab stayed ungrouped. Now any ungrouped tab Pi interacts with is pulled into this session's group, so the user can see exactly which tabs Pi is driving. Tabs already in a group (the user's or another session's) are left untouched.
|
package/README.md
CHANGED
|
@@ -15,34 +15,58 @@ You: [keeps coding — agent never asked you to log in]
|
|
|
15
15
|
|
|
16
16
|
---
|
|
17
17
|
|
|
18
|
-
## 60-second install
|
|
18
|
+
## 60-second install instruction
|
|
19
|
+
|
|
20
|
+
To install pi-chrome, run the following command:
|
|
19
21
|
|
|
20
22
|
```bash
|
|
21
23
|
pi install npm:pi-chrome
|
|
22
24
|
```
|
|
23
25
|
|
|
24
|
-
|
|
26
|
+
|
|
27
|
+
Then in Pi, run the next command, which will:
|
|
28
|
+
|
|
29
|
+
1. Reveal the bundled browser-extension folder in Finder, and copy the folder path to your clipboard.
|
|
30
|
+
2. Pop open the chrome://extensions webpage in Chrome.
|
|
31
|
+
|
|
32
|
+
In the Chrome Extensions page it opened, **YOU WILL NEED TO**:
|
|
33
|
+
|
|
34
|
+
1. Turn on **developer mode** (top right).
|
|
35
|
+
2. Click the **load unpacked** button (top left).
|
|
36
|
+
3. Use **Cmd + Shift + G** (Mac) or **Ctrl + L** (Windows/Linux) to open the folder path field.
|
|
37
|
+
4. **Cmd + V** (Mac) or **Ctrl + V** (Windows/Linux) to paste the copied path and press Enter.
|
|
38
|
+
5. You're done with the chrome extensions page, and you can continue with the rest of the installation commands
|
|
25
39
|
|
|
26
40
|
```text
|
|
27
41
|
/chrome onboard
|
|
28
42
|
```
|
|
29
43
|
|
|
30
|
-
|
|
44
|
+
Reload Pi so the newly installed package is actually loaded:
|
|
45
|
+
|
|
46
|
+
```text
|
|
47
|
+
/reload
|
|
48
|
+
```
|
|
49
|
+
|
|
31
50
|
|
|
32
|
-
Verify
|
|
51
|
+
Verify the chrome connection:
|
|
33
52
|
|
|
34
53
|
```text
|
|
35
54
|
/chrome doctor
|
|
36
|
-
/chrome authorize
|
|
37
55
|
```
|
|
56
|
+
In the output, you just need to make sure the following line is present (It's okay if the other ones are still not checked):
|
|
57
|
+
|
|
58
|
+
✓ Chrome is connected (companion extension v0.15.36, responded in 11ms).
|
|
38
59
|
|
|
60
|
+
Lastly, authorize the current session by running:
|
|
39
61
|
```text
|
|
40
|
-
|
|
41
|
-
pi-chrome v<version>
|
|
42
|
-
• Local bridge: mode=server, url=http://127.0.0.1:17318
|
|
43
|
-
✓ Companion Chrome extension responding (ID: <chrome-extension-id>, ext v<version>)
|
|
62
|
+
/chrome authorize
|
|
44
63
|
```
|
|
45
64
|
|
|
65
|
+
Run the following once more, and you should see all the lines checked:
|
|
66
|
+
|
|
67
|
+
```text
|
|
68
|
+
/chrome doctor
|
|
69
|
+
```
|
|
46
70
|
---
|
|
47
71
|
|
|
48
72
|
## Try this in 30 seconds after install
|
|
@@ -135,12 +159,12 @@ Agents can verify page state immediately instead of blindly retrying.
|
|
|
135
159
|
|
|
136
160
|
## What an agent gets
|
|
137
161
|
|
|
138
|
-
**
|
|
162
|
+
**21 tools**, grouped by job. Every one runs against your already-open tabs.
|
|
139
163
|
|
|
140
164
|
| Category | Tools |
|
|
141
165
|
| --------------- | ---------------------------------------------------------------------------------------------- |
|
|
142
166
|
| **Tabs** | `chrome_tab` (list/new/activate/close/version), `chrome_launch` |
|
|
143
|
-
| **Inspect** | `chrome_snapshot` (
|
|
167
|
+
| **Inspect** | `chrome_snapshot` (concise observation: layout, actions, forms, page map, query matches, diff + stable uids), `chrome_find` (query → ranked uids), `chrome_inspect` (deep single-element context), `chrome_screenshot`, `chrome_evaluate` |
|
|
144
168
|
| **Navigate** | `chrome_navigate` (with optional `initScript` at `document_start`), `chrome_wait_for` |
|
|
145
169
|
| **Interact** | `chrome_click`, `chrome_type`, `chrome_fill`, `chrome_key`, `chrome_hover` |
|
|
146
170
|
| **Gesture** | `chrome_drag` (Chrome pointer drag), `chrome_scroll` (wheel + momentum), `chrome_tap` (touch) |
|
|
@@ -833,12 +833,9 @@ async function dispatch(action, params) {
|
|
|
833
833
|
return { closed: tab.id };
|
|
834
834
|
}
|
|
835
835
|
case "page.snapshot":
|
|
836
|
-
return
|
|
837
|
-
|
|
838
|
-
|
|
839
|
-
params.roleFilter ?? null,
|
|
840
|
-
params.nearUid ?? null,
|
|
841
|
-
]);
|
|
836
|
+
return snapshotInTab(params);
|
|
837
|
+
case "page.inspect":
|
|
838
|
+
return inspectInTab(params);
|
|
842
839
|
case "page.evaluate":
|
|
843
840
|
return evaluateInTab(params);
|
|
844
841
|
case "page.click":
|
|
@@ -1120,12 +1117,95 @@ async function evaluateInTab(params) {
|
|
|
1120
1117
|
async function withOptionalSnapshot(params, actionFn) {
|
|
1121
1118
|
const result = await actionFn(params);
|
|
1122
1119
|
if (params.includeSnapshot) {
|
|
1123
|
-
const snapshot = await
|
|
1120
|
+
const snapshot = await snapshotInTab({ ...params, foreground: false });
|
|
1124
1121
|
return { result, snapshot };
|
|
1125
1122
|
}
|
|
1126
1123
|
return result;
|
|
1127
1124
|
}
|
|
1128
1125
|
|
|
1126
|
+
// Snapshot/inspect run from a packaged MAIN-world script (snapshot_injected.js) injected via
|
|
1127
|
+
// chrome.scripting.executeScript({ files }). That file is free of eval/new Function, so it works
|
|
1128
|
+
// on strict-CSP pages, and it installs globalThis.__piChromeSnapshotPage / __piChromeInspectTarget.
|
|
1129
|
+
// It shares window.__PI_CHROME_STATE__ (same el- uid scheme) with the CDP-injected input helpers.
|
|
1130
|
+
async function snapshotInTab(params) {
|
|
1131
|
+
const tab = await getTabByParams(params);
|
|
1132
|
+
if (params.foreground) await bringToFront(tab);
|
|
1133
|
+
const args = [
|
|
1134
|
+
params.maxElements || 80,
|
|
1135
|
+
params.containingText ?? null,
|
|
1136
|
+
params.roleFilter ?? null,
|
|
1137
|
+
params.nearUid ?? null,
|
|
1138
|
+
params.mode || "auto",
|
|
1139
|
+
params.query ?? null,
|
|
1140
|
+
params.maxTextChars ?? null,
|
|
1141
|
+
];
|
|
1142
|
+
await chrome.scripting.executeScript({
|
|
1143
|
+
target: { tabId: tab.id, frameIds: [0] },
|
|
1144
|
+
world: "MAIN",
|
|
1145
|
+
files: ["snapshot_injected.js"],
|
|
1146
|
+
});
|
|
1147
|
+
const results = await chrome.scripting.executeScript({
|
|
1148
|
+
target: { tabId: tab.id, frameIds: [0] },
|
|
1149
|
+
world: "MAIN",
|
|
1150
|
+
func: async (invocationArgs) => {
|
|
1151
|
+
try {
|
|
1152
|
+
const snapshotPage = globalThis.__piChromeSnapshotPage;
|
|
1153
|
+
if (typeof snapshotPage !== "function") throw new Error("snapshot_injected.js did not install __piChromeSnapshotPage");
|
|
1154
|
+
return { ok: true, value: await snapshotPage(...invocationArgs) };
|
|
1155
|
+
} catch (error) {
|
|
1156
|
+
return { ok: false, error: error?.stack || error?.message || String(error) };
|
|
1157
|
+
}
|
|
1158
|
+
},
|
|
1159
|
+
args: [args],
|
|
1160
|
+
});
|
|
1161
|
+
const first = results?.[0];
|
|
1162
|
+
if (first?.error) {
|
|
1163
|
+
const message = typeof first.error === "string" ? first.error : (first.error.message || JSON.stringify(first.error));
|
|
1164
|
+
throw new Error(message);
|
|
1165
|
+
}
|
|
1166
|
+
const envelope = first?.result;
|
|
1167
|
+
if (envelope && typeof envelope === "object" && envelope.ok === false) {
|
|
1168
|
+
throw new Error(envelope.error || "Chrome snapshot script failed");
|
|
1169
|
+
}
|
|
1170
|
+
return envelope?.value;
|
|
1171
|
+
}
|
|
1172
|
+
|
|
1173
|
+
async function inspectInTab(params) {
|
|
1174
|
+
if (!params.uid && !params.selector) throw new Error("chrome_inspect requires uid or selector");
|
|
1175
|
+
const tab = await getTabByParams(params);
|
|
1176
|
+
if (params.foreground) await bringToFront(tab);
|
|
1177
|
+
const args = [params.uid ?? null, params.selector ?? null, params.scrollIntoView === true];
|
|
1178
|
+
await chrome.scripting.executeScript({
|
|
1179
|
+
target: { tabId: tab.id, frameIds: [0] },
|
|
1180
|
+
world: "MAIN",
|
|
1181
|
+
files: ["snapshot_injected.js"],
|
|
1182
|
+
});
|
|
1183
|
+
const results = await chrome.scripting.executeScript({
|
|
1184
|
+
target: { tabId: tab.id, frameIds: [0] },
|
|
1185
|
+
world: "MAIN",
|
|
1186
|
+
func: async (invocationArgs) => {
|
|
1187
|
+
try {
|
|
1188
|
+
const inspectTarget = globalThis.__piChromeInspectTarget;
|
|
1189
|
+
if (typeof inspectTarget !== "function") throw new Error("snapshot_injected.js did not install __piChromeInspectTarget");
|
|
1190
|
+
return { ok: true, value: await inspectTarget(...invocationArgs) };
|
|
1191
|
+
} catch (error) {
|
|
1192
|
+
return { ok: false, error: error?.stack || error?.message || String(error) };
|
|
1193
|
+
}
|
|
1194
|
+
},
|
|
1195
|
+
args: [args],
|
|
1196
|
+
});
|
|
1197
|
+
const first = results?.[0];
|
|
1198
|
+
if (first?.error) {
|
|
1199
|
+
const message = typeof first.error === "string" ? first.error : (first.error.message || JSON.stringify(first.error));
|
|
1200
|
+
throw new Error(message);
|
|
1201
|
+
}
|
|
1202
|
+
const envelope = first?.result;
|
|
1203
|
+
if (envelope && typeof envelope === "object" && envelope.ok === false) {
|
|
1204
|
+
throw new Error(envelope.error || "Chrome inspect script failed");
|
|
1205
|
+
}
|
|
1206
|
+
return envelope?.value;
|
|
1207
|
+
}
|
|
1208
|
+
|
|
1129
1209
|
// One-shot init script registry, scoped per tab. The source is registered with CDP
|
|
1130
1210
|
// Page.addScriptToEvaluateOnNewDocument, which runs it at document_start in the page's MAIN
|
|
1131
1211
|
// world and is NOT subject to page CSP (the old func:(code)=>new Function(code) path was
|
|
@@ -1642,104 +1722,6 @@ function installEarlyCapture() {
|
|
|
1642
1722
|
state.instrumentationInstalled = true;
|
|
1643
1723
|
}
|
|
1644
1724
|
|
|
1645
|
-
function snapshotPage(maxElements, containingText, roleFilter, nearUid) {
|
|
1646
|
-
installPiChromeInstrumentation();
|
|
1647
|
-
const unique = (selector) => {
|
|
1648
|
-
try { return document.querySelectorAll(selector).length === 1; } catch { return false; }
|
|
1649
|
-
};
|
|
1650
|
-
const cssEscape = (value) => (window.CSS && CSS.escape) ? CSS.escape(value) : String(value).replace(/[^a-zA-Z0-9_-]/g, "\\$&");
|
|
1651
|
-
const selectorFor = (element) => {
|
|
1652
|
-
if (element.id && unique("#" + cssEscape(element.id))) return "#" + cssEscape(element.id);
|
|
1653
|
-
const attr = ["aria-label", "name", "placeholder", "data-testid", "role"].find((name) => element.getAttribute(name));
|
|
1654
|
-
if (attr) {
|
|
1655
|
-
const candidate = element.tagName.toLowerCase() + "[" + attr + "=" + JSON.stringify(element.getAttribute(attr)) + "]";
|
|
1656
|
-
if (unique(candidate)) return candidate;
|
|
1657
|
-
}
|
|
1658
|
-
const parts = [];
|
|
1659
|
-
let current = element;
|
|
1660
|
-
while (current && current.nodeType === Node.ELEMENT_NODE && parts.length < 5) {
|
|
1661
|
-
let part = current.tagName.toLowerCase();
|
|
1662
|
-
if (current.classList.length > 0) part += "." + Array.from(current.classList).slice(0, 2).map(cssEscape).join(".");
|
|
1663
|
-
const siblings = Array.from(current.parentElement?.children ?? []).filter((sibling) => sibling.tagName === current.tagName);
|
|
1664
|
-
if (siblings.length > 1) part += ":nth-of-type(" + (siblings.indexOf(current) + 1) + ")";
|
|
1665
|
-
parts.unshift(part);
|
|
1666
|
-
const candidate = parts.join(" > ");
|
|
1667
|
-
if (unique(candidate)) return candidate;
|
|
1668
|
-
current = current.parentElement;
|
|
1669
|
-
}
|
|
1670
|
-
return parts.join(" > ");
|
|
1671
|
-
};
|
|
1672
|
-
const visible = (element) => isElementVisible(element);
|
|
1673
|
-
const labelFor = (element) => (
|
|
1674
|
-
element.getAttribute("aria-label") ||
|
|
1675
|
-
element.getAttribute("title") ||
|
|
1676
|
-
element.getAttribute("placeholder") ||
|
|
1677
|
-
element.innerText ||
|
|
1678
|
-
element.value ||
|
|
1679
|
-
element.textContent ||
|
|
1680
|
-
""
|
|
1681
|
-
).trim().replace(/\s+/g, " ").slice(0, 160);
|
|
1682
|
-
let candidates = Array.from(document.querySelectorAll('a, button, input, textarea, select, summary, [role="button"], [role="link"], [role="menuitem"], [role="tab"], [role="checkbox"], [contenteditable="true"], [tabindex]:not([tabindex="-1"])'));
|
|
1683
|
-
if (containingText) {
|
|
1684
|
-
const needle = String(containingText).toLowerCase();
|
|
1685
|
-
candidates = candidates.filter((element) => labelFor(element).toLowerCase().includes(needle));
|
|
1686
|
-
}
|
|
1687
|
-
if (roleFilter) {
|
|
1688
|
-
const wanted = String(roleFilter).toLowerCase();
|
|
1689
|
-
candidates = candidates.filter((element) => {
|
|
1690
|
-
const role = (element.getAttribute("role") || element.tagName).toLowerCase();
|
|
1691
|
-
return role === wanted;
|
|
1692
|
-
});
|
|
1693
|
-
}
|
|
1694
|
-
let near;
|
|
1695
|
-
if (nearUid) {
|
|
1696
|
-
const state = getPiChromeState();
|
|
1697
|
-
near = state.elements[nearUid];
|
|
1698
|
-
}
|
|
1699
|
-
if (near) {
|
|
1700
|
-
const nearRect = near.getBoundingClientRect();
|
|
1701
|
-
const cx = nearRect.left + nearRect.width / 2;
|
|
1702
|
-
const cy = nearRect.top + nearRect.height / 2;
|
|
1703
|
-
candidates.sort((a, b) => {
|
|
1704
|
-
const ra = a.getBoundingClientRect();
|
|
1705
|
-
const rb = b.getBoundingClientRect();
|
|
1706
|
-
const da = Math.hypot(ra.left + ra.width / 2 - cx, ra.top + ra.height / 2 - cy);
|
|
1707
|
-
const db = Math.hypot(rb.left + rb.width / 2 - cx, rb.top + rb.height / 2 - cy);
|
|
1708
|
-
return da - db;
|
|
1709
|
-
});
|
|
1710
|
-
}
|
|
1711
|
-
const elements = candidates.filter(visible).slice(0, maxElements).map((element, index) => {
|
|
1712
|
-
const rect = element.getBoundingClientRect();
|
|
1713
|
-
const style = getComputedStyle(element);
|
|
1714
|
-
const cx = rect.left + rect.width / 2;
|
|
1715
|
-
const cy = rect.top + rect.height / 2;
|
|
1716
|
-
const occluded = occluderAt(cx, cy, element);
|
|
1717
|
-
return {
|
|
1718
|
-
index,
|
|
1719
|
-
uid: rememberElement(element),
|
|
1720
|
-
tag: element.tagName.toLowerCase(),
|
|
1721
|
-
selector: selectorFor(element),
|
|
1722
|
-
label: labelFor(element),
|
|
1723
|
-
href: element.href || undefined,
|
|
1724
|
-
type: element.getAttribute("type") || undefined,
|
|
1725
|
-
role: element.getAttribute("role") || undefined,
|
|
1726
|
-
disabled: Boolean(element.disabled || element.getAttribute("aria-disabled") === "true"),
|
|
1727
|
-
inert: Boolean(element.closest?.("[inert]")),
|
|
1728
|
-
pointerEvents: style.pointerEvents,
|
|
1729
|
-
occluded: occluded || undefined,
|
|
1730
|
-
rect: { x: Math.round(rect.x), y: Math.round(rect.y), width: Math.round(rect.width), height: Math.round(rect.height) },
|
|
1731
|
-
};
|
|
1732
|
-
});
|
|
1733
|
-
return {
|
|
1734
|
-
title: document.title,
|
|
1735
|
-
url: location.href,
|
|
1736
|
-
viewport: { width: innerWidth, height: innerHeight, scrollX, scrollY },
|
|
1737
|
-
text: document.body ? document.body.innerText.replace(/\s+\n/g, "\n").trim().slice(0, 30000) : "",
|
|
1738
|
-
elements,
|
|
1739
|
-
filter: { containingText: containingText || undefined, roleFilter: roleFilter || undefined, nearUid: nearUid || undefined },
|
|
1740
|
-
};
|
|
1741
|
-
}
|
|
1742
|
-
|
|
1743
1725
|
function probePage() {
|
|
1744
1726
|
// Sanity probe used by /chrome-doctor. Returns evidence that MAIN-world execution works.
|
|
1745
1727
|
return {
|
|
@@ -0,0 +1,677 @@
|
|
|
1
|
+
// Static MAIN-world snapshot implementation injected by the MV3 service worker.
|
|
2
|
+
// Keep this file free of eval/new Function so `chrome_snapshot` works on strict-CSP pages.
|
|
3
|
+
(() => {
|
|
4
|
+
function getPiChromeState() {
|
|
5
|
+
const state = window.__PI_CHROME_STATE__ || {
|
|
6
|
+
nextElementUid: 1,
|
|
7
|
+
elements: {},
|
|
8
|
+
console: [],
|
|
9
|
+
network: [],
|
|
10
|
+
nextRequestId: 1,
|
|
11
|
+
instrumentationInstalled: false,
|
|
12
|
+
lastSnapshotDigest: null,
|
|
13
|
+
};
|
|
14
|
+
window.__PI_CHROME_STATE__ = state;
|
|
15
|
+
return state;
|
|
16
|
+
}
|
|
17
|
+
|
|
18
|
+
function rememberElement(element) {
|
|
19
|
+
const state = getPiChromeState();
|
|
20
|
+
if (!element.__piChromeUid) element.__piChromeUid = "el-" + state.nextElementUid++;
|
|
21
|
+
state.elements[element.__piChromeUid] = element;
|
|
22
|
+
return element.__piChromeUid;
|
|
23
|
+
}
|
|
24
|
+
|
|
25
|
+
function isElementVisible(element) {
|
|
26
|
+
if (!element || !element.getBoundingClientRect) return false;
|
|
27
|
+
const style = getComputedStyle(element);
|
|
28
|
+
if (style.visibility === "hidden" || style.display === "none") return false;
|
|
29
|
+
const rect = element.getBoundingClientRect();
|
|
30
|
+
if (rect.width === 0 || rect.height === 0) return false;
|
|
31
|
+
if (rect.bottom < 0 || rect.right < 0) return false;
|
|
32
|
+
if (rect.top > innerHeight || rect.left > innerWidth) return false;
|
|
33
|
+
return true;
|
|
34
|
+
}
|
|
35
|
+
|
|
36
|
+
function occluderAt(x, y, expected) {
|
|
37
|
+
const top = document.elementFromPoint(x, y);
|
|
38
|
+
if (!top || top === expected) return null;
|
|
39
|
+
if (expected && expected.contains(top)) return null;
|
|
40
|
+
if (top.contains(expected)) return null;
|
|
41
|
+
return {
|
|
42
|
+
tag: top.tagName.toLowerCase(),
|
|
43
|
+
id: top.id || undefined,
|
|
44
|
+
className: typeof top.className === "string" ? top.className : undefined,
|
|
45
|
+
};
|
|
46
|
+
}
|
|
47
|
+
|
|
48
|
+
function textOf(element, max) {
|
|
49
|
+
return (element?.innerText || element?.textContent || "").replace(/\s+/g, " ").trim().slice(0, max || 500);
|
|
50
|
+
}
|
|
51
|
+
|
|
52
|
+
function accessibleLabel(element) {
|
|
53
|
+
if (!element) return "";
|
|
54
|
+
const labelledBy = element.getAttribute("aria-labelledby");
|
|
55
|
+
if (labelledBy) {
|
|
56
|
+
const text = labelledBy.split(/\s+/).map((id) => document.getElementById(id)?.innerText || "").join(" ").trim();
|
|
57
|
+
if (text) return text;
|
|
58
|
+
}
|
|
59
|
+
const id = element.id;
|
|
60
|
+
if (id) {
|
|
61
|
+
try {
|
|
62
|
+
const label = document.querySelector(`label[for="${cssEscape(id)}"]`);
|
|
63
|
+
if (label?.innerText) return label.innerText;
|
|
64
|
+
} catch {}
|
|
65
|
+
}
|
|
66
|
+
const wrappingLabel = element.closest?.("label");
|
|
67
|
+
return (
|
|
68
|
+
element.getAttribute("aria-label") ||
|
|
69
|
+
element.getAttribute("title") ||
|
|
70
|
+
element.getAttribute("placeholder") ||
|
|
71
|
+
wrappingLabel?.innerText ||
|
|
72
|
+
element.innerText ||
|
|
73
|
+
element.textContent ||
|
|
74
|
+
""
|
|
75
|
+
).trim().replace(/\s+/g, " ").slice(0, 180);
|
|
76
|
+
}
|
|
77
|
+
|
|
78
|
+
function cssEscape(value) {
|
|
79
|
+
return (window.CSS && CSS.escape) ? CSS.escape(value) : String(value).replace(/[^a-zA-Z0-9_-]/g, "\\$&");
|
|
80
|
+
}
|
|
81
|
+
|
|
82
|
+
function roleOf(element) {
|
|
83
|
+
const explicit = element.getAttribute("role");
|
|
84
|
+
if (explicit) return explicit.toLowerCase();
|
|
85
|
+
const tag = element.tagName.toLowerCase();
|
|
86
|
+
const type = (element.getAttribute("type") || "").toLowerCase();
|
|
87
|
+
if (tag === "a" && element.href) return "link";
|
|
88
|
+
if (tag === "button" || type === "button" || type === "submit" || type === "reset") return "button";
|
|
89
|
+
if (tag === "textarea") return "textbox";
|
|
90
|
+
if (tag === "select") return "combobox";
|
|
91
|
+
if (tag === "input") {
|
|
92
|
+
if (["checkbox", "radio", "range", "search", "email", "password", "tel", "url", "number"].includes(type)) return type === "checkbox" || type === "radio" || type === "range" ? type : "textbox";
|
|
93
|
+
return "textbox";
|
|
94
|
+
}
|
|
95
|
+
if (element.isContentEditable) return "textbox";
|
|
96
|
+
if (tag.match(/^h[1-6]$/)) return "heading";
|
|
97
|
+
return tag;
|
|
98
|
+
}
|
|
99
|
+
|
|
100
|
+
function isSensitiveField(element) {
|
|
101
|
+
if (!element) return false;
|
|
102
|
+
const tag = element.tagName?.toLowerCase?.() || "";
|
|
103
|
+
if (!/^(input|textarea|select)$/.test(tag) && !element.isContentEditable) return false;
|
|
104
|
+
const type = (element.getAttribute("type") || "").toLowerCase();
|
|
105
|
+
if (["password"].includes(type)) return true;
|
|
106
|
+
const haystack = [
|
|
107
|
+
type,
|
|
108
|
+
element.getAttribute("name"),
|
|
109
|
+
element.id,
|
|
110
|
+
element.getAttribute("autocomplete"),
|
|
111
|
+
element.getAttribute("aria-label"),
|
|
112
|
+
element.getAttribute("placeholder"),
|
|
113
|
+
element.getAttribute("data-testid"),
|
|
114
|
+
].filter(Boolean).join(" ").toLowerCase();
|
|
115
|
+
return /password|passwd|\bpwd\b|secret|token|bearer|api[-_ ]?key|access[-_ ]?key|auth[-_ ]?code|one[-_ ]?time|otp|2fa|mfa|verification[-_ ]?code|recovery[-_ ]?code|credit[-_ ]?card|card[-_ ]?number|cc-number|cc-csc|cvc|cvv|security[-_ ]?code|ssn|social[-_ ]?security/.test(haystack);
|
|
116
|
+
}
|
|
117
|
+
|
|
118
|
+
function installPiChromeInstrumentation() {
|
|
119
|
+
const state = getPiChromeState();
|
|
120
|
+
if (state.instrumentationInstalled) return;
|
|
121
|
+
state.instrumentationInstalled = true;
|
|
122
|
+
const pushConsole = (level, args) => {
|
|
123
|
+
state.console.push({
|
|
124
|
+
id: state.console.length + 1,
|
|
125
|
+
level,
|
|
126
|
+
timestamp: Date.now(),
|
|
127
|
+
url: location.href,
|
|
128
|
+
args: Array.from(args).map((arg) => {
|
|
129
|
+
try {
|
|
130
|
+
if (typeof arg === "string") return arg;
|
|
131
|
+
if (arg instanceof Error) return { name: arg.name, message: arg.message, stack: arg.stack };
|
|
132
|
+
return JSON.parse(JSON.stringify(arg));
|
|
133
|
+
} catch {
|
|
134
|
+
return String(arg);
|
|
135
|
+
}
|
|
136
|
+
}),
|
|
137
|
+
});
|
|
138
|
+
if (state.console.length > 500) state.console.splice(0, state.console.length - 500);
|
|
139
|
+
};
|
|
140
|
+
for (const level of ["debug", "log", "info", "warn", "error"]) {
|
|
141
|
+
const original = console[level];
|
|
142
|
+
if (typeof original !== "function" || original.__piChromeWrapped) continue;
|
|
143
|
+
const wrapped = function(...args) {
|
|
144
|
+
pushConsole(level, args);
|
|
145
|
+
return original.apply(this, args);
|
|
146
|
+
};
|
|
147
|
+
wrapped.__piChromeWrapped = true;
|
|
148
|
+
console[level] = wrapped;
|
|
149
|
+
}
|
|
150
|
+
window.addEventListener("error", (event) => pushConsole("pageerror", [event.message, event.filename + ":" + event.lineno + ":" + event.colno]));
|
|
151
|
+
window.addEventListener("unhandledrejection", (event) => pushConsole("unhandledrejection", [event.reason]));
|
|
152
|
+
|
|
153
|
+
const record = (entry) => {
|
|
154
|
+
state.network.push(entry);
|
|
155
|
+
if (state.network.length > 1000) state.network.splice(0, state.network.length - 1000);
|
|
156
|
+
return entry;
|
|
157
|
+
};
|
|
158
|
+
if (window.fetch && !window.fetch.__piChromeWrapped) {
|
|
159
|
+
const originalFetch = window.fetch.bind(window);
|
|
160
|
+
const wrappedFetch = async (...args) => {
|
|
161
|
+
const id = "req-" + state.nextRequestId++;
|
|
162
|
+
const startedAt = Date.now();
|
|
163
|
+
const input = args[0];
|
|
164
|
+
const init = args[1] || {};
|
|
165
|
+
const url = typeof input === "string" ? input : input?.url;
|
|
166
|
+
const method = (init.method || input?.method || "GET").toUpperCase();
|
|
167
|
+
const entry = record({ id, type: "fetch", method, url: String(url || ""), startedAt, pageUrl: location.href, status: "pending" });
|
|
168
|
+
try {
|
|
169
|
+
const response = await originalFetch(...args);
|
|
170
|
+
entry.status = response.status;
|
|
171
|
+
entry.statusText = response.statusText;
|
|
172
|
+
entry.ok = response.ok;
|
|
173
|
+
entry.responseUrl = response.url;
|
|
174
|
+
entry.durationMs = Date.now() - startedAt;
|
|
175
|
+
entry.responseHeaders = Array.from(response.headers.entries());
|
|
176
|
+
entry.responseBodyOmitted = "response body capture is disabled by default";
|
|
177
|
+
return response;
|
|
178
|
+
} catch (error) {
|
|
179
|
+
entry.error = error?.message || String(error);
|
|
180
|
+
entry.durationMs = Date.now() - startedAt;
|
|
181
|
+
throw error;
|
|
182
|
+
}
|
|
183
|
+
};
|
|
184
|
+
wrappedFetch.__piChromeWrapped = true;
|
|
185
|
+
window.fetch = wrappedFetch;
|
|
186
|
+
}
|
|
187
|
+
if (window.XMLHttpRequest && !XMLHttpRequest.prototype.open.__piChromeWrapped) {
|
|
188
|
+
const originalOpen = XMLHttpRequest.prototype.open;
|
|
189
|
+
const originalSend = XMLHttpRequest.prototype.send;
|
|
190
|
+
XMLHttpRequest.prototype.open = function(method, url, ...rest) {
|
|
191
|
+
this.__piChromeRequest = { method: String(method || "GET").toUpperCase(), url: String(url || "") };
|
|
192
|
+
return originalOpen.call(this, method, url, ...rest);
|
|
193
|
+
};
|
|
194
|
+
XMLHttpRequest.prototype.open.__piChromeWrapped = true;
|
|
195
|
+
XMLHttpRequest.prototype.send = function(body) {
|
|
196
|
+
const id = "req-" + state.nextRequestId++;
|
|
197
|
+
const startedAt = Date.now();
|
|
198
|
+
const info = this.__piChromeRequest || {};
|
|
199
|
+
const entry = record({ id, type: "xhr", method: info.method || "GET", url: info.url || "", startedAt, pageUrl: location.href, status: "pending" });
|
|
200
|
+
this.addEventListener("loadend", () => {
|
|
201
|
+
entry.status = this.status;
|
|
202
|
+
entry.statusText = this.statusText;
|
|
203
|
+
entry.responseUrl = this.responseURL;
|
|
204
|
+
entry.durationMs = Date.now() - startedAt;
|
|
205
|
+
try { entry.responseHeadersText = this.getAllResponseHeaders(); } catch {}
|
|
206
|
+
entry.responseBodyOmitted = "response body capture is disabled by default";
|
|
207
|
+
});
|
|
208
|
+
this.addEventListener("error", () => { entry.error = "XMLHttpRequest error"; entry.durationMs = Date.now() - startedAt; });
|
|
209
|
+
return originalSend.call(this, body);
|
|
210
|
+
};
|
|
211
|
+
}
|
|
212
|
+
}
|
|
213
|
+
|
|
214
|
+
function hashString(text) {
|
|
215
|
+
let h = 0;
|
|
216
|
+
for (let i = 0; i < text.length; i++) h = (h * 31 + text.charCodeAt(i)) | 0;
|
|
217
|
+
return h;
|
|
218
|
+
}
|
|
219
|
+
|
|
220
|
+
function selectorFor(element) {
|
|
221
|
+
const unique = (selector) => {
|
|
222
|
+
try { return document.querySelectorAll(selector).length === 1; } catch { return false; }
|
|
223
|
+
};
|
|
224
|
+
if (element.id && unique("#" + cssEscape(element.id))) return "#" + cssEscape(element.id);
|
|
225
|
+
const attr = ["aria-label", "name", "placeholder", "data-testid", "role"].find((name) => element.getAttribute(name));
|
|
226
|
+
if (attr) {
|
|
227
|
+
const candidate = element.tagName.toLowerCase() + "[" + attr + "=" + JSON.stringify(element.getAttribute(attr)) + "]";
|
|
228
|
+
if (unique(candidate)) return candidate;
|
|
229
|
+
}
|
|
230
|
+
const parts = [];
|
|
231
|
+
let current = element;
|
|
232
|
+
while (current && current.nodeType === Node.ELEMENT_NODE && parts.length < 5) {
|
|
233
|
+
let part = current.tagName.toLowerCase();
|
|
234
|
+
if (current.classList.length > 0) part += "." + Array.from(current.classList).slice(0, 2).map(cssEscape).join(".");
|
|
235
|
+
const siblings = Array.from(current.parentElement?.children ?? []).filter((sibling) => sibling.tagName === current.tagName);
|
|
236
|
+
if (siblings.length > 1) part += ":nth-of-type(" + (siblings.indexOf(current) + 1) + ")";
|
|
237
|
+
parts.unshift(part);
|
|
238
|
+
const candidate = parts.join(" > ");
|
|
239
|
+
if (unique(candidate)) return candidate;
|
|
240
|
+
current = current.parentElement;
|
|
241
|
+
}
|
|
242
|
+
return parts.join(" > ");
|
|
243
|
+
}
|
|
244
|
+
|
|
245
|
+
function directHeadingText(element) {
|
|
246
|
+
const labelledBy = element.getAttribute?.("aria-labelledby");
|
|
247
|
+
if (labelledBy) {
|
|
248
|
+
const text = labelledBy.split(/\s+/).map((id) => document.getElementById(id)?.innerText || "").join(" ").replace(/\s+/g, " ").trim();
|
|
249
|
+
if (text) return text.slice(0, 180);
|
|
250
|
+
}
|
|
251
|
+
const aria = element.getAttribute?.("aria-label");
|
|
252
|
+
if (aria) return aria.trim().slice(0, 180);
|
|
253
|
+
const heading = Array.from(element.querySelectorAll?.("h1,h2,h3,h4,[role='heading']") || []).find(isElementVisible);
|
|
254
|
+
if (heading) return textOf(heading, 180);
|
|
255
|
+
return "";
|
|
256
|
+
}
|
|
257
|
+
|
|
258
|
+
function meaningfulContainerFor(element) {
|
|
259
|
+
let current = element.parentElement;
|
|
260
|
+
let fallback = current;
|
|
261
|
+
let depth = 0;
|
|
262
|
+
while (current && current !== document.body && depth++ < 8) {
|
|
263
|
+
if (!isElementVisible(current)) { current = current.parentElement; continue; }
|
|
264
|
+
const tag = current.tagName.toLowerCase();
|
|
265
|
+
const role = (current.getAttribute("role") || "").toLowerCase();
|
|
266
|
+
const cls = typeof current.className === "string" ? current.className : "";
|
|
267
|
+
const id = current.id || "";
|
|
268
|
+
const named = Boolean(current.getAttribute("aria-label") || current.getAttribute("aria-labelledby") || directHeadingText(current));
|
|
269
|
+
const semantic = /^(form|dialog|section|article|nav|header|main|aside|footer|li|tr|td|fieldset)$/.test(tag) ||
|
|
270
|
+
/^(dialog|alertdialog|region|group|listitem|row|cell|tabpanel|menu|toolbar|navigation|main|banner|contentinfo|complementary)$/.test(role);
|
|
271
|
+
const classHint = /card|panel|pane|modal|dialog|section|content|container|toolbar|menu|list|item|row|cell|header|footer|sidebar|drawer|popover|dropdown/i.test(`${id} ${cls}`);
|
|
272
|
+
const rect = current.getBoundingClientRect();
|
|
273
|
+
const childActions = current.querySelectorAll?.('a, button, input, textarea, select, [role="button"], [role="link"], [tabindex]:not([tabindex="-1"])').length || 0;
|
|
274
|
+
if ((semantic || classHint || named) && rect.width > 20 && rect.height > 20 && childActions <= 80) return current;
|
|
275
|
+
if (!fallback && rect.width > 20 && rect.height > 20) fallback = current;
|
|
276
|
+
current = current.parentElement;
|
|
277
|
+
}
|
|
278
|
+
return fallback || document.body;
|
|
279
|
+
}
|
|
280
|
+
|
|
281
|
+
function contextForElement(element) {
|
|
282
|
+
const container = meaningfulContainerFor(element);
|
|
283
|
+
if (!container || container === document.body || container === element) return undefined;
|
|
284
|
+
return {
|
|
285
|
+
uid: rememberElement(container),
|
|
286
|
+
tag: container.tagName.toLowerCase(),
|
|
287
|
+
role: roleOf(container),
|
|
288
|
+
label: directHeadingText(container) || accessibleLabel(container) || textOf(container, 140),
|
|
289
|
+
rect: rectSummary(container),
|
|
290
|
+
};
|
|
291
|
+
}
|
|
292
|
+
|
|
293
|
+
function summarizeElement(element, index) {
|
|
294
|
+
const rect = element.getBoundingClientRect();
|
|
295
|
+
const style = getComputedStyle(element);
|
|
296
|
+
const cx = rect.left + rect.width / 2;
|
|
297
|
+
const cy = rect.top + rect.height / 2;
|
|
298
|
+
const occluded = occluderAt(cx, cy, element);
|
|
299
|
+
const role = roleOf(element);
|
|
300
|
+
const disabled = Boolean(element.disabled || element.getAttribute("aria-disabled") === "true");
|
|
301
|
+
const rawValue = "value" in element && typeof element.value === "string" ? element.value : undefined;
|
|
302
|
+
const sensitive = isSensitiveField(element);
|
|
303
|
+
const value = rawValue && !sensitive ? rawValue.slice(0, 120) : undefined;
|
|
304
|
+
const checked = "checked" in element ? Boolean(element.checked) : undefined;
|
|
305
|
+
return {
|
|
306
|
+
index,
|
|
307
|
+
uid: rememberElement(element),
|
|
308
|
+
tag: element.tagName.toLowerCase(),
|
|
309
|
+
role,
|
|
310
|
+
selector: selectorFor(element),
|
|
311
|
+
label: accessibleLabel(element),
|
|
312
|
+
href: element.href || undefined,
|
|
313
|
+
type: element.getAttribute("type") || undefined,
|
|
314
|
+
value: value || undefined,
|
|
315
|
+
hasValue: rawValue ? rawValue.length > 0 : undefined,
|
|
316
|
+
valueLength: rawValue && sensitive ? rawValue.length : undefined,
|
|
317
|
+
valueRedacted: sensitive && rawValue ? true : undefined,
|
|
318
|
+
checked,
|
|
319
|
+
disabled,
|
|
320
|
+
inert: Boolean(element.closest?.("[inert]")),
|
|
321
|
+
pointerEvents: style.pointerEvents,
|
|
322
|
+
occluded: occluded || undefined,
|
|
323
|
+
context: contextForElement(element),
|
|
324
|
+
rect: { x: Math.round(rect.x), y: Math.round(rect.y), width: Math.round(rect.width), height: Math.round(rect.height) },
|
|
325
|
+
};
|
|
326
|
+
}
|
|
327
|
+
|
|
328
|
+
function isInViewport(element) {
|
|
329
|
+
const rect = element.getBoundingClientRect();
|
|
330
|
+
return rect.bottom >= 0 && rect.right >= 0 && rect.top <= innerHeight && rect.left <= innerWidth;
|
|
331
|
+
}
|
|
332
|
+
|
|
333
|
+
function formSummaries() {
|
|
334
|
+
const fields = Array.from(document.querySelectorAll('input, textarea, select, [contenteditable="true"]'))
|
|
335
|
+
.filter(isElementVisible)
|
|
336
|
+
.slice(0, 80)
|
|
337
|
+
.map((element, index) => ({
|
|
338
|
+
...summarizeElement(element, index),
|
|
339
|
+
required: Boolean(element.required || element.getAttribute("aria-required") === "true"),
|
|
340
|
+
invalid: Boolean(element.matches?.(":invalid") || element.getAttribute("aria-invalid") === "true"),
|
|
341
|
+
autocomplete: element.getAttribute("autocomplete") || undefined,
|
|
342
|
+
}));
|
|
343
|
+
const submits = Array.from(document.querySelectorAll('button, input[type="submit"], [role="button"]'))
|
|
344
|
+
.filter(isElementVisible)
|
|
345
|
+
.filter((element) => /submit|save|continue|next|send|sign in|log in|create|update|done/i.test(accessibleLabel(element) + " " + (element.getAttribute("type") || "")))
|
|
346
|
+
.slice(0, 30)
|
|
347
|
+
.map((element, index) => summarizeElement(element, index));
|
|
348
|
+
return { fields, submits };
|
|
349
|
+
}
|
|
350
|
+
|
|
351
|
+
function pageMap() {
|
|
352
|
+
const landmarkSelectors = [
|
|
353
|
+
["header", 'header, [role="banner"]'],
|
|
354
|
+
["nav", 'nav, [role="navigation"]'],
|
|
355
|
+
["main", 'main, [role="main"]'],
|
|
356
|
+
["aside", 'aside, [role="complementary"]'],
|
|
357
|
+
["footer", 'footer, [role="contentinfo"]'],
|
|
358
|
+
["dialog", 'dialog, [role="dialog"], [aria-modal="true"]'],
|
|
359
|
+
["form", "form"],
|
|
360
|
+
];
|
|
361
|
+
const regions = [];
|
|
362
|
+
for (const [kind, selector] of landmarkSelectors) {
|
|
363
|
+
for (const element of Array.from(document.querySelectorAll(selector)).filter(isElementVisible).slice(0, 12)) {
|
|
364
|
+
const headings = Array.from(element.querySelectorAll("h1,h2,h3,[role='heading']")).filter(isElementVisible).slice(0, 6).map((h) => textOf(h, 120));
|
|
365
|
+
const actions = Array.from(element.querySelectorAll('a, button, input, textarea, select, [role="button"], [role="link"], [tabindex]:not([tabindex="-1"])')).filter(isElementVisible).slice(0, 8).map((a) => {
|
|
366
|
+
const summary = summarizeElement(a, 0);
|
|
367
|
+
return { uid: summary.uid, role: summary.role, label: summary.label || summary.selector, disabled: summary.disabled || undefined };
|
|
368
|
+
});
|
|
369
|
+
regions.push({ kind, uid: rememberElement(element), label: accessibleLabel(element) || headings[0] || textOf(element, 100), headings, actions });
|
|
370
|
+
}
|
|
371
|
+
}
|
|
372
|
+
const headings = Array.from(document.querySelectorAll("h1,h2,h3,[role='heading']")).filter(isElementVisible).slice(0, 30).map((element) => ({
|
|
373
|
+
uid: rememberElement(element),
|
|
374
|
+
level: Number(element.tagName?.slice(1)) || Number(element.getAttribute("aria-level")) || undefined,
|
|
375
|
+
text: textOf(element, 180),
|
|
376
|
+
}));
|
|
377
|
+
return { regions, headings };
|
|
378
|
+
}
|
|
379
|
+
|
|
380
|
+
function layoutSections(elements, forms) {
|
|
381
|
+
const byUid = new Map();
|
|
382
|
+
const addToSection = (summary, kind) => {
|
|
383
|
+
const source = getPiChromeState().elements[summary.uid];
|
|
384
|
+
const container = source ? meaningfulContainerFor(source) : null;
|
|
385
|
+
if (!container || container === document.body) return;
|
|
386
|
+
const uid = rememberElement(container);
|
|
387
|
+
let section = byUid.get(uid);
|
|
388
|
+
if (!section) {
|
|
389
|
+
const rect = rectSummary(container);
|
|
390
|
+
section = {
|
|
391
|
+
uid,
|
|
392
|
+
tag: container.tagName.toLowerCase(),
|
|
393
|
+
role: roleOf(container),
|
|
394
|
+
label: directHeadingText(container) || accessibleLabel(container) || textOf(container, 160),
|
|
395
|
+
text: textOf(container, 260),
|
|
396
|
+
rect,
|
|
397
|
+
actions: [],
|
|
398
|
+
fields: [],
|
|
399
|
+
};
|
|
400
|
+
byUid.set(uid, section);
|
|
401
|
+
}
|
|
402
|
+
const item = { uid: summary.uid, role: summary.role, label: summary.label || summary.selector, disabled: summary.disabled || undefined };
|
|
403
|
+
if (kind === "field") section.fields.push(item);
|
|
404
|
+
else section.actions.push(item);
|
|
405
|
+
};
|
|
406
|
+
for (const el of (elements || []).slice(0, 80)) addToSection(el, ["textbox", "checkbox", "radio", "combobox"].includes(el.role) ? "field" : "action");
|
|
407
|
+
for (const field of (forms?.fields || []).slice(0, 80)) addToSection(field, "field");
|
|
408
|
+
const sections = Array.from(byUid.values())
|
|
409
|
+
.filter((section) => section.actions.length || section.fields.length)
|
|
410
|
+
.sort((a, b) => a.rect.y - b.rect.y || a.rect.x - b.rect.x)
|
|
411
|
+
.slice(0, 18);
|
|
412
|
+
for (const section of sections) {
|
|
413
|
+
section.actions = section.actions.slice(0, 10);
|
|
414
|
+
section.fields = section.fields.slice(0, 10);
|
|
415
|
+
}
|
|
416
|
+
return sections;
|
|
417
|
+
}
|
|
418
|
+
|
|
419
|
+
function tokenScore(haystack, query) {
|
|
420
|
+
if (!query) return 0;
|
|
421
|
+
const hay = String(haystack || "").toLowerCase();
|
|
422
|
+
const tokens = String(query).toLowerCase().split(/\W+/).filter(Boolean);
|
|
423
|
+
if (!tokens.length) return 0;
|
|
424
|
+
let score = 0;
|
|
425
|
+
for (const token of tokens) {
|
|
426
|
+
if (hay.includes(token)) score += token.length <= 2 ? 1 : 3;
|
|
427
|
+
}
|
|
428
|
+
if (hay.includes(String(query).toLowerCase())) score += 8;
|
|
429
|
+
return score;
|
|
430
|
+
}
|
|
431
|
+
|
|
432
|
+
function queryMatches(query, elements, map) {
|
|
433
|
+
if (!query) return [];
|
|
434
|
+
const candidates = [];
|
|
435
|
+
for (const element of elements) {
|
|
436
|
+
const hay = [element.role, element.label, element.selector, element.type, element.href].filter(Boolean).join(" ");
|
|
437
|
+
const score = tokenScore(hay, query);
|
|
438
|
+
if (score > 0) candidates.push({ score, kind: "element", ...element });
|
|
439
|
+
}
|
|
440
|
+
const textNodes = [];
|
|
441
|
+
for (const block of Array.from(document.querySelectorAll("h1,h2,h3,h4,p,li,td,th,label,summary,[role='alert']")).filter(isElementVisible).slice(0, 300)) {
|
|
442
|
+
const text = textOf(block, 300);
|
|
443
|
+
const score = tokenScore(text, query);
|
|
444
|
+
if (score > 0) textNodes.push({ score, kind: "text", uid: rememberElement(block), tag: block.tagName.toLowerCase(), role: roleOf(block), text, rect: rectSummary(block) });
|
|
445
|
+
}
|
|
446
|
+
for (const region of map.regions || []) {
|
|
447
|
+
const score = tokenScore([region.kind, region.label, ...(region.headings || [])].join(" "), query);
|
|
448
|
+
if (score > 0) candidates.push({ score, kind: "region", ...region });
|
|
449
|
+
}
|
|
450
|
+
return candidates.concat(textNodes).sort((a, b) => b.score - a.score).slice(0, 20);
|
|
451
|
+
}
|
|
452
|
+
|
|
453
|
+
function rectSummary(element) {
|
|
454
|
+
const rect = element.getBoundingClientRect();
|
|
455
|
+
return { x: Math.round(rect.x), y: Math.round(rect.y), width: Math.round(rect.width), height: Math.round(rect.height) };
|
|
456
|
+
}
|
|
457
|
+
|
|
458
|
+
function activeElementSummary() {
|
|
459
|
+
const el = document.activeElement;
|
|
460
|
+
if (!el || el === document.body || el === document.documentElement) return null;
|
|
461
|
+
return summarizeElement(el, 0);
|
|
462
|
+
}
|
|
463
|
+
|
|
464
|
+
function modalSummary() {
|
|
465
|
+
const selectors = 'dialog[open], [role="dialog"], [aria-modal="true"], [role="alertdialog"]';
|
|
466
|
+
const modal = Array.from(document.querySelectorAll(selectors)).find(isElementVisible);
|
|
467
|
+
if (!modal) return null;
|
|
468
|
+
return { uid: rememberElement(modal), tag: modal.tagName.toLowerCase(), role: roleOf(modal), label: accessibleLabel(modal) || textOf(modal, 180), rect: rectSummary(modal) };
|
|
469
|
+
}
|
|
470
|
+
|
|
471
|
+
function digestFor(snapshot) {
|
|
472
|
+
return {
|
|
473
|
+
url: snapshot.url,
|
|
474
|
+
title: snapshot.title,
|
|
475
|
+
textHash: hashString(snapshot.text || ""),
|
|
476
|
+
focusedUid: snapshot.focused?.uid || null,
|
|
477
|
+
modalUid: snapshot.modal?.uid || null,
|
|
478
|
+
labels: (snapshot.elements || []).slice(0, 50).map((el) => ({ uid: el.uid, role: el.role, label: el.label, disabled: el.disabled, value: el.value, checked: el.checked })),
|
|
479
|
+
};
|
|
480
|
+
}
|
|
481
|
+
|
|
482
|
+
function diffSnapshot(previous, current) {
|
|
483
|
+
if (!previous) return { firstSnapshot: true };
|
|
484
|
+
const changes = [];
|
|
485
|
+
if (previous.url !== current.url) changes.push({ kind: "url", before: previous.url, after: current.url });
|
|
486
|
+
if (previous.title !== current.title) changes.push({ kind: "title", before: previous.title, after: current.title });
|
|
487
|
+
if (previous.textHash !== current.textHash) changes.push({ kind: "textChanged" });
|
|
488
|
+
if (previous.focusedUid !== current.focusedUid) changes.push({ kind: "focus", before: previous.focusedUid, after: current.focusedUid });
|
|
489
|
+
if (previous.modalUid !== current.modalUid) changes.push({ kind: "modal", before: previous.modalUid, after: current.modalUid });
|
|
490
|
+
const prevByUid = new Map((previous.labels || []).map((x) => [x.uid, x]));
|
|
491
|
+
const curByUid = new Map((current.labels || []).map((x) => [x.uid, x]));
|
|
492
|
+
const added = [];
|
|
493
|
+
const removed = [];
|
|
494
|
+
const updated = [];
|
|
495
|
+
for (const cur of current.labels || []) {
|
|
496
|
+
const prev = prevByUid.get(cur.uid);
|
|
497
|
+
if (!prev) added.push(cur);
|
|
498
|
+
else if (prev.label !== cur.label || prev.disabled !== cur.disabled || prev.value !== cur.value || prev.checked !== cur.checked) updated.push({ uid: cur.uid, before: prev, after: cur });
|
|
499
|
+
}
|
|
500
|
+
for (const prev of previous.labels || []) {
|
|
501
|
+
if (!curByUid.has(prev.uid)) removed.push(prev);
|
|
502
|
+
}
|
|
503
|
+
return { changes, added: added.slice(0, 12), removed: removed.slice(0, 12), updated: updated.slice(0, 12) };
|
|
504
|
+
}
|
|
505
|
+
|
|
506
|
+
function visibleTextSnippets(maxChars) {
|
|
507
|
+
const snippets = [];
|
|
508
|
+
const blocks = Array.from(document.querySelectorAll("h1,h2,h3,h4,p,li,td,th,label,summary,[role='alert']")).filter(isElementVisible);
|
|
509
|
+
let used = 0;
|
|
510
|
+
for (const block of blocks) {
|
|
511
|
+
if (!isInViewport(block) && snippets.length > 12) continue;
|
|
512
|
+
const text = textOf(block, 500);
|
|
513
|
+
if (!text || snippets.some((s) => s.text === text)) continue;
|
|
514
|
+
const next = { uid: rememberElement(block), tag: block.tagName.toLowerCase(), text, rect: rectSummary(block) };
|
|
515
|
+
snippets.push(next);
|
|
516
|
+
used += text.length;
|
|
517
|
+
if (used >= maxChars || snippets.length >= 40) break;
|
|
518
|
+
}
|
|
519
|
+
return snippets;
|
|
520
|
+
}
|
|
521
|
+
|
|
522
|
+
function snapshotPage(maxElements, containingText, roleFilter, nearUid, mode, query, maxTextChars) {
|
|
523
|
+
installPiChromeInstrumentation();
|
|
524
|
+
mode = ["auto", "interactive", "forms", "pageMap", "text", "changes", "full"].includes(mode) ? mode : "auto";
|
|
525
|
+
const fullTextLimit = Number(maxTextChars || (mode === "full" ? 30000 : mode === "text" ? 18000 : 6000));
|
|
526
|
+
let candidates = Array.from(document.querySelectorAll('a, button, input, textarea, select, summary, [role="button"], [role="link"], [role="menuitem"], [role="tab"], [role="checkbox"], [contenteditable="true"], [tabindex]:not([tabindex="-1"])'));
|
|
527
|
+
if (containingText) {
|
|
528
|
+
const needle = String(containingText).toLowerCase();
|
|
529
|
+
candidates = candidates.filter((element) => accessibleLabel(element).toLowerCase().includes(needle));
|
|
530
|
+
}
|
|
531
|
+
if (roleFilter) {
|
|
532
|
+
const wanted = String(roleFilter).toLowerCase();
|
|
533
|
+
candidates = candidates.filter((element) => roleOf(element) === wanted || element.tagName.toLowerCase() === wanted);
|
|
534
|
+
}
|
|
535
|
+
let near;
|
|
536
|
+
if (nearUid) near = getPiChromeState().elements[nearUid];
|
|
537
|
+
if (near) {
|
|
538
|
+
const nearRect = near.getBoundingClientRect();
|
|
539
|
+
const cx = nearRect.left + nearRect.width / 2;
|
|
540
|
+
const cy = nearRect.top + nearRect.height / 2;
|
|
541
|
+
candidates.sort((a, b) => {
|
|
542
|
+
const ra = a.getBoundingClientRect();
|
|
543
|
+
const rb = b.getBoundingClientRect();
|
|
544
|
+
const da = Math.hypot(ra.left + ra.width / 2 - cx, ra.top + ra.height / 2 - cy);
|
|
545
|
+
const db = Math.hypot(rb.left + rb.width / 2 - cx, rb.top + rb.height / 2 - cy);
|
|
546
|
+
return da - db;
|
|
547
|
+
});
|
|
548
|
+
} else {
|
|
549
|
+
candidates.sort((a, b) => {
|
|
550
|
+
const ar = a.getBoundingClientRect();
|
|
551
|
+
const br = b.getBoundingClientRect();
|
|
552
|
+
const avis = isInViewport(a) ? 0 : 1;
|
|
553
|
+
const bvis = isInViewport(b) ? 0 : 1;
|
|
554
|
+
return avis - bvis || ar.top - br.top || ar.left - br.left;
|
|
555
|
+
});
|
|
556
|
+
}
|
|
557
|
+
const visibleCandidates = candidates.filter(isElementVisible);
|
|
558
|
+
const elements = visibleCandidates.slice(0, maxElements).map((element, index) => summarizeElement(element, index));
|
|
559
|
+
const queryElements = query
|
|
560
|
+
? visibleCandidates.slice(0, Math.max(maxElements, 500)).map((element, index) => summarizeElement(element, index))
|
|
561
|
+
: elements;
|
|
562
|
+
const map = pageMap();
|
|
563
|
+
const forms = formSummaries();
|
|
564
|
+
const layout = layoutSections(elements, forms);
|
|
565
|
+
const focused = activeElementSummary();
|
|
566
|
+
const modal = modalSummary();
|
|
567
|
+
const bodyText = document.body ? document.body.innerText.replace(/\s+\n/g, "\n").trim() : "";
|
|
568
|
+
const text = bodyText.slice(0, fullTextLimit);
|
|
569
|
+
const snapshot = {
|
|
570
|
+
title: document.title,
|
|
571
|
+
url: location.href,
|
|
572
|
+
mode,
|
|
573
|
+
query: query || undefined,
|
|
574
|
+
viewport: { width: innerWidth, height: innerHeight, scrollX, scrollY },
|
|
575
|
+
summary: {
|
|
576
|
+
visibleText: textOf(document.body, 500),
|
|
577
|
+
visibleInteractiveCount: elements.filter((el) => el.rect.y >= 0 && el.rect.y <= innerHeight).length,
|
|
578
|
+
totalInteractiveSampled: elements.length,
|
|
579
|
+
totalInteractiveVisible: visibleCandidates.length,
|
|
580
|
+
focused: focused ? { uid: focused.uid, role: focused.role, label: focused.label } : undefined,
|
|
581
|
+
modal: modal ? { uid: modal.uid, label: modal.label } : undefined,
|
|
582
|
+
hints: [],
|
|
583
|
+
},
|
|
584
|
+
focused: focused || undefined,
|
|
585
|
+
modal: modal || undefined,
|
|
586
|
+
text,
|
|
587
|
+
textTruncated: bodyText.length > text.length,
|
|
588
|
+
textSnippets: visibleTextSnippets(mode === "text" ? 12000 : 3000),
|
|
589
|
+
elements,
|
|
590
|
+
forms,
|
|
591
|
+
layout,
|
|
592
|
+
pageMap: map,
|
|
593
|
+
matches: queryMatches(query, queryElements, map),
|
|
594
|
+
filter: { containingText: containingText || undefined, roleFilter: roleFilter || undefined, nearUid: nearUid || undefined },
|
|
595
|
+
};
|
|
596
|
+
if (snapshot.modal) snapshot.summary.hints.push("A modal/dialog is visible; interact with it before the underlying page.");
|
|
597
|
+
const disabledImportant = elements.find((el) => el.disabled && /submit|save|merge|continue|next|send|approve|login|sign in/i.test(el.label || ""));
|
|
598
|
+
if (disabledImportant) snapshot.summary.hints.push(`${disabledImportant.uid} '${disabledImportant.label}' is disabled.`);
|
|
599
|
+
const occluded = elements.find((el) => el.occluded);
|
|
600
|
+
if (occluded) snapshot.summary.hints.push(`${occluded.uid} '${occluded.label || occluded.role}' appears occluded by ${occluded.occluded.tag}.`);
|
|
601
|
+
|
|
602
|
+
const state = getPiChromeState();
|
|
603
|
+
const currentDigest = digestFor(snapshot);
|
|
604
|
+
snapshot.diff = diffSnapshot(state.lastSnapshotDigest, currentDigest);
|
|
605
|
+
state.lastSnapshotDigest = currentDigest;
|
|
606
|
+
|
|
607
|
+
if (mode === "interactive") {
|
|
608
|
+
delete snapshot.text;
|
|
609
|
+
delete snapshot.textSnippets;
|
|
610
|
+
delete snapshot.pageMap;
|
|
611
|
+
} else if (mode === "forms") {
|
|
612
|
+
delete snapshot.text;
|
|
613
|
+
delete snapshot.textSnippets;
|
|
614
|
+
snapshot.elements = elements.filter((el) => ["textbox", "checkbox", "radio", "combobox", "button"].includes(el.role));
|
|
615
|
+
} else if (mode === "pageMap") {
|
|
616
|
+
delete snapshot.text;
|
|
617
|
+
delete snapshot.textSnippets;
|
|
618
|
+
snapshot.elements = elements.slice(0, 20);
|
|
619
|
+
} else if (mode === "changes") {
|
|
620
|
+
delete snapshot.text;
|
|
621
|
+
delete snapshot.textSnippets;
|
|
622
|
+
delete snapshot.elements;
|
|
623
|
+
delete snapshot.forms;
|
|
624
|
+
delete snapshot.layout;
|
|
625
|
+
delete snapshot.pageMap;
|
|
626
|
+
} else if (mode === "text") {
|
|
627
|
+
snapshot.elements = elements.slice(0, 20);
|
|
628
|
+
} else if (mode !== "full") {
|
|
629
|
+
snapshot.elements = elements.slice(0, Math.min(maxElements, 40));
|
|
630
|
+
snapshot.text = text.slice(0, Math.min(text.length, 6000));
|
|
631
|
+
}
|
|
632
|
+
return snapshot;
|
|
633
|
+
}
|
|
634
|
+
|
|
635
|
+
function inspectTarget(uid, selector, shouldScrollIntoView) {
|
|
636
|
+
installPiChromeInstrumentation();
|
|
637
|
+
const state = getPiChromeState();
|
|
638
|
+
let element = null;
|
|
639
|
+
if (uid) element = state.elements[uid];
|
|
640
|
+
if (!element && selector) element = document.querySelector(selector);
|
|
641
|
+
if (!element || !element.isConnected) throw new Error(uid ? `No live element for uid: ${uid}. Take a fresh chrome_snapshot.` : `No element matches selector: ${selector}`);
|
|
642
|
+
if (shouldScrollIntoView) element.scrollIntoView?.({ block: "center", inline: "center", behavior: "instant" });
|
|
643
|
+
const summary = summarizeElement(element, 0);
|
|
644
|
+
const ancestors = [];
|
|
645
|
+
let current = element.parentElement;
|
|
646
|
+
while (current && current !== document.body && ancestors.length < 6) {
|
|
647
|
+
ancestors.push({ uid: rememberElement(current), tag: current.tagName.toLowerCase(), role: roleOf(current), label: accessibleLabel(current) || textOf(current, 100), selector: selectorFor(current) });
|
|
648
|
+
current = current.parentElement;
|
|
649
|
+
}
|
|
650
|
+
const container = element.closest?.('form, dialog, [role="dialog"], [aria-modal="true"], section, article, main, aside') || element.parentElement || document.body;
|
|
651
|
+
const nearbyText = Array.from(container.querySelectorAll("h1,h2,h3,h4,p,li,label,[role='alert']"))
|
|
652
|
+
.filter(isElementVisible)
|
|
653
|
+
.slice(0, 24)
|
|
654
|
+
.map((node) => ({ uid: rememberElement(node), tag: node.tagName.toLowerCase(), text: textOf(node, 240), rect: rectSummary(node) }))
|
|
655
|
+
.filter((entry) => entry.text);
|
|
656
|
+
const nearbyActions = Array.from(container.querySelectorAll('a, button, input, textarea, select, [role="button"], [role="link"], [tabindex]:not([tabindex="-1"])'))
|
|
657
|
+
.filter(isElementVisible)
|
|
658
|
+
.slice(0, 30)
|
|
659
|
+
.map((node, index) => summarizeElement(node, index));
|
|
660
|
+
const form = element.closest?.("form");
|
|
661
|
+
const formContext = form ? {
|
|
662
|
+
uid: rememberElement(form),
|
|
663
|
+
label: accessibleLabel(form) || textOf(form, 160),
|
|
664
|
+
fields: Array.from(form.querySelectorAll('input, textarea, select, [contenteditable="true"]')).filter(isElementVisible).slice(0, 30).map((node, index) => summarizeElement(node, index)),
|
|
665
|
+
actions: Array.from(form.querySelectorAll('button, input[type="submit"], [role="button"]')).filter(isElementVisible).slice(0, 12).map((node, index) => summarizeElement(node, index)),
|
|
666
|
+
} : undefined;
|
|
667
|
+
const rect = element.getBoundingClientRect();
|
|
668
|
+
const center = { x: Math.round(rect.left + rect.width / 2), y: Math.round(rect.top + rect.height / 2) };
|
|
669
|
+
const clickSuggestion = summary.disabled || summary.inert || summary.pointerEvents === "none"
|
|
670
|
+
? undefined
|
|
671
|
+
: { uid: summary.uid, x: center.x, y: center.y };
|
|
672
|
+
return { target: summary, ancestors, nearbyText, nearbyActions, formContext, clickSuggestion };
|
|
673
|
+
}
|
|
674
|
+
|
|
675
|
+
globalThis.__piChromeSnapshotPage = snapshotPage;
|
|
676
|
+
globalThis.__piChromeInspectTarget = inspectTarget;
|
|
677
|
+
})();
|
|
@@ -74,6 +74,137 @@ function safeJson(value: unknown): string {
|
|
|
74
74
|
return JSON.stringify(value, null, 2);
|
|
75
75
|
}
|
|
76
76
|
|
|
77
|
+
const snapshotModeValues = ["auto", "interactive", "forms", "pageMap", "text", "changes", "full"] as const;
|
|
78
|
+
|
|
79
|
+
function compactLine(value: unknown, max = 140): string {
|
|
80
|
+
const text = String(value ?? "").replace(/\s+/g, " ").trim();
|
|
81
|
+
return text.length > max ? `${text.slice(0, max - 1)}…` : text;
|
|
82
|
+
}
|
|
83
|
+
|
|
84
|
+
function rectText(rect: any): string {
|
|
85
|
+
if (!rect) return "?";
|
|
86
|
+
return `${rect.x},${rect.y} ${rect.width}x${rect.height}`;
|
|
87
|
+
}
|
|
88
|
+
|
|
89
|
+
function formatChromeSnapshot(snapshot: any): string {
|
|
90
|
+
if (!snapshot || typeof snapshot !== "object") return safeJson(snapshot);
|
|
91
|
+
if (snapshot.mode === "full") return truncateText(safeJson(snapshot));
|
|
92
|
+
const lines: string[] = [];
|
|
93
|
+
lines.push(`# Chrome snapshot${snapshot.mode ? ` (${snapshot.mode})` : ""}`);
|
|
94
|
+
lines.push(`${snapshot.title || "(untitled)"}`);
|
|
95
|
+
if (snapshot.url) lines.push(`${snapshot.url}`);
|
|
96
|
+
if (snapshot.viewport) lines.push(`viewport=${snapshot.viewport.width}x${snapshot.viewport.height} scroll=${snapshot.viewport.scrollX || 0},${snapshot.viewport.scrollY || 0}`);
|
|
97
|
+
if (snapshot.summary?.modal) lines.push(`modal: ${snapshot.summary.modal.uid} ${compactLine(snapshot.summary.modal.label)}`);
|
|
98
|
+
if (snapshot.summary?.focused) lines.push(`focused: ${snapshot.summary.focused.uid} ${snapshot.summary.focused.role || ""} ${compactLine(snapshot.summary.focused.label)}`);
|
|
99
|
+
if (Array.isArray(snapshot.summary?.hints) && snapshot.summary.hints.length) {
|
|
100
|
+
lines.push("\n## Hints");
|
|
101
|
+
for (const hint of snapshot.summary.hints.slice(0, 6)) lines.push(`- ${hint}`);
|
|
102
|
+
}
|
|
103
|
+
if (snapshot.diff && !snapshot.diff.firstSnapshot) {
|
|
104
|
+
const changed = [
|
|
105
|
+
...(snapshot.diff.changes || []).map((c: any) => c.kind === "textChanged" ? "text changed" : `${c.kind}: ${compactLine(c.before, 50)} → ${compactLine(c.after, 50)}`),
|
|
106
|
+
...(snapshot.diff.added || []).slice(0, 4).map((e: any) => `added ${e.uid} ${e.role || ""} ${compactLine(e.label)}`),
|
|
107
|
+
...(snapshot.diff.updated || []).slice(0, 4).map((u: any) => `updated ${u.uid} ${compactLine(u.after?.label || u.before?.label)}`),
|
|
108
|
+
];
|
|
109
|
+
if (changed.length) {
|
|
110
|
+
lines.push("\n## Changed since last snapshot");
|
|
111
|
+
for (const item of changed.slice(0, 10)) lines.push(`- ${item}`);
|
|
112
|
+
}
|
|
113
|
+
}
|
|
114
|
+
if (Array.isArray(snapshot.matches) && snapshot.matches.length) {
|
|
115
|
+
lines.push(`\n## Matches for "${snapshot.query}"`);
|
|
116
|
+
for (const match of snapshot.matches.slice(0, 12)) {
|
|
117
|
+
if (match.kind === "text") lines.push(`- ${match.uid} text ${compactLine(match.text)} @ ${rectText(match.rect)}`);
|
|
118
|
+
else if (match.kind === "region") lines.push(`- ${match.uid} region ${compactLine(match.label)} headings=${(match.headings || []).map((h: string) => compactLine(h, 50)).join(" | ")}`);
|
|
119
|
+
else lines.push(`- ${match.uid} ${match.role || match.tag || "element"}${match.disabled ? " disabled" : ""} ${compactLine(match.label || match.selector)} @ ${rectText(match.rect)}`);
|
|
120
|
+
}
|
|
121
|
+
}
|
|
122
|
+
if (snapshot.mode === "pageMap" && snapshot.pageMap) {
|
|
123
|
+
lines.push("\n## Page map");
|
|
124
|
+
for (const region of (snapshot.pageMap.regions || []).slice(0, 18)) {
|
|
125
|
+
lines.push(`- ${region.uid} ${region.kind}: ${compactLine(region.label)}`);
|
|
126
|
+
for (const action of (region.actions || []).slice(0, 5)) lines.push(` - ${action.uid} ${action.role || ""}${action.disabled ? " disabled" : ""} ${compactLine(action.label)}`);
|
|
127
|
+
}
|
|
128
|
+
if (snapshot.pageMap.headings?.length) {
|
|
129
|
+
lines.push("\nHeadings:");
|
|
130
|
+
for (const h of snapshot.pageMap.headings.slice(0, 20)) lines.push(`- ${h.uid} h${h.level || ""} ${compactLine(h.text)}`);
|
|
131
|
+
}
|
|
132
|
+
}
|
|
133
|
+
if (Array.isArray(snapshot.layout) && snapshot.layout.length && snapshot.mode !== "changes") {
|
|
134
|
+
lines.push("\n## Layout / context");
|
|
135
|
+
for (const section of snapshot.layout.slice(0, snapshot.mode === "pageMap" ? 18 : 8)) {
|
|
136
|
+
const bits = [`${section.uid}`, section.role || section.tag, compactLine(section.label || section.text || "(unnamed section)", 110), `@ ${rectText(section.rect)}`];
|
|
137
|
+
lines.push(`- ${bits.filter(Boolean).join(" ")}`);
|
|
138
|
+
const fieldLabels = (section.fields || []).slice(0, 4).map((f: any) => `${f.uid} ${compactLine(f.label || f.role, 40)}`);
|
|
139
|
+
const actionLabels = (section.actions || []).slice(0, 5).map((a: any) => `${a.uid}${a.disabled ? " disabled" : ""} ${compactLine(a.label || a.role, 40)}`);
|
|
140
|
+
if (fieldLabels.length) lines.push(` fields: ${fieldLabels.join("; ")}`);
|
|
141
|
+
if (actionLabels.length) lines.push(` actions: ${actionLabels.join("; ")}`);
|
|
142
|
+
}
|
|
143
|
+
}
|
|
144
|
+
if ((snapshot.mode === "forms" || snapshot.forms?.fields?.length) && snapshot.mode !== "pageMap") {
|
|
145
|
+
const fields = snapshot.forms?.fields || [];
|
|
146
|
+
const submits = snapshot.forms?.submits || [];
|
|
147
|
+
if (fields.length || submits.length) lines.push("\n## Forms");
|
|
148
|
+
for (const field of fields.slice(0, snapshot.mode === "forms" ? 40 : 12)) {
|
|
149
|
+
const bits = [field.uid, field.role || field.tag, field.required ? "required" : "", field.invalid ? "invalid" : "", field.disabled ? "disabled" : "", compactLine(field.label || field.selector, 90)];
|
|
150
|
+
if (field.value) bits.push(`value=${compactLine(field.value, 50)}`);
|
|
151
|
+
else if (field.valueRedacted) bits.push("value=[redacted]");
|
|
152
|
+
lines.push(`- ${bits.filter(Boolean).join(" ")} @ ${rectText(field.rect)}`);
|
|
153
|
+
}
|
|
154
|
+
for (const submit of submits.slice(0, 8)) lines.push(`- ${submit.uid} submit/action${submit.disabled ? " disabled" : ""} ${compactLine(submit.label || submit.selector)} @ ${rectText(submit.rect)}`);
|
|
155
|
+
}
|
|
156
|
+
if (Array.isArray(snapshot.elements) && snapshot.mode !== "pageMap") {
|
|
157
|
+
lines.push("\n## Visible actions");
|
|
158
|
+
for (const el of snapshot.elements.slice(0, snapshot.mode === "interactive" ? 60 : 25)) {
|
|
159
|
+
const flags = [el.disabled ? "disabled" : "", el.occluded ? `occluded-by-${el.occluded.tag}` : ""].filter(Boolean).join(",");
|
|
160
|
+
const context = el.context?.label ? ` in ${el.context.uid} ${compactLine(el.context.label, 60)}` : "";
|
|
161
|
+
lines.push(`- ${el.uid} ${el.role || el.tag}${flags ? ` [${flags}]` : ""} ${compactLine(el.label || el.selector)}${context} @ ${rectText(el.rect)}`);
|
|
162
|
+
}
|
|
163
|
+
if (snapshot.elements.length > (snapshot.mode === "interactive" ? 60 : 25)) lines.push(`- … ${snapshot.elements.length - (snapshot.mode === "interactive" ? 60 : 25)} more; retry with maxElements or mode=interactive`);
|
|
164
|
+
}
|
|
165
|
+
if ((snapshot.mode === "text" || snapshot.mode === "auto") && Array.isArray(snapshot.textSnippets) && snapshot.textSnippets.length) {
|
|
166
|
+
lines.push("\n## Text snippets");
|
|
167
|
+
for (const snip of snapshot.textSnippets.slice(0, snapshot.mode === "text" ? 40 : 14)) lines.push(`- ${snip.uid} ${compactLine(snip.text, snapshot.mode === "text" ? 240 : 160)}`);
|
|
168
|
+
if (snapshot.textTruncated) lines.push("- … page text truncated; retry with mode=text or maxTextChars for more");
|
|
169
|
+
}
|
|
170
|
+
lines.push("\nTip: use chrome_snapshot({query:'...', mode:'interactive|forms|pageMap|text|changes|full'}) or nearUid to zoom in.");
|
|
171
|
+
return truncateText(lines.join("\n"));
|
|
172
|
+
}
|
|
173
|
+
|
|
174
|
+
function formatIncludedSnapshotText(raw: unknown, text: string): string {
|
|
175
|
+
const snapshot = raw && typeof raw === "object" ? (raw as { snapshot?: unknown }).snapshot : undefined;
|
|
176
|
+
return snapshot ? `${text}\n\n${formatChromeSnapshot(snapshot)}` : text;
|
|
177
|
+
}
|
|
178
|
+
|
|
179
|
+
function formatChromeInspect(inspect: any): string {
|
|
180
|
+
if (!inspect || typeof inspect !== "object") return safeJson(inspect);
|
|
181
|
+
const t = inspect.target || {};
|
|
182
|
+
const lines: string[] = [];
|
|
183
|
+
lines.push(`# Chrome inspect ${t.uid || ""}`.trim());
|
|
184
|
+
lines.push(`${t.role || t.tag || "element"}${t.disabled ? " disabled" : ""}${t.occluded ? ` occluded-by-${t.occluded.tag}` : ""} ${compactLine(t.label || t.selector)}`);
|
|
185
|
+
if (t.selector) lines.push(`selector: ${t.selector}`);
|
|
186
|
+
if (t.rect) lines.push(`rect: ${rectText(t.rect)}`);
|
|
187
|
+
if (inspect.clickSuggestion) lines.push(`suggested click: chrome_click({ uid: "${inspect.clickSuggestion.uid}" }) or x=${inspect.clickSuggestion.x}, y=${inspect.clickSuggestion.y}`);
|
|
188
|
+
if (Array.isArray(inspect.nearbyText) && inspect.nearbyText.length) {
|
|
189
|
+
lines.push("\n## Nearby text");
|
|
190
|
+
for (const item of inspect.nearbyText.slice(0, 12)) lines.push(`- ${item.uid} ${compactLine(item.text, 180)}`);
|
|
191
|
+
}
|
|
192
|
+
if (inspect.formContext) {
|
|
193
|
+
lines.push("\n## Form context");
|
|
194
|
+
for (const field of (inspect.formContext.fields || []).slice(0, 20)) lines.push(`- ${field.uid} ${field.role || field.tag}${field.disabled ? " disabled" : ""} ${compactLine(field.label || field.selector)}${field.value ? ` value=${compactLine(field.value, 60)}` : field.valueRedacted ? " value=[redacted]" : ""}`);
|
|
195
|
+
for (const action of (inspect.formContext.actions || []).slice(0, 10)) lines.push(`- ${action.uid} action${action.disabled ? " disabled" : ""} ${compactLine(action.label || action.selector)}`);
|
|
196
|
+
}
|
|
197
|
+
if (Array.isArray(inspect.nearbyActions) && inspect.nearbyActions.length) {
|
|
198
|
+
lines.push("\n## Nearby actions");
|
|
199
|
+
for (const action of inspect.nearbyActions.slice(0, 18)) lines.push(`- ${action.uid} ${action.role || action.tag}${action.disabled ? " disabled" : ""} ${compactLine(action.label || action.selector)} @ ${rectText(action.rect)}`);
|
|
200
|
+
}
|
|
201
|
+
if (Array.isArray(inspect.ancestors) && inspect.ancestors.length) {
|
|
202
|
+
lines.push("\n## Ancestors");
|
|
203
|
+
for (const a of inspect.ancestors.slice(0, 6)) lines.push(`- ${a.uid} ${a.role || a.tag} ${compactLine(a.label || a.selector, 120)}`);
|
|
204
|
+
}
|
|
205
|
+
return truncateText(lines.join("\n"));
|
|
206
|
+
}
|
|
207
|
+
|
|
77
208
|
function extensionRoot(): string {
|
|
78
209
|
// Resolve relative to this extension file, not ctx.cwd. ctx.cwd can temporarily be
|
|
79
210
|
// an attachment/clipboard path when Pi is handling pasted images.
|
|
@@ -1109,13 +1240,16 @@ Usage rules:
|
|
|
1109
1240
|
name: "chrome_snapshot",
|
|
1110
1241
|
label: "Chrome Snapshot",
|
|
1111
1242
|
description:
|
|
1112
|
-
"Inspect a page in the user's existing Chrome profile
|
|
1113
|
-
promptSnippet: "
|
|
1243
|
+
"Inspect a page in the user's existing Chrome profile. Default output is a concise, agent-friendly observation with structural layout/context, stable uids, visible actions, form fields, page hints, and changes since the previous snapshot. Use mode/query/nearUid to zoom instead of dumping the whole page. Runs in the background by default; pass background=false to bring Chrome to the foreground so the user can watch.",
|
|
1244
|
+
promptSnippet: "Observe the current Chrome page: concise summary, structural layout, visible actions, forms, page map, query matches, and stable uids.",
|
|
1114
1245
|
parameters: Type.Object({
|
|
1115
1246
|
targetId: Type.Optional(Type.String()),
|
|
1116
1247
|
urlIncludes: Type.Optional(Type.String()),
|
|
1117
1248
|
titleIncludes: Type.Optional(Type.String()),
|
|
1118
1249
|
maxElements: Type.Optional(Type.Number({ default: MAX_ELEMENTS })),
|
|
1250
|
+
mode: Type.Optional(StringEnum(snapshotModeValues)),
|
|
1251
|
+
query: Type.Optional(Type.String({ description: "Find/rank elements, regions, and text matching this phrase, e.g. 'merge button', 'email error', 'approve PR'." })),
|
|
1252
|
+
maxTextChars: Type.Optional(Type.Number({ description: "Max body text chars included in the underlying snapshot. Defaults are smaller for concise modes." })),
|
|
1119
1253
|
containingText: Type.Optional(Type.String({ description: "Only return elements whose label/text contains this string (case-insensitive). Useful when the page has many controls." })),
|
|
1120
1254
|
roleFilter: Type.Optional(Type.String({ description: "Only return elements matching this ARIA role or tag name (case-insensitive). e.g. 'button', 'link', 'textbox'." })),
|
|
1121
1255
|
nearUid: Type.Optional(Type.String({ description: "Sort elements by proximity to this snapshot uid. Useful for finding controls near a known anchor." })),
|
|
@@ -1132,7 +1266,84 @@ Usage rules:
|
|
|
1132
1266
|
DEFAULT_TIMEOUT_MS,
|
|
1133
1267
|
signal,
|
|
1134
1268
|
);
|
|
1135
|
-
return { content: [{ type: "text", text:
|
|
1269
|
+
return { content: [{ type: "text", text: formatChromeSnapshot(snapshot) }], details: { snapshot } };
|
|
1270
|
+
},
|
|
1271
|
+
});
|
|
1272
|
+
|
|
1273
|
+
pi.registerTool({
|
|
1274
|
+
name: "chrome_find",
|
|
1275
|
+
label: "Chrome Find",
|
|
1276
|
+
description:
|
|
1277
|
+
"Find elements, page regions, or text on the current Chrome page by query. Returns ranked matches with stable uids and coordinates. This is a focused wrapper around chrome_snapshot({ query }).",
|
|
1278
|
+
promptSnippet: "Find matching controls/text/regions in Chrome by natural-language query and return stable uids.",
|
|
1279
|
+
parameters: Type.Object({
|
|
1280
|
+
query: Type.String({ description: "What to find, e.g. 'merge button', 'email error', 'approve PR', 'search box'." }),
|
|
1281
|
+
mode: Type.Optional(StringEnum(snapshotModeValues)),
|
|
1282
|
+
maxElements: Type.Optional(Type.Number({ default: MAX_ELEMENTS })),
|
|
1283
|
+
targetId: Type.Optional(Type.String()),
|
|
1284
|
+
urlIncludes: Type.Optional(Type.String()),
|
|
1285
|
+
titleIncludes: Type.Optional(Type.String()),
|
|
1286
|
+
background: Type.Optional(
|
|
1287
|
+
Type.Boolean({ description: "If true (the default), run silently in the background without focusing Chrome; pass false so Chrome focuses + the tab activates and the user can watch." }),
|
|
1288
|
+
),
|
|
1289
|
+
host: Type.Optional(Type.String()),
|
|
1290
|
+
port: Type.Optional(Type.Number()),
|
|
1291
|
+
}),
|
|
1292
|
+
async execute(_id, params, signal): Promise<ToolTextResult> {
|
|
1293
|
+
const snapshot = await authorizedBridgeSend(
|
|
1294
|
+
"page.snapshot",
|
|
1295
|
+
withBackground({ ...params, mode: params.mode || "auto", maxElements: params.maxElements ?? MAX_ELEMENTS }),
|
|
1296
|
+
DEFAULT_TIMEOUT_MS,
|
|
1297
|
+
signal,
|
|
1298
|
+
);
|
|
1299
|
+
return { content: [{ type: "text", text: formatChromeSnapshot(snapshot) }], details: { snapshot } };
|
|
1300
|
+
},
|
|
1301
|
+
});
|
|
1302
|
+
|
|
1303
|
+
pi.registerTool({
|
|
1304
|
+
name: "chrome_inspect",
|
|
1305
|
+
label: "Chrome Inspect Element",
|
|
1306
|
+
description:
|
|
1307
|
+
"Inspect one snapshot uid or selector deeply: nearby text, nearby actions, form context, ancestors, and suggested click target. Use after chrome_snapshot/chrome_find when you need context around one element.",
|
|
1308
|
+
promptSnippet: "Inspect a Chrome snapshot uid deeply for nearby text, form context, and suggested actions.",
|
|
1309
|
+
parameters: Type.Object({
|
|
1310
|
+
uid: Type.Optional(Type.String({ description: "Stable element uid from chrome_snapshot/chrome_find." })),
|
|
1311
|
+
selector: Type.Optional(Type.String({ description: "CSS selector if uid is unavailable." })),
|
|
1312
|
+
scrollIntoView: Type.Optional(Type.Boolean({ description: "If true, scroll the target into view before inspecting. Default false to avoid changing page state." })),
|
|
1313
|
+
targetId: Type.Optional(Type.String()),
|
|
1314
|
+
urlIncludes: Type.Optional(Type.String()),
|
|
1315
|
+
titleIncludes: Type.Optional(Type.String()),
|
|
1316
|
+
background: Type.Optional(
|
|
1317
|
+
Type.Boolean({ description: "If true (the default), run silently in the background without focusing Chrome; pass false so Chrome focuses + the tab activates and the user can watch." }),
|
|
1318
|
+
),
|
|
1319
|
+
host: Type.Optional(Type.String()),
|
|
1320
|
+
port: Type.Optional(Type.Number()),
|
|
1321
|
+
}),
|
|
1322
|
+
async execute(_id, params, signal): Promise<ToolTextResult> {
|
|
1323
|
+
try {
|
|
1324
|
+
const inspect = await authorizedBridgeSend("page.inspect", withBackground(params), DEFAULT_TIMEOUT_MS, signal);
|
|
1325
|
+
return { content: [{ type: "text", text: formatChromeInspect(inspect) }], details: { inspect } };
|
|
1326
|
+
} catch (error) {
|
|
1327
|
+
const message = error instanceof Error ? error.message : String(error);
|
|
1328
|
+
if (!/Unknown action: page\.inspect/i.test(message)) throw error;
|
|
1329
|
+
// Compatibility fallback for a loaded Chrome extension service worker that has not
|
|
1330
|
+
// been reloaded since chrome_inspect was added. It is less rich than page.inspect,
|
|
1331
|
+
// but still gives useful nearby candidates instead of failing the workflow.
|
|
1332
|
+
const snapshot = await authorizedBridgeSend(
|
|
1333
|
+
"page.snapshot",
|
|
1334
|
+
withBackground({
|
|
1335
|
+
...params,
|
|
1336
|
+
mode: "interactive",
|
|
1337
|
+
maxElements: MAX_ELEMENTS,
|
|
1338
|
+
nearUid: params.uid,
|
|
1339
|
+
query: params.selector,
|
|
1340
|
+
}),
|
|
1341
|
+
DEFAULT_TIMEOUT_MS,
|
|
1342
|
+
signal,
|
|
1343
|
+
);
|
|
1344
|
+
const text = `chrome_inspect fallback: loaded Chrome extension does not yet support page.inspect; reload it at chrome://extensions for deep inspect.\n\n${formatChromeSnapshot(snapshot)}`;
|
|
1345
|
+
return { content: [{ type: "text", text }], details: { snapshot, fallback: "page.snapshot" } };
|
|
1346
|
+
}
|
|
1136
1347
|
},
|
|
1137
1348
|
});
|
|
1138
1349
|
|
|
@@ -1219,7 +1430,7 @@ Usage rules:
|
|
|
1219
1430
|
const summary = summarizeActionResult(result);
|
|
1220
1431
|
const target = params.uid ?? params.selector ?? `${params.x},${params.y}`;
|
|
1221
1432
|
const text = summary ? `Clicked ${target} — ${summary}` : `Clicked ${target}`;
|
|
1222
|
-
return { content: [{ type: "text", text }], details: { result: raw as Json } };
|
|
1433
|
+
return { content: [{ type: "text", text: formatIncludedSnapshotText(raw, text) }], details: { result: raw as Json } };
|
|
1223
1434
|
},
|
|
1224
1435
|
});
|
|
1225
1436
|
|
|
@@ -1251,7 +1462,8 @@ Usage rules:
|
|
|
1251
1462
|
const summary = summarizeActionResult(result);
|
|
1252
1463
|
const into = params.uid || params.selector ? ` into ${params.uid ?? params.selector}` : "";
|
|
1253
1464
|
const base = `Typed ${params.text.length} character(s)${into}.`;
|
|
1254
|
-
|
|
1465
|
+
const text = summary ? `${base} (${summary})` : base;
|
|
1466
|
+
return { content: [{ type: "text", text: formatIncludedSnapshotText(raw, text) }], details: { result: raw as Json } };
|
|
1255
1467
|
},
|
|
1256
1468
|
});
|
|
1257
1469
|
|
|
@@ -1283,7 +1495,8 @@ Usage rules:
|
|
|
1283
1495
|
const summary = summarizeActionResult(result);
|
|
1284
1496
|
const into = params.uid || params.selector ? ` into ${params.uid ?? params.selector}` : "";
|
|
1285
1497
|
const base = `Filled ${params.text.length} character(s)${into}.`;
|
|
1286
|
-
|
|
1498
|
+
const text = summary ? `${base} (${summary})` : base;
|
|
1499
|
+
return { content: [{ type: "text", text: formatIncludedSnapshotText(raw, text) }], details: { result: raw as Json } };
|
|
1287
1500
|
},
|
|
1288
1501
|
});
|
|
1289
1502
|
|
|
@@ -1317,7 +1530,8 @@ Usage rules:
|
|
|
1317
1530
|
const result = (params.includeSnapshot ? (raw as { result: unknown }).result : raw) as Json;
|
|
1318
1531
|
const summary = summarizeActionResult(result);
|
|
1319
1532
|
const base = `Pressed ${params.key}.`;
|
|
1320
|
-
|
|
1533
|
+
const text = summary ? `${base} (${summary})` : base;
|
|
1534
|
+
return { content: [{ type: "text", text: formatIncludedSnapshotText(raw, text) }], details: { result: raw as Json } };
|
|
1321
1535
|
},
|
|
1322
1536
|
});
|
|
1323
1537
|
|