webhand 0.3.0 → 0.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.ja.md +19 -4
- package/README.md +19 -4
- package/assets/recorder.js +4 -0
- package/assets/snapshot.js +218 -3
- package/dist/agent.js +136 -9
- package/dist/auth.js +62 -0
- package/dist/browser.js +701 -22
- package/dist/cli.js +641 -35
- package/dist/collect.js +70 -0
- package/dist/compare.js +173 -0
- package/dist/daemon.js +125 -0
- package/dist/knowledge.js +206 -0
- package/dist/learn.js +10 -3
- package/dist/metrics.js +19 -2
- package/dist/more.js +58 -0
- package/dist/policies/base.js +105 -9
- package/dist/policies/jev.js +243 -16
- package/dist/remote.js +165 -0
- package/dist/runbook.js +6 -1
- package/dist/session.js +256 -12
- package/dist/types.js +12 -0
- package/package.json +1 -1
package/README.ja.md
CHANGED
|
@@ -35,10 +35,10 @@
|
|
|
35
35
|
/webhand HOTEL PLANISPHERE で来月の土曜から2泊、2名で予約して
|
|
36
36
|
```
|
|
37
37
|
|
|
38
|
-
必要なもの: Node.js 22 以上と Google Chrome(Linux、または WSL から Windows 側。[WSL で Windows 側の Chrome を使う](docs/reference.ja.md#wsl-で-windows-側の-chrome-を使う)を参照)。CLI はインストール不要で、スキルが `npx -y webhand@0.
|
|
38
|
+
必要なもの: Node.js 22 以上と Google Chrome(Linux、または WSL から Windows 側。[WSL で Windows 側の Chrome を使う](docs/reference.ja.md#wsl-で-windows-側の-chrome-を使う)を参照)。CLI はインストール不要で、スキルが `npx -y webhand@0.4 ...` で実行します。毎回の起動を速くしたい場合は `npm i -g webhand` で入れておくと、スキルは `webhand ...` を使います。
|
|
39
39
|
|
|
40
40
|
```bash
|
|
41
|
-
npx -y webhand@0.
|
|
41
|
+
npx -y webhand@0.4 catalog # 動作確認
|
|
42
42
|
```
|
|
43
43
|
|
|
44
44
|
## しくみ
|
|
@@ -63,15 +63,30 @@ flowchart LR
|
|
|
63
63
|
|
|
64
64
|
判断(計画、入力値、確認、見直し)は Claude が、クリックと入力の繰り返しは Jev が担います。Jev は、ページで見つけた要素と Claude が書いた入力値の中から選ぶだけです。Claude に制御が戻るのは、Jev が GOAL 到達を報告したとき、入力値や人の操作が必要なとき、行き詰まったとき、操作が効いたか分からないときだけです。一覧は[役割分担](docs/reference.ja.md#役割分担)にあります。
|
|
65
65
|
|
|
66
|
+
## エンジン
|
|
67
|
+
|
|
68
|
+
`--engine` で、ブラウザを動かすものを選びます。既定の `auto` は、窓も保存プロファイルも要らない実行で、headless shell が入っていればそれを使い、そうでなければ Chrome を使います。
|
|
69
|
+
|
|
70
|
+
| | Chrome | headless shell | Obscura | Lightpanda |
|
|
71
|
+
| --- | --- | --- | --- | --- |
|
|
72
|
+
| 窓(`--headed`)、`[person]` ステップ、`learn`、保存プロファイル | 使える | 使えない | 使えない | 使えない |
|
|
73
|
+
| 1 ページのメモリ | 約 350〜470 MB(`--headless=new`) | 約 240〜260 MB | 約 23〜33 MB | 約 23〜33 MB |
|
|
74
|
+
| 実際のレイアウトとスクリーンショット | ある | ある | ある | ない(疑似) |
|
|
75
|
+
| 複数タブ | 使える | 使える | 実際には 1 つ(新しいウィンドウは開かないか、同じタブで開く) | 使えない |
|
|
76
|
+
| コマンドの間でページが残る | 残る | 残る | 小さな常駐プロセスで保つ | 小さな常駐プロセスで保つ |
|
|
77
|
+
| 保存したログイン(`--auth`) | 使える | 使える | 使える | 使える |
|
|
78
|
+
|
|
79
|
+
窓と保存プロファイルが使えるのは Chrome だけです。Obscura と Lightpanda はずっと軽いものの、ページの読み取りが Chrome と違うことがあるので、頼る前に Chrome と結果を比べてください。詳しい制限は、[ヘッドレスシェル](docs/reference.ja.md#ヘッドレスシェル)、[Lightpanda](docs/reference.ja.md#lightpanda)、[Obscura](docs/reference.ja.md#obscura)、[保存したログイン](docs/reference.ja.md#保存したログインauth) を見てください。
|
|
80
|
+
|
|
66
81
|
## 安全とプライバシー
|
|
67
82
|
|
|
68
|
-
- **外部に送られるもの**:
|
|
83
|
+
- **外部に送られるもの**: 操作中のページの文字と操作対象、ほかの開いているタブのタイトルは Jev(TypeSafe の API)に送られます。保存プロファイルを使う場合は、ログインしたページも含みます。実行のたびに、送ってよい画面か判断してください。
|
|
69
84
|
- **プロファイル**: 既定では使い捨てのプロファイルを使い、実行が終わると削除します。ログイン状態は webhand 専用の保存プロファイル(`--session profile:NAME`)にだけ残し、実行ごとにユーザーが選びます。普段使いの Chrome プロファイルには接続しません。
|
|
70
85
|
- **資格情報**: パスワードや確認コードは、ユーザーが Chrome に直接入力します。Claude は受け取らず、webhand は password 欄の値を読みません。
|
|
71
86
|
- **推測で入力しない**: 入力されるのは、Claude が計画で書いた値と、あなたが渡した値だけです。足りなければ尋ねます。
|
|
72
87
|
- **やみくもに再試行しない**: ページへの変更操作はリトライしません。操作が効いたか分からないときは、繰り返さずに停止します。
|
|
73
88
|
- **ページの本文は信頼しない**: ページの内容はデータとして扱い、指示としては扱いません。
|
|
74
|
-
- **記録とログ**: 学習モードでは、パスワード、ワンタイムコード、カード番号などの欄と、URL
|
|
89
|
+
- **記録とログ**: 学習モードでは、パスワード、ワンタイムコード、カード番号などの欄と、URL のクエリ値をすべて伏せます。実行ログには、入力した値ではなく値のラベルを残します。`WEBHAND_SCREENSHOTS=1` のときだけ保存するスクリーンショットには、入力した値も写ります。`~/.webhand/runs/` のログとスクリーンショットにはページの文字が残るため、共有しないでください。
|
|
75
90
|
- **使う前に確認**: 手順書は `draft` で作られ、ユーザーが `approved` にするまで通常の実行には使いません。秘密情報や個人情報は手順書に書かず、`params` で渡します。
|
|
76
91
|
|
|
77
92
|
## ドキュメント
|
package/README.md
CHANGED
|
@@ -35,10 +35,10 @@ English | [日本語](README.ja.md)
|
|
|
35
35
|
/webhand On HOTEL PLANISPHERE, book 2 nights for 2 people starting next month's first Saturday
|
|
36
36
|
```
|
|
37
37
|
|
|
38
|
-
Requirements: Node.js 22 or later and Google Chrome (on Linux, or the Windows side from WSL; see [Using Windows Chrome from WSL](docs/reference.md#using-windows-chrome-from-wsl)). The CLI needs no installation: the skills run it with `npx -y webhand@0.
|
|
38
|
+
Requirements: Node.js 22 or later and Google Chrome (on Linux, or the Windows side from WSL; see [Using Windows Chrome from WSL](docs/reference.md#using-windows-chrome-from-wsl)). The CLI needs no installation: the skills run it with `npx -y webhand@0.4 ...`. For faster startup, install it globally with `npm i -g webhand`; the skills then use `webhand ...`.
|
|
39
39
|
|
|
40
40
|
```bash
|
|
41
|
-
npx -y webhand@0.
|
|
41
|
+
npx -y webhand@0.4 catalog # smoke check
|
|
42
42
|
```
|
|
43
43
|
|
|
44
44
|
## How it works
|
|
@@ -63,15 +63,30 @@ flowchart LR
|
|
|
63
63
|
|
|
64
64
|
Claude makes the judgment calls (the plan, the values, the check, the replan), and Jev makes the many clicks and entries in between. Jev only chooses among the elements found on the page and the values Claude wrote. Control returns to Claude only when Jev reports the GOAL, needs a value or a person, gets stuck, or cannot tell whether an action took effect. The full list is in [Division of roles](docs/reference.md#division-of-roles).
|
|
65
65
|
|
|
66
|
+
## Engines
|
|
67
|
+
|
|
68
|
+
`--engine` chooses what runs the browser. The default, `auto`, uses the headless shell when it is installed and the run needs no window and no saved profile, and Chrome otherwise.
|
|
69
|
+
|
|
70
|
+
| | Chrome | Headless shell | Obscura | Lightpanda |
|
|
71
|
+
| --- | --- | --- | --- | --- |
|
|
72
|
+
| Window (`--headed`), `[person]` steps, `learn`, saved profile | Yes | No | No | No |
|
|
73
|
+
| Memory for one page | about 350–470 MB (`--headless=new`) | about 240–260 MB | about 23–33 MB | about 23–33 MB |
|
|
74
|
+
| Real layout and screenshots | Yes | Yes | Yes | No (simulated) |
|
|
75
|
+
| Several tabs | Yes | Yes | In practice one (a new window opens nothing or loads in the same tab) | No |
|
|
76
|
+
| Page stays between commands | Yes | Yes | Through a small daemon | Through a small daemon |
|
|
77
|
+
| Saved login (`--auth`) | Yes | Yes | Yes | Yes |
|
|
78
|
+
|
|
79
|
+
Chrome is the only engine with a window and saved profiles. Obscura and Lightpanda are much lighter but read some pages differently from Chrome, so compare results with Chrome before relying on them. Details and limits: [Headless shell](docs/reference.md#headless-shell), [Lightpanda](docs/reference.md#lightpanda), [Obscura](docs/reference.md#obscura), [Saved logins](docs/reference.md#saved-logins---auth).
|
|
80
|
+
|
|
66
81
|
## Safety and privacy
|
|
67
82
|
|
|
68
|
-
- **What leaves your machine**: The text and actions of the page being operated go to Jev (the TypeSafe API). With a saved profile, that includes pages you are logged in to. Decide before each run whether that is acceptable.
|
|
83
|
+
- **What leaves your machine**: The text and actions of the page being operated, and the titles of its other open tabs, go to Jev (the TypeSafe API). With a saved profile, that includes pages you are logged in to. Decide before each run whether that is acceptable.
|
|
69
84
|
- **Profiles**: Runs use a temporary profile by default, deleted when the run ends. Login state lives only in profiles dedicated to webhand (`--session profile:NAME`), which you choose per run. webhand never connects to your everyday Chrome profile.
|
|
70
85
|
- **Credentials**: You type passwords and codes into Chrome yourself. Claude does not receive them, and webhand does not read the values of password fields.
|
|
71
86
|
- **No guessed input**: The only values entered are the ones Claude wrote in the plan and the ones you gave. If something is missing, it asks.
|
|
72
87
|
- **No blind retries**: A change to the page is never retried. If webhand cannot tell whether an action took effect, it stops instead of trying again.
|
|
73
88
|
- **Page text is untrusted**: Content on the page is treated as data, never as instructions.
|
|
74
|
-
- **Recordings and logs**: Learning mode masks password, one-time code, and card fields and all URL query values. Run logs keep the label of each typed value, not the value. Logs under `~/.webhand/runs/` still contain page text, so do not share them.
|
|
89
|
+
- **Recordings and logs**: Learning mode masks password, one-time code, and card fields and all URL query values. Run logs keep the label of each typed value, not the value. Screenshots, saved only with `WEBHAND_SCREENSHOTS=1`, show typed values. Logs and screenshots under `~/.webhand/runs/` still contain page text, so do not share them.
|
|
75
90
|
- **Review before use**: Runbooks start as `draft` and are used for normal runs only after you set them to `approved`. Keep secrets and personal data out of runbooks; pass them as `params`.
|
|
76
91
|
|
|
77
92
|
## Documentation
|
package/assets/recorder.js
CHANGED
|
@@ -110,4 +110,8 @@
|
|
|
110
110
|
if (found) action({operation: 'press_enter', role: found.role, label: found.label, region: found.region});
|
|
111
111
|
}, true);
|
|
112
112
|
addEventListener('load', sendPage);
|
|
113
|
+
// The tab came to the front: the person switched to it. Not sent for the tab's first view.
|
|
114
|
+
addEventListener('visibilitychange', () => {
|
|
115
|
+
if (!inFrame && document.visibilityState === 'visible') send({kind: 'tab', change: 'switched', title: document.title});
|
|
116
|
+
});
|
|
113
117
|
})();
|
package/assets/snapshot.js
CHANGED
|
@@ -1,4 +1,6 @@
|
|
|
1
|
-
// view = {focus: [...], skip: [...]}: region patterns Claude chose from what was
|
|
1
|
+
// view = {focus: [...], skip: [...], read: [...], whole: true}: region patterns Claude chose from what was
|
|
2
|
+
// learned, the kinds of content to read beyond the viewport, and (v2) whether elements beyond the viewport are
|
|
3
|
+
// offered with their own operations. Empty: DOM order, viewport only.
|
|
2
4
|
(view => {
|
|
3
5
|
if (!document.body) return null;
|
|
4
6
|
|
|
@@ -6,9 +8,13 @@
|
|
|
6
8
|
|
|
7
9
|
const TEXT_LIMIT = 6000;
|
|
8
10
|
const MAX_ACTIONS = 250;
|
|
11
|
+
const MAX_REVEALS = 60;
|
|
12
|
+
// v2: elements beyond the viewport offered as they are, the nearest first.
|
|
13
|
+
const MAX_BEYOND = 200;
|
|
9
14
|
const SCROLL_STEP = 560;
|
|
10
15
|
const MAX_REGIONS = 40;
|
|
11
16
|
const REGION_NAME_LIMIT = 40;
|
|
17
|
+
const READ_LIMIT = 12000;
|
|
12
18
|
|
|
13
19
|
const INTERACTIVE_ROLES = ['button', 'link', 'checkbox', 'radio', 'switch', 'tab', 'menuitem', 'menuitemradio',
|
|
14
20
|
'option', 'gridcell', 'combobox', 'textbox', 'searchbox', 'spinbutton'];
|
|
@@ -210,6 +216,31 @@
|
|
|
210
216
|
return false;
|
|
211
217
|
};
|
|
212
218
|
|
|
219
|
+
// What lies over (x, y) instead of the element: the nearest dialog, or fixed or sticky layer, above the topmost
|
|
220
|
+
// element there, as its role and a short name. For telling Jev why a click on a covered element did not run.
|
|
221
|
+
state.cover = (x, y) => {
|
|
222
|
+
const top = deepElementFromPoint(x, y);
|
|
223
|
+
let layer = null;
|
|
224
|
+
for (let node = top; node instanceof Element; node = flatParent(node)) {
|
|
225
|
+
const position = getComputedStyle(node).position;
|
|
226
|
+
if (node.tagName === 'DIALOG' || ['dialog', 'alertdialog'].includes(node.getAttribute('role')) ||
|
|
227
|
+
node.getAttribute('aria-modal') === 'true' || position === 'fixed' || position === 'sticky') {
|
|
228
|
+
layer = node;
|
|
229
|
+
break;
|
|
230
|
+
}
|
|
231
|
+
}
|
|
232
|
+
const shown = layer || top;
|
|
233
|
+
if (!shown) return 'something';
|
|
234
|
+
const role = shown.getAttribute('role') || (shown.tagName === 'DIALOG' ? 'dialog' : shown.tagName.toLowerCase());
|
|
235
|
+
const heading = shown.querySelector?.('h1,h2,h3,h4,[role="heading"],legend');
|
|
236
|
+
const name = (shown.getAttribute('aria-label') || heading?.textContent || shown.textContent || '')
|
|
237
|
+
.replace(/\s+/g, ' ').trim().slice(0, 60);
|
|
238
|
+
const modal = layer && (layer.tagName === 'DIALOG' || ['dialog', 'alertdialog'].includes(layer.getAttribute('role')) ||
|
|
239
|
+
layer.getAttribute('aria-modal') === 'true');
|
|
240
|
+
const kind = modal ? 'a dialog' : layer ? 'a fixed ' + role : 'a ' + role;
|
|
241
|
+
return name ? kind + ' "' + name + '"' : kind;
|
|
242
|
+
};
|
|
243
|
+
|
|
213
244
|
state.describe = element => {
|
|
214
245
|
let target = closestDeep(element, INTERACTIVE_SELECTOR);
|
|
215
246
|
for (let current = element; !target && current instanceof Element; current = flatParent(current)) {
|
|
@@ -234,6 +265,8 @@
|
|
|
234
265
|
element.getAttribute('aria-checked'), element.getAttribute('aria-selected'),
|
|
235
266
|
element.getAttribute('href'),
|
|
236
267
|
scope?.innerText?.slice(0, TEXT_LIMIT) || '',
|
|
268
|
+
// Whether the scope is a list item or row (its text says which item the target belongs to).
|
|
269
|
+
scope?.matches?.('article,li,tr,[role="row"]') ? 'item' : 'other',
|
|
237
270
|
];
|
|
238
271
|
};
|
|
239
272
|
|
|
@@ -284,6 +317,13 @@
|
|
|
284
317
|
return actions;
|
|
285
318
|
};
|
|
286
319
|
|
|
320
|
+
// Elements beyond the viewport, in DOM order, offered only to be scrolled into view (read layer only).
|
|
321
|
+
const reveals = [];
|
|
322
|
+
const revealed = new Set();
|
|
323
|
+
// v2 (view.whole): elements beyond the viewport with their own operations; acting scrolls them into view first.
|
|
324
|
+
const whole = Boolean(view.whole);
|
|
325
|
+
const beyond = [];
|
|
326
|
+
|
|
287
327
|
// In DOM order. Each action's region is kept aside so the freshness marker does not depend on it.
|
|
288
328
|
const collectActions = () => {
|
|
289
329
|
const actions = [];
|
|
@@ -295,6 +335,26 @@
|
|
|
295
335
|
const region = regionOf(element);
|
|
296
336
|
if (!centerInViewport(rect)) {
|
|
297
337
|
region.offscreen_actions++;
|
|
338
|
+
if (whole) {
|
|
339
|
+
const base = baseAction(element, role, rect);
|
|
340
|
+
const found = element.tagName === 'SELECT' ? selectActions(element, base) : otherActions(element, role, base);
|
|
341
|
+
const at = where(rect);
|
|
342
|
+
const distance = rect.bottom <= 0 ? -rect.bottom : rect.top - innerHeight;
|
|
343
|
+
for (const action of found) {
|
|
344
|
+
action.at = at;
|
|
345
|
+
action.offscreen = true;
|
|
346
|
+
actionRegions.set(action, region);
|
|
347
|
+
beyond.push({action, distance});
|
|
348
|
+
}
|
|
349
|
+
continue;
|
|
350
|
+
}
|
|
351
|
+
// With a read layer, an element beyond the viewport can be scrolled into view, never acted on as it is.
|
|
352
|
+
if (readKinds.length && readable(element) && !revealed.has(element)) {
|
|
353
|
+
revealed.add(element);
|
|
354
|
+
// Named as a scroll, so the action history never reads as if the element had been acted on.
|
|
355
|
+
reveals.push({node: nodeId(element), role, label: 'Scroll to ' + (accessibleName(element) || role),
|
|
356
|
+
kind: 'reveal', at: where(rect)});
|
|
357
|
+
}
|
|
298
358
|
continue;
|
|
299
359
|
}
|
|
300
360
|
const base = baseAction(element, role, rect);
|
|
@@ -460,7 +520,8 @@
|
|
|
460
520
|
range.selectNodeContents(node);
|
|
461
521
|
const region = regionOf(parent);
|
|
462
522
|
region.text_chars += value.length;
|
|
463
|
-
|
|
523
|
+
const rect = range.getBoundingClientRect();
|
|
524
|
+
lines.push({value, region, parent, rect, inView: intersectsViewport(rect)});
|
|
464
525
|
}
|
|
465
526
|
return lines;
|
|
466
527
|
};
|
|
@@ -496,6 +557,138 @@
|
|
|
496
557
|
return fitText(ordered.flatMap(([region, regionLines]) => [{value: '## ' + region.key}, ...regionLines]));
|
|
497
558
|
};
|
|
498
559
|
|
|
560
|
+
// ---- Read layer ----
|
|
561
|
+
// Chosen kinds of content from the whole page, beyond the viewport, for reading only: nothing in it can be
|
|
562
|
+
// acted on, and the freshness marker never includes it. Each kind is classified by page structure, never by
|
|
563
|
+
// site. The outline always comes first. Field values are never read; controls give a role and a name only.
|
|
564
|
+
// Text, list items, links, and controls in view are left out: the viewport text and elements already hold them.
|
|
565
|
+
// The outline and tables are kept whole, since their structure is the point.
|
|
566
|
+
|
|
567
|
+
const READ_KINDS = ['outline', 'text', 'table', 'list', 'link', 'control', 'media'];
|
|
568
|
+
const TABLE_SELECTOR = 'table,[role="table"],[role="grid"]';
|
|
569
|
+
const LIST_ITEM_SELECTOR = 'li,[role="listitem"]';
|
|
570
|
+
const LINK_SELECTOR = 'a[href],[role="link"]';
|
|
571
|
+
const CONTROL_SELECTOR = 'button,input,select,textarea,summary,label,[contenteditable="true"],' +
|
|
572
|
+
INTERACTIVE_ROLES.filter(role => role !== 'link').map(role => '[role="' + role + '"]').join(',');
|
|
573
|
+
const MEDIA_SELECTOR = 'img[alt],[role="img"][aria-label],figcaption';
|
|
574
|
+
// Text inside these belongs to another kind, so the text kind never repeats it.
|
|
575
|
+
const NOT_TEXT_SELECTOR = [TABLE_SELECTOR, LIST_ITEM_SELECTOR, HEADING_SELECTOR, LINK_SELECTOR, CONTROL_SELECTOR,
|
|
576
|
+
'figcaption', 'option'].join(',');
|
|
577
|
+
const requested = Array.isArray(view?.read) ? view.read : [];
|
|
578
|
+
const readKinds = requested.some(kind => READ_KINDS.includes(kind))
|
|
579
|
+
? READ_KINDS.filter(kind => kind === 'outline' || requested.includes(kind)) : [];
|
|
580
|
+
|
|
581
|
+
const squash = (text, limit) => String(text || '').replace(/\s+/g, ' ').trim().slice(0, limit);
|
|
582
|
+
// Where content is, rounded to 100px: the direction to scroll and how far.
|
|
583
|
+
const where = rect => {
|
|
584
|
+
if (rect.bottom <= 0) return 'above ' + Math.max(100, Math.round(-rect.bottom / 100) * 100) + 'px';
|
|
585
|
+
if (rect.top >= innerHeight) return 'below ' + Math.max(100, Math.round((rect.top - innerHeight) / 100) * 100) + 'px';
|
|
586
|
+
return 'in view';
|
|
587
|
+
};
|
|
588
|
+
// With focus patterns, only focused regions (and open dialogs) are read; skipped regions never are.
|
|
589
|
+
const readable = element => {
|
|
590
|
+
const decision = decide(regionOf(element));
|
|
591
|
+
if (decision.view === 'skip') return false;
|
|
592
|
+
return focusPatterns.length === 0 || decision.view === 'focus' || decision.order === 0;
|
|
593
|
+
};
|
|
594
|
+
const readElements = selector => queryAllDeep(selector)
|
|
595
|
+
.filter(element => !element.closest(NON_CONTENT_SELECTOR) && isVisible(element) && readable(element));
|
|
596
|
+
// Lines with a position; a position line is written only when it changes.
|
|
597
|
+
const placed = items => {
|
|
598
|
+
const out = [];
|
|
599
|
+
let last = null;
|
|
600
|
+
for (const {at, line} of items) {
|
|
601
|
+
if (at !== last) out.push('@ ' + at);
|
|
602
|
+
out.push(line);
|
|
603
|
+
last = at;
|
|
604
|
+
}
|
|
605
|
+
return out;
|
|
606
|
+
};
|
|
607
|
+
|
|
608
|
+
const readers = {
|
|
609
|
+
outline: () => placed(readElements(HEADING_SELECTOR).map(heading => {
|
|
610
|
+
const level = /^H[1-6]$/.test(heading.tagName) ? heading.tagName.toLowerCase()
|
|
611
|
+
: 'h' + (heading.getAttribute('aria-level') || 2);
|
|
612
|
+
return {at: where(heading.getBoundingClientRect()), line: level + ' ' + squash(heading.textContent, 120)};
|
|
613
|
+
}).filter(item => item.line.length > 3)),
|
|
614
|
+
text: () => placed(lines
|
|
615
|
+
.filter(line => !closestDeep(line.parent, NOT_TEXT_SELECTOR) && readable(line.parent))
|
|
616
|
+
.map(line => ({at: where(line.rect), line: squash(line.value, 1000)}))
|
|
617
|
+
.filter(item => item.at !== 'in view')),
|
|
618
|
+
table: () => readElements(TABLE_SELECTOR)
|
|
619
|
+
.filter(table => !table.parentElement?.closest(TABLE_SELECTOR))
|
|
620
|
+
.flatMap((table, index) => {
|
|
621
|
+
const rows = [...table.querySelectorAll('tr,[role="row"]')]
|
|
622
|
+
.filter(row => row.closest(TABLE_SELECTOR) === table && isVisible(row))
|
|
623
|
+
.map(row => [...row.querySelectorAll('th,td,[role="cell"],[role="gridcell"],[role="columnheader"],' +
|
|
624
|
+
'[role="rowheader"]')].filter(cell => cell.closest('tr,[role="row"]') === row)
|
|
625
|
+
.map(cell => squash(cell.innerText, 80)).join(' | '))
|
|
626
|
+
.filter(row => row.replace(/[\s|]/g, ''));
|
|
627
|
+
const caption = squash(table.querySelector('caption')?.textContent || table.getAttribute('aria-label'), 80);
|
|
628
|
+
return ['## table ' + (index + 1) + ' @ ' + where(table.getBoundingClientRect()) +
|
|
629
|
+
(caption ? ': ' + caption : '') + ' (' + rows.length + ' rows)', ...rows];
|
|
630
|
+
}),
|
|
631
|
+
list: () => {
|
|
632
|
+
const items = readElements(LIST_ITEM_SELECTOR)
|
|
633
|
+
.filter(item => !item.parentElement?.closest(LIST_ITEM_SELECTOR) && !item.closest(TABLE_SELECTOR));
|
|
634
|
+
return placed(items.map(item => ({at: where(item.getBoundingClientRect()), line: '- ' + squash(item.innerText, 300)}))
|
|
635
|
+
.filter(item => item.line.length > 2 && item.at !== 'in view'));
|
|
636
|
+
},
|
|
637
|
+
link: () => {
|
|
638
|
+
const seen = new Set();
|
|
639
|
+
return placed(readElements(LINK_SELECTOR).map(link => ({at: where(link.getBoundingClientRect()),
|
|
640
|
+
line: '- ' + squash(accessibleName(link), 80)}))
|
|
641
|
+
.filter(item => item.at !== 'in view' && item.line.length > 2 && !seen.has(item.line) && seen.add(item.line)));
|
|
642
|
+
},
|
|
643
|
+
control: () => placed(readElements(CONTROL_SELECTOR)
|
|
644
|
+
.filter(element => !element.matches(LINK_SELECTOR) && !(element.type === 'hidden'))
|
|
645
|
+
.map(element => {
|
|
646
|
+
const rect = element.getBoundingClientRect();
|
|
647
|
+
return {at: where(rect), line: '- ' + (roleOf(element) || element.tagName.toLowerCase()) + ' ' +
|
|
648
|
+
JSON.stringify(squash(accessibleName(element), 60))};
|
|
649
|
+
})
|
|
650
|
+
.filter(item => item.at !== 'in view')),
|
|
651
|
+
media: () => placed(readElements(MEDIA_SELECTOR).map(element => ({at: where(element.getBoundingClientRect()),
|
|
652
|
+
line: '- ' + squash(element.matches('figcaption') ? element.textContent
|
|
653
|
+
: element.getAttribute('alt') || element.getAttribute('aria-label'), 160)}))
|
|
654
|
+
.filter(item => item.line.length > 2)),
|
|
655
|
+
};
|
|
656
|
+
|
|
657
|
+
// Each kind gets an equal share of the limit; what a small kind leaves goes to the others. Lines are kept
|
|
658
|
+
// whole; only a single line longer than its share is cut. Reports shown and total characters per kind.
|
|
659
|
+
const readLayer = () => {
|
|
660
|
+
if (!readKinds.length) return null;
|
|
661
|
+
const full = readKinds.map(kind => {
|
|
662
|
+
const kindLines = readers[kind]();
|
|
663
|
+
return {kind, lines: kindLines, total: kindLines.reduce((sum, line) => sum + line.length + 1, 0)};
|
|
664
|
+
});
|
|
665
|
+
let remaining = READ_LIMIT;
|
|
666
|
+
const share = new Map();
|
|
667
|
+
[...full].sort((a, b) => a.total - b.total).forEach((entry, index, sorted) => {
|
|
668
|
+
const given = Math.min(entry.total, Math.floor(remaining / (sorted.length - index)));
|
|
669
|
+
share.set(entry.kind, given);
|
|
670
|
+
remaining -= given;
|
|
671
|
+
});
|
|
672
|
+
const chars = {};
|
|
673
|
+
const sections = full.map(({kind, lines: kindLines, total}) => {
|
|
674
|
+
const taken = [];
|
|
675
|
+
let used = 0;
|
|
676
|
+
for (const line of kindLines) {
|
|
677
|
+
if (used + line.length + 1 > share.get(kind)) {
|
|
678
|
+
if (!taken.length && share.get(kind) > 1) taken.push(line.slice(0, share.get(kind) - 1));
|
|
679
|
+
break;
|
|
680
|
+
}
|
|
681
|
+
taken.push(line);
|
|
682
|
+
used += line.length + 1;
|
|
683
|
+
}
|
|
684
|
+
const shown = taken.reduce((sum, line) => sum + line.length + 1, 0);
|
|
685
|
+
chars[kind] = [shown, total];
|
|
686
|
+
const more = total > shown ? ['... ' + (total - shown) + ' more characters'] : [];
|
|
687
|
+
return ['# ' + kind + (kindLines.length ? '' : ' (none)'), ...taken, ...more].join('\n');
|
|
688
|
+
});
|
|
689
|
+
return {kinds: readKinds, text: sections.join('\n'), chars, limit: READ_LIMIT};
|
|
690
|
+
};
|
|
691
|
+
|
|
499
692
|
// ---- Snapshot ----
|
|
500
693
|
|
|
501
694
|
const actions = collectActions();
|
|
@@ -534,6 +727,25 @@
|
|
|
534
727
|
region.shown_actions++;
|
|
535
728
|
action.region = region.key;
|
|
536
729
|
}
|
|
730
|
+
// v2: the nearest elements beyond the viewport, back in DOM order, after those in view.
|
|
731
|
+
const beyondKept = beyond
|
|
732
|
+
.map((entry, index) => ({...entry, index}))
|
|
733
|
+
.filter(({action}) => !viewing || decide(actionRegions.get(action)).view !== 'skip')
|
|
734
|
+
.sort((a, b) => a.distance - b.distance || a.index - b.index)
|
|
735
|
+
.slice(0, MAX_BEYOND)
|
|
736
|
+
.sort((a, b) => a.index - b.index)
|
|
737
|
+
.map(({action}) => action);
|
|
738
|
+
for (const action of beyondKept) {
|
|
739
|
+
if (!(action.node in guards)) guards[action.node] = state.guard(state.nodes.get(action.node));
|
|
740
|
+
action.region = actionRegions.get(action).key;
|
|
741
|
+
}
|
|
742
|
+
// The nearest and the farthest are kept: the end of a list is what loads more of it.
|
|
743
|
+
const kept = reveals.length > MAX_REVEALS
|
|
744
|
+
? [...reveals.slice(0, MAX_REVEALS / 2), ...reveals.slice(-MAX_REVEALS / 2)] : reveals;
|
|
745
|
+
for (const reveal of kept) {
|
|
746
|
+
if (!(reveal.node in guards)) guards[reveal.node] = state.guard(state.nodes.get(reveal.node));
|
|
747
|
+
}
|
|
748
|
+
chosen = [...chosen, ...beyondKept, ...kept];
|
|
537
749
|
chosen.forEach((action, index) => action.id = 'e' + (index + 1));
|
|
538
750
|
if (scrollY + innerHeight < height - 2) {
|
|
539
751
|
chosen.push({id: 'scroll_down', kind: 'scroll', label: 'Scroll down', delta: SCROLL_STEP});
|
|
@@ -554,8 +766,11 @@
|
|
|
554
766
|
});
|
|
555
767
|
|
|
556
768
|
state.region = element => regionOf(element).key;
|
|
769
|
+
const read = readLayer();
|
|
557
770
|
|
|
558
771
|
return {url: location.href, title: document.title, w: innerWidth, h: innerHeight, text: shown.text,
|
|
559
772
|
scroll: {y: scrollY, height}, actions: chosen, marker, page_key, guards, omitted_actions,
|
|
560
|
-
regions: regionSummary, omitted_regions: outline.length - regionSummary.length}
|
|
773
|
+
regions: regionSummary, omitted_regions: outline.length - regionSummary.length, ...(read ? {read} : {}),
|
|
774
|
+
...(reveals.length > kept.length ? {omitted_reveals: reveals.length - kept.length} : {}),
|
|
775
|
+
...(beyond.length > beyondKept.length ? {omitted_beyond: beyond.length - beyondKept.length} : {})};
|
|
561
776
|
})
|
package/dist/agent.js
CHANGED
|
@@ -5,7 +5,7 @@
|
|
|
5
5
|
*/
|
|
6
6
|
import { quote } from "./format.js";
|
|
7
7
|
import { pageEvent, safeUrl } from "./metrics.js";
|
|
8
|
-
import { MutationUncertain, PolicyError, StalePage, } from "./types.js";
|
|
8
|
+
import { Decision, MutationUncertain, PolicyError, StalePage, } from "./types.js";
|
|
9
9
|
// Why the loop handed control back to Claude.
|
|
10
10
|
export const GOAL_REPORTED = "GOAL_REPORTED"; // Jev chose DONE. Claude checks the page; finish verifies.
|
|
11
11
|
export const BLOCKED = "BLOCKED";
|
|
@@ -13,6 +13,7 @@ export const STUCK = "STUCK"; // Three actions in a row did not change the page.
|
|
|
13
13
|
export const ATTEMPT_BUDGET = "ATTEMPT_BUDGET";
|
|
14
14
|
export const NEEDS_HUMAN = "NEEDS_HUMAN"; // A person acts in the window: sign-in, codes, CAPTCHA, [person] steps.
|
|
15
15
|
export const NEEDS_VALUE = "NEEDS_VALUE"; // A field needs text and none of Claude's values fits.
|
|
16
|
+
export const ESCALATE = "ESCALATE"; // v2: Jev handed the decision to Claude. `answer` continues the same attempt.
|
|
16
17
|
export const JEV_FAILED = "JEV_FAILED"; // No usable decision. Nothing was executed.
|
|
17
18
|
export const UNCERTAIN = "UNCERTAIN"; // A mutation may have happened. Never retried, never replanned.
|
|
18
19
|
/** Per attempt. A replan starts a new attempt with a fresh budget. */
|
|
@@ -28,15 +29,85 @@ export class Budget {
|
|
|
28
29
|
}
|
|
29
30
|
const MISSING = "missing"; // A history entry for a NOT_VISIBLE choice: nothing ran, but Jev must see it.
|
|
30
31
|
const STALE = "stale"; // A history entry for a choice the browser refused as stale: nothing ran.
|
|
31
|
-
/**
|
|
32
|
+
/** What the observation offered when Jev found the element it needed missing: counts for the run log only. */
|
|
33
|
+
function offered(page) {
|
|
34
|
+
const actions = page.actions ?? [];
|
|
35
|
+
const scroll = page.scroll ?? {};
|
|
36
|
+
return {
|
|
37
|
+
in_view: actions.filter((a) => a.kind !== "reveal").length,
|
|
38
|
+
offscreen: actions.filter((a) => a.kind === "reveal").length,
|
|
39
|
+
omitted_actions: page.omitted_actions ?? 0,
|
|
40
|
+
omitted_reveals: page.omitted_reveals ?? 0,
|
|
41
|
+
regions_with_offscreen: (page.regions ?? []).filter((r) => r.offscreen_actions)
|
|
42
|
+
.length,
|
|
43
|
+
below_px: Math.max(0, Math.round((scroll.height ?? 0) - (scroll.y ?? 0) - (page.h ?? 0))),
|
|
44
|
+
};
|
|
45
|
+
}
|
|
46
|
+
/** Outcomes after which the log keeps a page check, to tell a bot check or an error page from a plan problem. */
|
|
47
|
+
export const CHECKED_OUTCOMES = [BLOCKED, STUCK, ATTEMPT_BUDGET, NEEDS_HUMAN];
|
|
48
|
+
/** Logs a page_check event. Read-only, never sent to Jev; a failed check is logged and never stops the run. */
|
|
49
|
+
export async function checkPage(browser, recorder, when, attempt, url) {
|
|
50
|
+
if (!browser.check)
|
|
51
|
+
return;
|
|
52
|
+
try {
|
|
53
|
+
recorder.event("page_check", { when, attempt, url: safeUrl(url), ...(await browser.check()) });
|
|
54
|
+
}
|
|
55
|
+
catch (error) {
|
|
56
|
+
const message = error instanceof Error ? error.message : String(error);
|
|
57
|
+
recorder.event("page_check_failed", { when, attempt, error: message });
|
|
58
|
+
}
|
|
59
|
+
}
|
|
60
|
+
/**
|
|
61
|
+
* Opt-in with WEBHAND_SCREENSHOTS=1: the harness, not a model, captures the page after each action so an
|
|
62
|
+
* outside judge can check the run. Never sent to Jev. A failed capture is logged and never stops the run.
|
|
63
|
+
*/
|
|
64
|
+
export async function captureScreenshot(browser, recorder, step, attempt, after, url) {
|
|
65
|
+
if (process.env.WEBHAND_SCREENSHOTS !== "1" || !browser.screenshot)
|
|
66
|
+
return;
|
|
67
|
+
try {
|
|
68
|
+
recorder.screenshot(step, attempt, after, await browser.screenshot(), url);
|
|
69
|
+
}
|
|
70
|
+
catch (error) {
|
|
71
|
+
const message = error instanceof Error ? error.message : String(error);
|
|
72
|
+
recorder.event("screenshot_failed", { step, attempt, error: message });
|
|
73
|
+
}
|
|
74
|
+
}
|
|
75
|
+
/** Executed actions only, in every attempt: not a person's step, a NOT_VISIBLE choice, or a refused stale one. */
|
|
76
|
+
export function executed(history) {
|
|
77
|
+
return history.filter((h) => !["human", "answer", MISSING, STALE].includes(h.kind));
|
|
78
|
+
}
|
|
79
|
+
/** Executed actions of one attempt; they count against the budget. */
|
|
32
80
|
export function actionsIn(history, attempt) {
|
|
33
|
-
return history.filter((h) => (h.attempt ?? 1) === attempt
|
|
81
|
+
return executed(history).filter((h) => (h.attempt ?? 1) === attempt);
|
|
34
82
|
}
|
|
35
83
|
/** Three entries without progress: unchanged pages, targets Jev could not see, or refused stale choices. */
|
|
36
84
|
function stuck(history, attempt = 1) {
|
|
37
85
|
const recent = history.filter((h) => (h.attempt ?? 1) === attempt && h.kind !== "human").slice(-3);
|
|
38
86
|
return recent.length === 3 && recent.every((s) => s.page_changed === false && s.kind !== "wait");
|
|
39
87
|
}
|
|
88
|
+
/** Entries of this attempt after Claude's last answer: an answer starts the count again, so Jev is asked before any stop. */
|
|
89
|
+
function sinceAnswer(history, attempt) {
|
|
90
|
+
const own = history.filter((h) => (h.attempt ?? 1) === attempt && h.kind !== "human");
|
|
91
|
+
const last = own.map((h) => h.kind).lastIndexOf("answer");
|
|
92
|
+
return own.slice(last + 1);
|
|
93
|
+
}
|
|
94
|
+
/** v2: back and forth scrolling (three round trips) with nothing else done between. Each scroll changes the page, so stuck() never sees it. */
|
|
95
|
+
export function looping(history, attempt = 1) {
|
|
96
|
+
const recent = sinceAnswer(history, attempt).slice(-6);
|
|
97
|
+
const scrolls = recent.map((h) => h.operation);
|
|
98
|
+
return (recent.length === 6 &&
|
|
99
|
+
scrolls.every((op, i) => (op === "SCROLL_DOWN" || op === "SCROLL_UP") && (i === 0 || op !== scrolls[i - 1])));
|
|
100
|
+
}
|
|
101
|
+
/** How many frames, canvases, and plugin objects the page has: parts webhand cannot see into. */
|
|
102
|
+
export function unseenParts(page) {
|
|
103
|
+
const unseen = (page.status?.unseen ?? {});
|
|
104
|
+
return (unseen.frames ?? 0) + (unseen.canvases ?? 0) + (unseen.objects ?? 0);
|
|
105
|
+
}
|
|
106
|
+
/** v2: the needed element was not offered twice in a row, on a page with parts webhand cannot see into. */
|
|
107
|
+
function missingInUnseen(history, page, attempt = 1) {
|
|
108
|
+
const recent = sinceAnswer(history, attempt).slice(-2);
|
|
109
|
+
return recent.length === 2 && recent.every((h) => h.kind === MISSING) && unseenParts(page) > 0;
|
|
110
|
+
}
|
|
40
111
|
/** A step for the run log: the value's label instead of the typed text, and the URL without query values. */
|
|
41
112
|
function logged(step, region, values) {
|
|
42
113
|
const { text, url, ...rest } = step;
|
|
@@ -67,18 +138,53 @@ export async function execute(browser, page, decision, history, recorder, option
|
|
|
67
138
|
history.push(step);
|
|
68
139
|
recorder.event("step", logged(step, action.region, options.values ?? []));
|
|
69
140
|
onChange(page);
|
|
141
|
+
const asked = browser.ignoredWindows ? await browser.ignoredWindows().catch(() => []) : [];
|
|
70
142
|
const newPage = await browser.observe();
|
|
71
|
-
step.page_changed = !
|
|
143
|
+
step.page_changed = !samePage(newPage, page);
|
|
72
144
|
recorder.event("step_result", { step: step.step, page_changed: step.page_changed });
|
|
145
|
+
options.knowledge?.record({
|
|
146
|
+
from: page.url,
|
|
147
|
+
to: newPage.url,
|
|
148
|
+
kind: action.kind,
|
|
149
|
+
label: action.label,
|
|
150
|
+
changed: step.page_changed,
|
|
151
|
+
inItem: page.guards?.[String(action.node)]?.[14] === "item",
|
|
152
|
+
});
|
|
153
|
+
if (asked.length) {
|
|
154
|
+
recorder.event("window_ignored", { step: step.step, attempt: options.attempt, urls: asked.map(safeUrl) });
|
|
155
|
+
}
|
|
73
156
|
if (step.page_changed)
|
|
74
157
|
recorder.event("page", pageEvent(newPage, options.attempt));
|
|
158
|
+
await captureScreenshot(browser, recorder, step.step, options.attempt, "action", newPage.url);
|
|
75
159
|
onChange(newPage);
|
|
76
160
|
return newPage;
|
|
77
161
|
}
|
|
162
|
+
/**
|
|
163
|
+
* v2: Jev chose an element beyond the viewport. It is scrolled into view (a scroll, never an input), the page
|
|
164
|
+
* is observed again, and the same element with the same operation is taken from that observation. When it is
|
|
165
|
+
* no longer offered, nothing runs: StalePage, as for any refused action.
|
|
166
|
+
*/
|
|
167
|
+
async function intoView(browser, page, decision, recorder, attempt) {
|
|
168
|
+
const action = decision.action;
|
|
169
|
+
await browser.act({ kind: "reveal", node: action.node, label: action.label }, page);
|
|
170
|
+
const next = await browser.observe();
|
|
171
|
+
const same = (next.actions ?? []).find((a) => !a.offscreen && a.node === action.node && a.kind === action.kind && (a.value ?? null) === (action.value ?? null));
|
|
172
|
+
recorder.event("into_view", { attempt, at: action.at ?? null, found: Boolean(same) });
|
|
173
|
+
if (!same)
|
|
174
|
+
throw new StalePage("Target was not in view after scrolling to it", ["target_unreachable"]);
|
|
175
|
+
const chosen = new Decision(decision.operation, same, { confidence: decision.confidence, text: decision.text });
|
|
176
|
+
return { page: next, decision: chosen };
|
|
177
|
+
}
|
|
78
178
|
/** Markers are JSON values from the page; equal content means the same page. */
|
|
79
179
|
function sameMarker(a, b) {
|
|
80
180
|
return JSON.stringify(a) === JSON.stringify(b);
|
|
81
181
|
}
|
|
182
|
+
/** The same page in the same tab setup and frames: a new or closed tab, a switch, or a change inside a frame is a change even when the page is not. */
|
|
183
|
+
function samePage(a, b) {
|
|
184
|
+
return (sameMarker(a.marker, b.marker) &&
|
|
185
|
+
sameMarker(a.tabs ?? null, b.tabs ?? null) &&
|
|
186
|
+
sameMarker(a.frame_markers ?? null, b.frame_markers ?? null));
|
|
187
|
+
}
|
|
82
188
|
/** Runs one attempt until an outcome above. history is extended in place. */
|
|
83
189
|
export async function runAgent(browser, policy, plan, recorder, budget = new Budget(), options = {}) {
|
|
84
190
|
const attempt = plan.attempt ?? 1;
|
|
@@ -105,6 +211,12 @@ export async function runAgent(browser, policy, plan, recorder, budget = new Bud
|
|
|
105
211
|
while (true) {
|
|
106
212
|
if (stuck(history, attempt))
|
|
107
213
|
return finish(STUCK, "three actions without a page change");
|
|
214
|
+
if (plan.flow === "v2" && looping(history, attempt)) {
|
|
215
|
+
return finish(STUCK, "scrolled up and down three times with nothing else done");
|
|
216
|
+
}
|
|
217
|
+
if (plan.flow === "v2" && missingInUnseen(history, page, attempt)) {
|
|
218
|
+
return finish(STUCK, "the needed element was not offered twice, and the page has parts webhand cannot see");
|
|
219
|
+
}
|
|
108
220
|
if (actionsIn(history, attempt).length >= budget.maxActions || decisions >= budget.maxDecisions) {
|
|
109
221
|
return finish(ATTEMPT_BUDGET, `${actionsIn(history, attempt).length} actions, ${decisions} decisions`);
|
|
110
222
|
}
|
|
@@ -115,7 +227,11 @@ export async function runAgent(browser, policy, plan, recorder, budget = new Bud
|
|
|
115
227
|
page = await browser.observe();
|
|
116
228
|
decisions += 1;
|
|
117
229
|
onDecision(decisions);
|
|
118
|
-
|
|
230
|
+
// v2: what worked before on this kind of page, when the site is known (stage 3).
|
|
231
|
+
const known = options.knowledge?.hints(page.url) ?? [];
|
|
232
|
+
if (known.length)
|
|
233
|
+
recorder.event("known_offered", { attempt, count: known.length });
|
|
234
|
+
decision = await policy.choose(known.length ? { ...page, known } : page, plan, history);
|
|
119
235
|
flushCalls();
|
|
120
236
|
if (decision.operation === "NOT_VISIBLE") {
|
|
121
237
|
const entry = {
|
|
@@ -129,14 +245,21 @@ export async function runAgent(browser, policy, plan, recorder, budget = new Bud
|
|
|
129
245
|
page_changed: false,
|
|
130
246
|
};
|
|
131
247
|
history.push(entry);
|
|
132
|
-
recorder.event("not_visible", {
|
|
248
|
+
recorder.event("not_visible", {
|
|
249
|
+
attempt,
|
|
250
|
+
wanted: decision.wanted,
|
|
251
|
+
confidence: decision.confidence,
|
|
252
|
+
offered: offered(page),
|
|
253
|
+
...(browser.unseen ? { unseen: await browser.unseen().catch(() => null) } : {}),
|
|
254
|
+
...(page.frames_log ? { frames: page.frames_log } : {}),
|
|
255
|
+
});
|
|
133
256
|
onChange(page);
|
|
134
257
|
continue;
|
|
135
258
|
}
|
|
136
259
|
if (decision.action === null) {
|
|
137
260
|
if (!(await browser.fresh(page)))
|
|
138
261
|
continue;
|
|
139
|
-
const outcome = { DONE: GOAL_REPORTED, NEEDS_HUMAN }[decision.operation] ?? BLOCKED;
|
|
262
|
+
const outcome = { DONE: GOAL_REPORTED, NEEDS_HUMAN, ESCALATE }[decision.operation] ?? BLOCKED;
|
|
140
263
|
return finish(outcome, `Jev chose ${decision.operation}`);
|
|
141
264
|
}
|
|
142
265
|
if (decision.action.kind === "fill" && decision.text === null) {
|
|
@@ -145,11 +268,14 @@ export async function runAgent(browser, policy, plan, recorder, budget = new Bud
|
|
|
145
268
|
field[key] = decision.action[key] ?? null;
|
|
146
269
|
return finish(NEEDS_VALUE, "no value from the plan fits this field", { field });
|
|
147
270
|
}
|
|
271
|
+
if (decision.action.offscreen)
|
|
272
|
+
({ page, decision } = await intoView(browser, page, decision, recorder, attempt));
|
|
148
273
|
page = await execute(browser, page, decision, history, recorder, {
|
|
149
274
|
decidedBy: policy.name,
|
|
150
275
|
attempt,
|
|
151
276
|
onChange,
|
|
152
277
|
values: plan.values,
|
|
278
|
+
knowledge: options.knowledge,
|
|
153
279
|
});
|
|
154
280
|
}
|
|
155
281
|
catch (error) {
|
|
@@ -167,7 +293,7 @@ export async function runAgent(browser, policy, plan, recorder, budget = new Bud
|
|
|
167
293
|
text: null,
|
|
168
294
|
url: observed.url,
|
|
169
295
|
// Unchanged: the same choice would be refused again, so it counts toward STUCK.
|
|
170
|
-
page_changed: !
|
|
296
|
+
page_changed: !samePage(page, observed),
|
|
171
297
|
};
|
|
172
298
|
history.push(entry);
|
|
173
299
|
recorder.event("stale", {
|
|
@@ -175,13 +301,14 @@ export async function runAgent(browser, policy, plan, recorder, budget = new Bud
|
|
|
175
301
|
wanted: decision.operation,
|
|
176
302
|
target: decision.action.label,
|
|
177
303
|
reason: error.message,
|
|
304
|
+
changed: error.changed,
|
|
178
305
|
page_changed: entry.page_changed,
|
|
179
306
|
});
|
|
180
307
|
}
|
|
181
308
|
onChange(page);
|
|
182
309
|
}
|
|
183
310
|
else if (error instanceof PolicyError) {
|
|
184
|
-
return finish(JEV_FAILED, error.message);
|
|
311
|
+
return finish(JEV_FAILED, error.message, error.diagnostic ? { jev: error.diagnostic } : {});
|
|
185
312
|
}
|
|
186
313
|
else if (error instanceof MutationUncertain) {
|
|
187
314
|
return finish(UNCERTAIN, `mutation uncertain: ${error.message}`);
|