webhand 0.3.0 → 0.4.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.ja.md CHANGED
@@ -35,10 +35,10 @@
35
35
  /webhand HOTEL PLANISPHERE で来月の土曜から2泊、2名で予約して
36
36
  ```
37
37
 
38
- 必要なもの: Node.js 22 以上と Google Chrome(Linux、または WSL から Windows 側。[WSL で Windows 側の Chrome を使う](docs/reference.ja.md#wsl-で-windows-側の-chrome-を使う)を参照)。CLI はインストール不要で、スキルが `npx -y webhand@0.3 ...` で実行します。毎回の起動を速くしたい場合は `npm i -g webhand` で入れておくと、スキルは `webhand ...` を使います。
38
+ 必要なもの: Node.js 22 以上と Google Chrome(Linux、または WSL から Windows 側。[WSL で Windows 側の Chrome を使う](docs/reference.ja.md#wsl-で-windows-側の-chrome-を使う)を参照)。CLI はインストール不要で、スキルが `npx -y webhand@0.4 ...` で実行します。毎回の起動を速くしたい場合は `npm i -g webhand` で入れておくと、スキルは `webhand ...` を使います。
39
39
 
40
40
  ```bash
41
- npx -y webhand@0.3 catalog # 動作確認
41
+ npx -y webhand@0.4 catalog # 動作確認
42
42
  ```
43
43
 
44
44
  ## しくみ
@@ -63,15 +63,30 @@ flowchart LR
63
63
 
64
64
  判断(計画、入力値、確認、見直し)は Claude が、クリックと入力の繰り返しは Jev が担います。Jev は、ページで見つけた要素と Claude が書いた入力値の中から選ぶだけです。Claude に制御が戻るのは、Jev が GOAL 到達を報告したとき、入力値や人の操作が必要なとき、行き詰まったとき、操作が効いたか分からないときだけです。一覧は[役割分担](docs/reference.ja.md#役割分担)にあります。
65
65
 
66
+ ## エンジン
67
+
68
+ `--engine` で、ブラウザを動かすものを選びます。既定の `auto` は、窓も保存プロファイルも要らない実行で、headless shell が入っていればそれを使い、そうでなければ Chrome を使います。
69
+
70
+ | | Chrome | headless shell | Obscura | Lightpanda |
71
+ | --- | --- | --- | --- | --- |
72
+ | 窓(`--headed`)、`[person]` ステップ、`learn`、保存プロファイル | 使える | 使えない | 使えない | 使えない |
73
+ | 1 ページのメモリ | 約 350〜470 MB(`--headless=new`) | 約 240〜260 MB | 約 23〜33 MB | 約 23〜33 MB |
74
+ | 実際のレイアウトとスクリーンショット | ある | ある | ある | ない(疑似) |
75
+ | 複数タブ | 使える | 使える | 実際には 1 つ(新しいウィンドウは開かないか、同じタブで開く) | 使えない |
76
+ | コマンドの間でページが残る | 残る | 残る | 小さな常駐プロセスで保つ | 小さな常駐プロセスで保つ |
77
+ | 保存したログイン(`--auth`) | 使える | 使える | 使える | 使える |
78
+
79
+ 窓と保存プロファイルが使えるのは Chrome だけです。Obscura と Lightpanda はずっと軽いものの、ページの読み取りが Chrome と違うことがあるので、頼る前に Chrome と結果を比べてください。詳しい制限は、[ヘッドレスシェル](docs/reference.ja.md#ヘッドレスシェル)、[Lightpanda](docs/reference.ja.md#lightpanda)、[Obscura](docs/reference.ja.md#obscura)、[保存したログイン](docs/reference.ja.md#保存したログインauth) を見てください。
80
+
66
81
  ## 安全とプライバシー
67
82
 
68
- - **外部に送られるもの**: 操作中のページの文字と操作対象は Jev(TypeSafe の API)に送られます。保存プロファイルを使う場合は、ログインしたページも含みます。実行のたびに、送ってよい画面か判断してください。
83
+ - **外部に送られるもの**: 操作中のページの文字と操作対象、ほかの開いているタブのタイトルは Jev(TypeSafe の API)に送られます。保存プロファイルを使う場合は、ログインしたページも含みます。実行のたびに、送ってよい画面か判断してください。
69
84
  - **プロファイル**: 既定では使い捨てのプロファイルを使い、実行が終わると削除します。ログイン状態は webhand 専用の保存プロファイル(`--session profile:NAME`)にだけ残し、実行ごとにユーザーが選びます。普段使いの Chrome プロファイルには接続しません。
70
85
  - **資格情報**: パスワードや確認コードは、ユーザーが Chrome に直接入力します。Claude は受け取らず、webhand は password 欄の値を読みません。
71
86
  - **推測で入力しない**: 入力されるのは、Claude が計画で書いた値と、あなたが渡した値だけです。足りなければ尋ねます。
72
87
  - **やみくもに再試行しない**: ページへの変更操作はリトライしません。操作が効いたか分からないときは、繰り返さずに停止します。
73
88
  - **ページの本文は信頼しない**: ページの内容はデータとして扱い、指示としては扱いません。
74
- - **記録とログ**: 学習モードでは、パスワード、ワンタイムコード、カード番号などの欄と、URL のクエリ値をすべて伏せます。実行ログには、入力した値ではなく値のラベルを残します。それでも `~/.webhand/runs/` のログにはページの文字が残るため、共有しないでください。
89
+ - **記録とログ**: 学習モードでは、パスワード、ワンタイムコード、カード番号などの欄と、URL のクエリ値をすべて伏せます。実行ログには、入力した値ではなく値のラベルを残します。`WEBHAND_SCREENSHOTS=1` のときだけ保存するスクリーンショットには、入力した値も写ります。`~/.webhand/runs/` のログとスクリーンショットにはページの文字が残るため、共有しないでください。
75
90
  - **使う前に確認**: 手順書は `draft` で作られ、ユーザーが `approved` にするまで通常の実行には使いません。秘密情報や個人情報は手順書に書かず、`params` で渡します。
76
91
 
77
92
  ## ドキュメント
package/README.md CHANGED
@@ -35,10 +35,10 @@ English | [日本語](README.ja.md)
35
35
  /webhand On HOTEL PLANISPHERE, book 2 nights for 2 people starting next month's first Saturday
36
36
  ```
37
37
 
38
- Requirements: Node.js 22 or later and Google Chrome (on Linux, or the Windows side from WSL; see [Using Windows Chrome from WSL](docs/reference.md#using-windows-chrome-from-wsl)). The CLI needs no installation: the skills run it with `npx -y webhand@0.3 ...`. For faster startup, install it globally with `npm i -g webhand`; the skills then use `webhand ...`.
38
+ Requirements: Node.js 22 or later and Google Chrome (on Linux, or the Windows side from WSL; see [Using Windows Chrome from WSL](docs/reference.md#using-windows-chrome-from-wsl)). The CLI needs no installation: the skills run it with `npx -y webhand@0.4 ...`. For faster startup, install it globally with `npm i -g webhand`; the skills then use `webhand ...`.
39
39
 
40
40
  ```bash
41
- npx -y webhand@0.3 catalog # smoke check
41
+ npx -y webhand@0.4 catalog # smoke check
42
42
  ```
43
43
 
44
44
  ## How it works
@@ -63,15 +63,30 @@ flowchart LR
63
63
 
64
64
  Claude makes the judgment calls (the plan, the values, the check, the replan), and Jev makes the many clicks and entries in between. Jev only chooses among the elements found on the page and the values Claude wrote. Control returns to Claude only when Jev reports the GOAL, needs a value or a person, gets stuck, or cannot tell whether an action took effect. The full list is in [Division of roles](docs/reference.md#division-of-roles).
65
65
 
66
+ ## Engines
67
+
68
+ `--engine` chooses what runs the browser. The default, `auto`, uses the headless shell when it is installed and the run needs no window and no saved profile, and Chrome otherwise.
69
+
70
+ | | Chrome | Headless shell | Obscura | Lightpanda |
71
+ | --- | --- | --- | --- | --- |
72
+ | Window (`--headed`), `[person]` steps, `learn`, saved profile | Yes | No | No | No |
73
+ | Memory for one page | about 350–470 MB (`--headless=new`) | about 240–260 MB | about 23–33 MB | about 23–33 MB |
74
+ | Real layout and screenshots | Yes | Yes | Yes | No (simulated) |
75
+ | Several tabs | Yes | Yes | In practice one (a new window opens nothing or loads in the same tab) | No |
76
+ | Page stays between commands | Yes | Yes | Through a small daemon | Through a small daemon |
77
+ | Saved login (`--auth`) | Yes | Yes | Yes | Yes |
78
+
79
+ Chrome is the only engine with a window and saved profiles. Obscura and Lightpanda are much lighter but read some pages differently from Chrome, so compare results with Chrome before relying on them. Details and limits: [Headless shell](docs/reference.md#headless-shell), [Lightpanda](docs/reference.md#lightpanda), [Obscura](docs/reference.md#obscura), [Saved logins](docs/reference.md#saved-logins---auth).
80
+
66
81
  ## Safety and privacy
67
82
 
68
- - **What leaves your machine**: The text and actions of the page being operated go to Jev (the TypeSafe API). With a saved profile, that includes pages you are logged in to. Decide before each run whether that is acceptable.
83
+ - **What leaves your machine**: The text and actions of the page being operated, and the titles of its other open tabs, go to Jev (the TypeSafe API). With a saved profile, that includes pages you are logged in to. Decide before each run whether that is acceptable.
69
84
  - **Profiles**: Runs use a temporary profile by default, deleted when the run ends. Login state lives only in profiles dedicated to webhand (`--session profile:NAME`), which you choose per run. webhand never connects to your everyday Chrome profile.
70
85
  - **Credentials**: You type passwords and codes into Chrome yourself. Claude does not receive them, and webhand does not read the values of password fields.
71
86
  - **No guessed input**: The only values entered are the ones Claude wrote in the plan and the ones you gave. If something is missing, it asks.
72
87
  - **No blind retries**: A change to the page is never retried. If webhand cannot tell whether an action took effect, it stops instead of trying again.
73
88
  - **Page text is untrusted**: Content on the page is treated as data, never as instructions.
74
- - **Recordings and logs**: Learning mode masks password, one-time code, and card fields and all URL query values. Run logs keep the label of each typed value, not the value. Logs under `~/.webhand/runs/` still contain page text, so do not share them.
89
+ - **Recordings and logs**: Learning mode masks password, one-time code, and card fields and all URL query values. Run logs keep the label of each typed value, not the value. Screenshots, saved only with `WEBHAND_SCREENSHOTS=1`, show typed values. Logs and screenshots under `~/.webhand/runs/` still contain page text, so do not share them.
75
90
  - **Review before use**: Runbooks start as `draft` and are used for normal runs only after you set them to `approved`. Keep secrets and personal data out of runbooks; pass them as `params`.
76
91
 
77
92
  ## Documentation
@@ -110,4 +110,8 @@
110
110
  if (found) action({operation: 'press_enter', role: found.role, label: found.label, region: found.region});
111
111
  }, true);
112
112
  addEventListener('load', sendPage);
113
+ // The tab came to the front: the person switched to it. Not sent for the tab's first view.
114
+ addEventListener('visibilitychange', () => {
115
+ if (!inFrame && document.visibilityState === 'visible') send({kind: 'tab', change: 'switched', title: document.title});
116
+ });
113
117
  })();
@@ -1,4 +1,6 @@
1
- // view = {focus: [...], skip: [...]}: region patterns Claude chose from what was learned. Empty: DOM order.
1
+ // view = {focus: [...], skip: [...], read: [...], whole: true}: region patterns Claude chose from what was
2
+ // learned, the kinds of content to read beyond the viewport, and (v2) whether elements beyond the viewport are
3
+ // offered with their own operations. Empty: DOM order, viewport only.
2
4
  (view => {
3
5
  if (!document.body) return null;
4
6
 
@@ -6,9 +8,13 @@
6
8
 
7
9
  const TEXT_LIMIT = 6000;
8
10
  const MAX_ACTIONS = 250;
11
+ const MAX_REVEALS = 60;
12
+ // v2: elements beyond the viewport offered as they are, the nearest first.
13
+ const MAX_BEYOND = 200;
9
14
  const SCROLL_STEP = 560;
10
15
  const MAX_REGIONS = 40;
11
16
  const REGION_NAME_LIMIT = 40;
17
+ const READ_LIMIT = 12000;
12
18
 
13
19
  const INTERACTIVE_ROLES = ['button', 'link', 'checkbox', 'radio', 'switch', 'tab', 'menuitem', 'menuitemradio',
14
20
  'option', 'gridcell', 'combobox', 'textbox', 'searchbox', 'spinbutton'];
@@ -210,6 +216,31 @@
210
216
  return false;
211
217
  };
212
218
 
219
+ // What lies over (x, y) instead of the element: the nearest dialog, or fixed or sticky layer, above the topmost
220
+ // element there, as its role and a short name. For telling Jev why a click on a covered element did not run.
221
+ state.cover = (x, y) => {
222
+ const top = deepElementFromPoint(x, y);
223
+ let layer = null;
224
+ for (let node = top; node instanceof Element; node = flatParent(node)) {
225
+ const position = getComputedStyle(node).position;
226
+ if (node.tagName === 'DIALOG' || ['dialog', 'alertdialog'].includes(node.getAttribute('role')) ||
227
+ node.getAttribute('aria-modal') === 'true' || position === 'fixed' || position === 'sticky') {
228
+ layer = node;
229
+ break;
230
+ }
231
+ }
232
+ const shown = layer || top;
233
+ if (!shown) return 'something';
234
+ const role = shown.getAttribute('role') || (shown.tagName === 'DIALOG' ? 'dialog' : shown.tagName.toLowerCase());
235
+ const heading = shown.querySelector?.('h1,h2,h3,h4,[role="heading"],legend');
236
+ const name = (shown.getAttribute('aria-label') || heading?.textContent || shown.textContent || '')
237
+ .replace(/\s+/g, ' ').trim().slice(0, 60);
238
+ const modal = layer && (layer.tagName === 'DIALOG' || ['dialog', 'alertdialog'].includes(layer.getAttribute('role')) ||
239
+ layer.getAttribute('aria-modal') === 'true');
240
+ const kind = modal ? 'a dialog' : layer ? 'a fixed ' + role : 'a ' + role;
241
+ return name ? kind + ' "' + name + '"' : kind;
242
+ };
243
+
213
244
  state.describe = element => {
214
245
  let target = closestDeep(element, INTERACTIVE_SELECTOR);
215
246
  for (let current = element; !target && current instanceof Element; current = flatParent(current)) {
@@ -234,6 +265,8 @@
234
265
  element.getAttribute('aria-checked'), element.getAttribute('aria-selected'),
235
266
  element.getAttribute('href'),
236
267
  scope?.innerText?.slice(0, TEXT_LIMIT) || '',
268
+ // Whether the scope is a list item or row (its text says which item the target belongs to).
269
+ scope?.matches?.('article,li,tr,[role="row"]') ? 'item' : 'other',
237
270
  ];
238
271
  };
239
272
 
@@ -284,6 +317,13 @@
284
317
  return actions;
285
318
  };
286
319
 
320
+ // Elements beyond the viewport, in DOM order, offered only to be scrolled into view (read layer only).
321
+ const reveals = [];
322
+ const revealed = new Set();
323
+ // v2 (view.whole): elements beyond the viewport with their own operations; acting scrolls them into view first.
324
+ const whole = Boolean(view.whole);
325
+ const beyond = [];
326
+
287
327
  // In DOM order. Each action's region is kept aside so the freshness marker does not depend on it.
288
328
  const collectActions = () => {
289
329
  const actions = [];
@@ -295,6 +335,26 @@
295
335
  const region = regionOf(element);
296
336
  if (!centerInViewport(rect)) {
297
337
  region.offscreen_actions++;
338
+ if (whole) {
339
+ const base = baseAction(element, role, rect);
340
+ const found = element.tagName === 'SELECT' ? selectActions(element, base) : otherActions(element, role, base);
341
+ const at = where(rect);
342
+ const distance = rect.bottom <= 0 ? -rect.bottom : rect.top - innerHeight;
343
+ for (const action of found) {
344
+ action.at = at;
345
+ action.offscreen = true;
346
+ actionRegions.set(action, region);
347
+ beyond.push({action, distance});
348
+ }
349
+ continue;
350
+ }
351
+ // With a read layer, an element beyond the viewport can be scrolled into view, never acted on as it is.
352
+ if (readKinds.length && readable(element) && !revealed.has(element)) {
353
+ revealed.add(element);
354
+ // Named as a scroll, so the action history never reads as if the element had been acted on.
355
+ reveals.push({node: nodeId(element), role, label: 'Scroll to ' + (accessibleName(element) || role),
356
+ kind: 'reveal', at: where(rect)});
357
+ }
298
358
  continue;
299
359
  }
300
360
  const base = baseAction(element, role, rect);
@@ -460,7 +520,8 @@
460
520
  range.selectNodeContents(node);
461
521
  const region = regionOf(parent);
462
522
  region.text_chars += value.length;
463
- lines.push({value, region, inView: intersectsViewport(range.getBoundingClientRect())});
523
+ const rect = range.getBoundingClientRect();
524
+ lines.push({value, region, parent, rect, inView: intersectsViewport(rect)});
464
525
  }
465
526
  return lines;
466
527
  };
@@ -496,6 +557,138 @@
496
557
  return fitText(ordered.flatMap(([region, regionLines]) => [{value: '## ' + region.key}, ...regionLines]));
497
558
  };
498
559
 
560
+ // ---- Read layer ----
561
+ // Chosen kinds of content from the whole page, beyond the viewport, for reading only: nothing in it can be
562
+ // acted on, and the freshness marker never includes it. Each kind is classified by page structure, never by
563
+ // site. The outline always comes first. Field values are never read; controls give a role and a name only.
564
+ // Text, list items, links, and controls in view are left out: the viewport text and elements already hold them.
565
+ // The outline and tables are kept whole, since their structure is the point.
566
+
567
+ const READ_KINDS = ['outline', 'text', 'table', 'list', 'link', 'control', 'media'];
568
+ const TABLE_SELECTOR = 'table,[role="table"],[role="grid"]';
569
+ const LIST_ITEM_SELECTOR = 'li,[role="listitem"]';
570
+ const LINK_SELECTOR = 'a[href],[role="link"]';
571
+ const CONTROL_SELECTOR = 'button,input,select,textarea,summary,label,[contenteditable="true"],' +
572
+ INTERACTIVE_ROLES.filter(role => role !== 'link').map(role => '[role="' + role + '"]').join(',');
573
+ const MEDIA_SELECTOR = 'img[alt],[role="img"][aria-label],figcaption';
574
+ // Text inside these belongs to another kind, so the text kind never repeats it.
575
+ const NOT_TEXT_SELECTOR = [TABLE_SELECTOR, LIST_ITEM_SELECTOR, HEADING_SELECTOR, LINK_SELECTOR, CONTROL_SELECTOR,
576
+ 'figcaption', 'option'].join(',');
577
+ const requested = Array.isArray(view?.read) ? view.read : [];
578
+ const readKinds = requested.some(kind => READ_KINDS.includes(kind))
579
+ ? READ_KINDS.filter(kind => kind === 'outline' || requested.includes(kind)) : [];
580
+
581
+ const squash = (text, limit) => String(text || '').replace(/\s+/g, ' ').trim().slice(0, limit);
582
+ // Where content is, rounded to 100px: the direction to scroll and how far.
583
+ const where = rect => {
584
+ if (rect.bottom <= 0) return 'above ' + Math.max(100, Math.round(-rect.bottom / 100) * 100) + 'px';
585
+ if (rect.top >= innerHeight) return 'below ' + Math.max(100, Math.round((rect.top - innerHeight) / 100) * 100) + 'px';
586
+ return 'in view';
587
+ };
588
+ // With focus patterns, only focused regions (and open dialogs) are read; skipped regions never are.
589
+ const readable = element => {
590
+ const decision = decide(regionOf(element));
591
+ if (decision.view === 'skip') return false;
592
+ return focusPatterns.length === 0 || decision.view === 'focus' || decision.order === 0;
593
+ };
594
+ const readElements = selector => queryAllDeep(selector)
595
+ .filter(element => !element.closest(NON_CONTENT_SELECTOR) && isVisible(element) && readable(element));
596
+ // Lines with a position; a position line is written only when it changes.
597
+ const placed = items => {
598
+ const out = [];
599
+ let last = null;
600
+ for (const {at, line} of items) {
601
+ if (at !== last) out.push('@ ' + at);
602
+ out.push(line);
603
+ last = at;
604
+ }
605
+ return out;
606
+ };
607
+
608
+ const readers = {
609
+ outline: () => placed(readElements(HEADING_SELECTOR).map(heading => {
610
+ const level = /^H[1-6]$/.test(heading.tagName) ? heading.tagName.toLowerCase()
611
+ : 'h' + (heading.getAttribute('aria-level') || 2);
612
+ return {at: where(heading.getBoundingClientRect()), line: level + ' ' + squash(heading.textContent, 120)};
613
+ }).filter(item => item.line.length > 3)),
614
+ text: () => placed(lines
615
+ .filter(line => !closestDeep(line.parent, NOT_TEXT_SELECTOR) && readable(line.parent))
616
+ .map(line => ({at: where(line.rect), line: squash(line.value, 1000)}))
617
+ .filter(item => item.at !== 'in view')),
618
+ table: () => readElements(TABLE_SELECTOR)
619
+ .filter(table => !table.parentElement?.closest(TABLE_SELECTOR))
620
+ .flatMap((table, index) => {
621
+ const rows = [...table.querySelectorAll('tr,[role="row"]')]
622
+ .filter(row => row.closest(TABLE_SELECTOR) === table && isVisible(row))
623
+ .map(row => [...row.querySelectorAll('th,td,[role="cell"],[role="gridcell"],[role="columnheader"],' +
624
+ '[role="rowheader"]')].filter(cell => cell.closest('tr,[role="row"]') === row)
625
+ .map(cell => squash(cell.innerText, 80)).join(' | '))
626
+ .filter(row => row.replace(/[\s|]/g, ''));
627
+ const caption = squash(table.querySelector('caption')?.textContent || table.getAttribute('aria-label'), 80);
628
+ return ['## table ' + (index + 1) + ' @ ' + where(table.getBoundingClientRect()) +
629
+ (caption ? ': ' + caption : '') + ' (' + rows.length + ' rows)', ...rows];
630
+ }),
631
+ list: () => {
632
+ const items = readElements(LIST_ITEM_SELECTOR)
633
+ .filter(item => !item.parentElement?.closest(LIST_ITEM_SELECTOR) && !item.closest(TABLE_SELECTOR));
634
+ return placed(items.map(item => ({at: where(item.getBoundingClientRect()), line: '- ' + squash(item.innerText, 300)}))
635
+ .filter(item => item.line.length > 2 && item.at !== 'in view'));
636
+ },
637
+ link: () => {
638
+ const seen = new Set();
639
+ return placed(readElements(LINK_SELECTOR).map(link => ({at: where(link.getBoundingClientRect()),
640
+ line: '- ' + squash(accessibleName(link), 80)}))
641
+ .filter(item => item.at !== 'in view' && item.line.length > 2 && !seen.has(item.line) && seen.add(item.line)));
642
+ },
643
+ control: () => placed(readElements(CONTROL_SELECTOR)
644
+ .filter(element => !element.matches(LINK_SELECTOR) && !(element.type === 'hidden'))
645
+ .map(element => {
646
+ const rect = element.getBoundingClientRect();
647
+ return {at: where(rect), line: '- ' + (roleOf(element) || element.tagName.toLowerCase()) + ' ' +
648
+ JSON.stringify(squash(accessibleName(element), 60))};
649
+ })
650
+ .filter(item => item.at !== 'in view')),
651
+ media: () => placed(readElements(MEDIA_SELECTOR).map(element => ({at: where(element.getBoundingClientRect()),
652
+ line: '- ' + squash(element.matches('figcaption') ? element.textContent
653
+ : element.getAttribute('alt') || element.getAttribute('aria-label'), 160)}))
654
+ .filter(item => item.line.length > 2)),
655
+ };
656
+
657
+ // Each kind gets an equal share of the limit; what a small kind leaves goes to the others. Lines are kept
658
+ // whole; only a single line longer than its share is cut. Reports shown and total characters per kind.
659
+ const readLayer = () => {
660
+ if (!readKinds.length) return null;
661
+ const full = readKinds.map(kind => {
662
+ const kindLines = readers[kind]();
663
+ return {kind, lines: kindLines, total: kindLines.reduce((sum, line) => sum + line.length + 1, 0)};
664
+ });
665
+ let remaining = READ_LIMIT;
666
+ const share = new Map();
667
+ [...full].sort((a, b) => a.total - b.total).forEach((entry, index, sorted) => {
668
+ const given = Math.min(entry.total, Math.floor(remaining / (sorted.length - index)));
669
+ share.set(entry.kind, given);
670
+ remaining -= given;
671
+ });
672
+ const chars = {};
673
+ const sections = full.map(({kind, lines: kindLines, total}) => {
674
+ const taken = [];
675
+ let used = 0;
676
+ for (const line of kindLines) {
677
+ if (used + line.length + 1 > share.get(kind)) {
678
+ if (!taken.length && share.get(kind) > 1) taken.push(line.slice(0, share.get(kind) - 1));
679
+ break;
680
+ }
681
+ taken.push(line);
682
+ used += line.length + 1;
683
+ }
684
+ const shown = taken.reduce((sum, line) => sum + line.length + 1, 0);
685
+ chars[kind] = [shown, total];
686
+ const more = total > shown ? ['... ' + (total - shown) + ' more characters'] : [];
687
+ return ['# ' + kind + (kindLines.length ? '' : ' (none)'), ...taken, ...more].join('\n');
688
+ });
689
+ return {kinds: readKinds, text: sections.join('\n'), chars, limit: READ_LIMIT};
690
+ };
691
+
499
692
  // ---- Snapshot ----
500
693
 
501
694
  const actions = collectActions();
@@ -534,6 +727,25 @@
534
727
  region.shown_actions++;
535
728
  action.region = region.key;
536
729
  }
730
+ // v2: the nearest elements beyond the viewport, back in DOM order, after those in view.
731
+ const beyondKept = beyond
732
+ .map((entry, index) => ({...entry, index}))
733
+ .filter(({action}) => !viewing || decide(actionRegions.get(action)).view !== 'skip')
734
+ .sort((a, b) => a.distance - b.distance || a.index - b.index)
735
+ .slice(0, MAX_BEYOND)
736
+ .sort((a, b) => a.index - b.index)
737
+ .map(({action}) => action);
738
+ for (const action of beyondKept) {
739
+ if (!(action.node in guards)) guards[action.node] = state.guard(state.nodes.get(action.node));
740
+ action.region = actionRegions.get(action).key;
741
+ }
742
+ // The nearest and the farthest are kept: the end of a list is what loads more of it.
743
+ const kept = reveals.length > MAX_REVEALS
744
+ ? [...reveals.slice(0, MAX_REVEALS / 2), ...reveals.slice(-MAX_REVEALS / 2)] : reveals;
745
+ for (const reveal of kept) {
746
+ if (!(reveal.node in guards)) guards[reveal.node] = state.guard(state.nodes.get(reveal.node));
747
+ }
748
+ chosen = [...chosen, ...beyondKept, ...kept];
537
749
  chosen.forEach((action, index) => action.id = 'e' + (index + 1));
538
750
  if (scrollY + innerHeight < height - 2) {
539
751
  chosen.push({id: 'scroll_down', kind: 'scroll', label: 'Scroll down', delta: SCROLL_STEP});
@@ -554,8 +766,11 @@
554
766
  });
555
767
 
556
768
  state.region = element => regionOf(element).key;
769
+ const read = readLayer();
557
770
 
558
771
  return {url: location.href, title: document.title, w: innerWidth, h: innerHeight, text: shown.text,
559
772
  scroll: {y: scrollY, height}, actions: chosen, marker, page_key, guards, omitted_actions,
560
- regions: regionSummary, omitted_regions: outline.length - regionSummary.length};
773
+ regions: regionSummary, omitted_regions: outline.length - regionSummary.length, ...(read ? {read} : {}),
774
+ ...(reveals.length > kept.length ? {omitted_reveals: reveals.length - kept.length} : {}),
775
+ ...(beyond.length > beyondKept.length ? {omitted_beyond: beyond.length - beyondKept.length} : {})};
561
776
  })
package/dist/agent.js CHANGED
@@ -5,7 +5,7 @@
5
5
  */
6
6
  import { quote } from "./format.js";
7
7
  import { pageEvent, safeUrl } from "./metrics.js";
8
- import { MutationUncertain, PolicyError, StalePage, } from "./types.js";
8
+ import { Decision, MutationUncertain, PolicyError, StalePage, } from "./types.js";
9
9
  // Why the loop handed control back to Claude.
10
10
  export const GOAL_REPORTED = "GOAL_REPORTED"; // Jev chose DONE. Claude checks the page; finish verifies.
11
11
  export const BLOCKED = "BLOCKED";
@@ -13,6 +13,7 @@ export const STUCK = "STUCK"; // Three actions in a row did not change the page.
13
13
  export const ATTEMPT_BUDGET = "ATTEMPT_BUDGET";
14
14
  export const NEEDS_HUMAN = "NEEDS_HUMAN"; // A person acts in the window: sign-in, codes, CAPTCHA, [person] steps.
15
15
  export const NEEDS_VALUE = "NEEDS_VALUE"; // A field needs text and none of Claude's values fits.
16
+ export const ESCALATE = "ESCALATE"; // v2: Jev handed the decision to Claude. `answer` continues the same attempt.
16
17
  export const JEV_FAILED = "JEV_FAILED"; // No usable decision. Nothing was executed.
17
18
  export const UNCERTAIN = "UNCERTAIN"; // A mutation may have happened. Never retried, never replanned.
18
19
  /** Per attempt. A replan starts a new attempt with a fresh budget. */
@@ -28,15 +29,85 @@ export class Budget {
28
29
  }
29
30
  const MISSING = "missing"; // A history entry for a NOT_VISIBLE choice: nothing ran, but Jev must see it.
30
31
  const STALE = "stale"; // A history entry for a choice the browser refused as stale: nothing ran.
31
- /** Executed actions only; they count against the budget. */
32
+ /** What the observation offered when Jev found the element it needed missing: counts for the run log only. */
33
+ function offered(page) {
34
+ const actions = page.actions ?? [];
35
+ const scroll = page.scroll ?? {};
36
+ return {
37
+ in_view: actions.filter((a) => a.kind !== "reveal").length,
38
+ offscreen: actions.filter((a) => a.kind === "reveal").length,
39
+ omitted_actions: page.omitted_actions ?? 0,
40
+ omitted_reveals: page.omitted_reveals ?? 0,
41
+ regions_with_offscreen: (page.regions ?? []).filter((r) => r.offscreen_actions)
42
+ .length,
43
+ below_px: Math.max(0, Math.round((scroll.height ?? 0) - (scroll.y ?? 0) - (page.h ?? 0))),
44
+ };
45
+ }
46
+ /** Outcomes after which the log keeps a page check, to tell a bot check or an error page from a plan problem. */
47
+ export const CHECKED_OUTCOMES = [BLOCKED, STUCK, ATTEMPT_BUDGET, NEEDS_HUMAN];
48
+ /** Logs a page_check event. Read-only, never sent to Jev; a failed check is logged and never stops the run. */
49
+ export async function checkPage(browser, recorder, when, attempt, url) {
50
+ if (!browser.check)
51
+ return;
52
+ try {
53
+ recorder.event("page_check", { when, attempt, url: safeUrl(url), ...(await browser.check()) });
54
+ }
55
+ catch (error) {
56
+ const message = error instanceof Error ? error.message : String(error);
57
+ recorder.event("page_check_failed", { when, attempt, error: message });
58
+ }
59
+ }
60
+ /**
61
+ * Opt-in with WEBHAND_SCREENSHOTS=1: the harness, not a model, captures the page after each action so an
62
+ * outside judge can check the run. Never sent to Jev. A failed capture is logged and never stops the run.
63
+ */
64
+ export async function captureScreenshot(browser, recorder, step, attempt, after, url) {
65
+ if (process.env.WEBHAND_SCREENSHOTS !== "1" || !browser.screenshot)
66
+ return;
67
+ try {
68
+ recorder.screenshot(step, attempt, after, await browser.screenshot(), url);
69
+ }
70
+ catch (error) {
71
+ const message = error instanceof Error ? error.message : String(error);
72
+ recorder.event("screenshot_failed", { step, attempt, error: message });
73
+ }
74
+ }
75
+ /** Executed actions only, in every attempt: not a person's step, a NOT_VISIBLE choice, or a refused stale one. */
76
+ export function executed(history) {
77
+ return history.filter((h) => !["human", "answer", MISSING, STALE].includes(h.kind));
78
+ }
79
+ /** Executed actions of one attempt; they count against the budget. */
32
80
  export function actionsIn(history, attempt) {
33
- return history.filter((h) => (h.attempt ?? 1) === attempt && !["human", MISSING, STALE].includes(h.kind));
81
+ return executed(history).filter((h) => (h.attempt ?? 1) === attempt);
34
82
  }
35
83
  /** Three entries without progress: unchanged pages, targets Jev could not see, or refused stale choices. */
36
84
  function stuck(history, attempt = 1) {
37
85
  const recent = history.filter((h) => (h.attempt ?? 1) === attempt && h.kind !== "human").slice(-3);
38
86
  return recent.length === 3 && recent.every((s) => s.page_changed === false && s.kind !== "wait");
39
87
  }
88
+ /** Entries of this attempt after Claude's last answer: an answer starts the count again, so Jev is asked before any stop. */
89
+ function sinceAnswer(history, attempt) {
90
+ const own = history.filter((h) => (h.attempt ?? 1) === attempt && h.kind !== "human");
91
+ const last = own.map((h) => h.kind).lastIndexOf("answer");
92
+ return own.slice(last + 1);
93
+ }
94
+ /** v2: back and forth scrolling (three round trips) with nothing else done between. Each scroll changes the page, so stuck() never sees it. */
95
+ export function looping(history, attempt = 1) {
96
+ const recent = sinceAnswer(history, attempt).slice(-6);
97
+ const scrolls = recent.map((h) => h.operation);
98
+ return (recent.length === 6 &&
99
+ scrolls.every((op, i) => (op === "SCROLL_DOWN" || op === "SCROLL_UP") && (i === 0 || op !== scrolls[i - 1])));
100
+ }
101
+ /** How many frames, canvases, and plugin objects the page has: parts webhand cannot see into. */
102
+ export function unseenParts(page) {
103
+ const unseen = (page.status?.unseen ?? {});
104
+ return (unseen.frames ?? 0) + (unseen.canvases ?? 0) + (unseen.objects ?? 0);
105
+ }
106
+ /** v2: the needed element was not offered twice in a row, on a page with parts webhand cannot see into. */
107
+ function missingInUnseen(history, page, attempt = 1) {
108
+ const recent = sinceAnswer(history, attempt).slice(-2);
109
+ return recent.length === 2 && recent.every((h) => h.kind === MISSING) && unseenParts(page) > 0;
110
+ }
40
111
  /** A step for the run log: the value's label instead of the typed text, and the URL without query values. */
41
112
  function logged(step, region, values) {
42
113
  const { text, url, ...rest } = step;
@@ -67,18 +138,53 @@ export async function execute(browser, page, decision, history, recorder, option
67
138
  history.push(step);
68
139
  recorder.event("step", logged(step, action.region, options.values ?? []));
69
140
  onChange(page);
141
+ const asked = browser.ignoredWindows ? await browser.ignoredWindows().catch(() => []) : [];
70
142
  const newPage = await browser.observe();
71
- step.page_changed = !sameMarker(newPage.marker, page.marker);
143
+ step.page_changed = !samePage(newPage, page);
72
144
  recorder.event("step_result", { step: step.step, page_changed: step.page_changed });
145
+ options.knowledge?.record({
146
+ from: page.url,
147
+ to: newPage.url,
148
+ kind: action.kind,
149
+ label: action.label,
150
+ changed: step.page_changed,
151
+ inItem: page.guards?.[String(action.node)]?.[14] === "item",
152
+ });
153
+ if (asked.length) {
154
+ recorder.event("window_ignored", { step: step.step, attempt: options.attempt, urls: asked.map(safeUrl) });
155
+ }
73
156
  if (step.page_changed)
74
157
  recorder.event("page", pageEvent(newPage, options.attempt));
158
+ await captureScreenshot(browser, recorder, step.step, options.attempt, "action", newPage.url);
75
159
  onChange(newPage);
76
160
  return newPage;
77
161
  }
162
+ /**
163
+ * v2: Jev chose an element beyond the viewport. It is scrolled into view (a scroll, never an input), the page
164
+ * is observed again, and the same element with the same operation is taken from that observation. When it is
165
+ * no longer offered, nothing runs: StalePage, as for any refused action.
166
+ */
167
+ async function intoView(browser, page, decision, recorder, attempt) {
168
+ const action = decision.action;
169
+ await browser.act({ kind: "reveal", node: action.node, label: action.label }, page);
170
+ const next = await browser.observe();
171
+ const same = (next.actions ?? []).find((a) => !a.offscreen && a.node === action.node && a.kind === action.kind && (a.value ?? null) === (action.value ?? null));
172
+ recorder.event("into_view", { attempt, at: action.at ?? null, found: Boolean(same) });
173
+ if (!same)
174
+ throw new StalePage("Target was not in view after scrolling to it", ["target_unreachable"]);
175
+ const chosen = new Decision(decision.operation, same, { confidence: decision.confidence, text: decision.text });
176
+ return { page: next, decision: chosen };
177
+ }
78
178
  /** Markers are JSON values from the page; equal content means the same page. */
79
179
  function sameMarker(a, b) {
80
180
  return JSON.stringify(a) === JSON.stringify(b);
81
181
  }
182
+ /** The same page in the same tab setup and frames: a new or closed tab, a switch, or a change inside a frame is a change even when the page is not. */
183
+ function samePage(a, b) {
184
+ return (sameMarker(a.marker, b.marker) &&
185
+ sameMarker(a.tabs ?? null, b.tabs ?? null) &&
186
+ sameMarker(a.frame_markers ?? null, b.frame_markers ?? null));
187
+ }
82
188
  /** Runs one attempt until an outcome above. history is extended in place. */
83
189
  export async function runAgent(browser, policy, plan, recorder, budget = new Budget(), options = {}) {
84
190
  const attempt = plan.attempt ?? 1;
@@ -105,6 +211,12 @@ export async function runAgent(browser, policy, plan, recorder, budget = new Bud
105
211
  while (true) {
106
212
  if (stuck(history, attempt))
107
213
  return finish(STUCK, "three actions without a page change");
214
+ if (plan.flow === "v2" && looping(history, attempt)) {
215
+ return finish(STUCK, "scrolled up and down three times with nothing else done");
216
+ }
217
+ if (plan.flow === "v2" && missingInUnseen(history, page, attempt)) {
218
+ return finish(STUCK, "the needed element was not offered twice, and the page has parts webhand cannot see");
219
+ }
108
220
  if (actionsIn(history, attempt).length >= budget.maxActions || decisions >= budget.maxDecisions) {
109
221
  return finish(ATTEMPT_BUDGET, `${actionsIn(history, attempt).length} actions, ${decisions} decisions`);
110
222
  }
@@ -115,7 +227,11 @@ export async function runAgent(browser, policy, plan, recorder, budget = new Bud
115
227
  page = await browser.observe();
116
228
  decisions += 1;
117
229
  onDecision(decisions);
118
- decision = await policy.choose(page, plan, history);
230
+ // v2: what worked before on this kind of page, when the site is known (stage 3).
231
+ const known = options.knowledge?.hints(page.url) ?? [];
232
+ if (known.length)
233
+ recorder.event("known_offered", { attempt, count: known.length });
234
+ decision = await policy.choose(known.length ? { ...page, known } : page, plan, history);
119
235
  flushCalls();
120
236
  if (decision.operation === "NOT_VISIBLE") {
121
237
  const entry = {
@@ -129,14 +245,21 @@ export async function runAgent(browser, policy, plan, recorder, budget = new Bud
129
245
  page_changed: false,
130
246
  };
131
247
  history.push(entry);
132
- recorder.event("not_visible", { attempt, wanted: decision.wanted, confidence: decision.confidence });
248
+ recorder.event("not_visible", {
249
+ attempt,
250
+ wanted: decision.wanted,
251
+ confidence: decision.confidence,
252
+ offered: offered(page),
253
+ ...(browser.unseen ? { unseen: await browser.unseen().catch(() => null) } : {}),
254
+ ...(page.frames_log ? { frames: page.frames_log } : {}),
255
+ });
133
256
  onChange(page);
134
257
  continue;
135
258
  }
136
259
  if (decision.action === null) {
137
260
  if (!(await browser.fresh(page)))
138
261
  continue;
139
- const outcome = { DONE: GOAL_REPORTED, NEEDS_HUMAN }[decision.operation] ?? BLOCKED;
262
+ const outcome = { DONE: GOAL_REPORTED, NEEDS_HUMAN, ESCALATE }[decision.operation] ?? BLOCKED;
140
263
  return finish(outcome, `Jev chose ${decision.operation}`);
141
264
  }
142
265
  if (decision.action.kind === "fill" && decision.text === null) {
@@ -145,11 +268,14 @@ export async function runAgent(browser, policy, plan, recorder, budget = new Bud
145
268
  field[key] = decision.action[key] ?? null;
146
269
  return finish(NEEDS_VALUE, "no value from the plan fits this field", { field });
147
270
  }
271
+ if (decision.action.offscreen)
272
+ ({ page, decision } = await intoView(browser, page, decision, recorder, attempt));
148
273
  page = await execute(browser, page, decision, history, recorder, {
149
274
  decidedBy: policy.name,
150
275
  attempt,
151
276
  onChange,
152
277
  values: plan.values,
278
+ knowledge: options.knowledge,
153
279
  });
154
280
  }
155
281
  catch (error) {
@@ -167,7 +293,7 @@ export async function runAgent(browser, policy, plan, recorder, budget = new Bud
167
293
  text: null,
168
294
  url: observed.url,
169
295
  // Unchanged: the same choice would be refused again, so it counts toward STUCK.
170
- page_changed: !sameMarker(page.marker, observed.marker),
296
+ page_changed: !samePage(page, observed),
171
297
  };
172
298
  history.push(entry);
173
299
  recorder.event("stale", {
@@ -175,13 +301,14 @@ export async function runAgent(browser, policy, plan, recorder, budget = new Bud
175
301
  wanted: decision.operation,
176
302
  target: decision.action.label,
177
303
  reason: error.message,
304
+ changed: error.changed,
178
305
  page_changed: entry.page_changed,
179
306
  });
180
307
  }
181
308
  onChange(page);
182
309
  }
183
310
  else if (error instanceof PolicyError) {
184
- return finish(JEV_FAILED, error.message);
311
+ return finish(JEV_FAILED, error.message, error.diagnostic ? { jev: error.diagnostic } : {});
185
312
  }
186
313
  else if (error instanceof MutationUncertain) {
187
314
  return finish(UNCERTAIN, `mutation uncertain: ${error.message}`);