@53able/pi-agent-browser 0.1.1 → 0.2.0

package/CHANGELOG.md

@@ -1,5 +1,12 @@
 # Changelog
 
+## Unreleased
+
+### Changed
+
+- Moved the Japanese README to `README.ja-JP.md` and added top-level language switchers.
+- Added a documentation localization policy based on Trendshift repository examples.
+
 ## 0.1.1
 
 ### Changed

package/README.ja-JP.md

@@ -0,0 +1,194 @@
+# pi-agent-browser
+
+[English](README.md) | 日本語
+
+`pi-agent-browser` は、Pi から実ブラウザを操作するための typed tools を追加する拡張です。内部では [`agent-browser`](https://github.com/vercel-labs/agent-browser) を使います。
+
+Pi が Web ページを開き、rendered context を抽出し、ログイン状態を保ち、スクリーンショットを証跡として残す必要があるときに使います。raw shell command を毎回組み立てる代わりに、Pi の tool として呼び出せます。
+
+## できること
+
+- **Rendered page の context extraction** — 公開ページやログイン済み browser tab から text を読む
+- **安定した操作対象** — accessibility snapshot の `@ref` を使って click / fill する
+- **証跡保存** — screenshot や抽出 text を artifact path に保存する
+- **Session persistence** — 名前付き `agent-browser` session と restore でログイン状態を再利用する
+- **安全な browsing control** — `allowedDomains`、`maxOutput`、state management で操作範囲を絞る
+- **Typed Pi tools** — 1つの自由形式 command string ではなく、操作ごとの schema を使う
+
+## Requirements
+
+- package support 付きの Pi / Feynman runtime
+- [`agent-browser`](https://github.com/vercel-labs/agent-browser) が `PATH` 上にあること
+- `agent-browser install` で Chrome for Testing が入っていること
+
+確認:
+
+```bash
+agent-browser doctor
+```
+
+`agent-browser` が `PATH` 外にある場合:
+
+```bash
+export AGENT_BROWSER_BIN=/absolute/path/to/agent-browser
+```
+
+## Installation
+
+npm から install:
+
+```bash
+pi install npm:@53able/pi-agent-browser
+```
+
+GitHub から install:
+
+```bash
+pi install https://github.com/53able/pi-agent-browser
+```
+
+local checkout から install:
+
+```bash
+git clone https://github.com/53able/pi-agent-browser.git
+pi install ./pi-agent-browser
+```
+
+## Registered tools
+
+- `agent_browser_open` — URL を開く、または `about:blank` を起動する
+- `agent_browser_read` — URL または active rendered tab から text / JSON を抽出する
+- `agent_browser_snapshot` — 操作用の accessibility tree refs を取得する
+- `agent_browser_click` — `@ref` または CSS selector で click する
+- `agent_browser_fill` — `@ref` または CSS selector で input を埋める
+- `agent_browser_screenshot` — screenshot を disk に保存する
+- `agent_browser_eval` — active tab で JavaScript を評価する
+- `agent_browser_state` — saved state を save / load / list / show / rename / clear / clean する
+- `agent_browser_close` — browser session を閉じる
+- `agent_browser_handoff` — 2FA・CAPTCHA・本人確認のために一時停止し、人間に操作を戻す
+- `agent_browser_commit` — 決済・削除・投稿・送信など、やり直しのきかない操作の確認ゲート
+- `agent_browser_doctor` — `agent-browser doctor` を実行する
+
+## よく使う流れ
+
+### ページを読んで抽出結果を保存する
+
+Pi にはそのまま依頼できます。
+
+```text
+https://example.com をブラウザ経由で読み、outputs/browser/example.md に保存して
+```
+
+`agent_browser_read` の引数で表すと次の形です。
+
+```json
+{
+  "url": "https://example.com",
+  "outputPath": "outputs/browser/example.md",
+  "allowedDomains": ["example.com"]
+}
+```
+
+### rendered page を操作する
+
+1. ページを開きます。
+
+```json
+{
+  "url": "https://example.com",
+  "session": "research-example",
+  "allowedDomains": ["example.com"]
+}
+```
+
+2. `agent_browser_snapshot` で新しい `@ref` を取得します。
+3. `agent_browser_click` または `agent_browser_fill` にその ref を渡します。
+4. text が必要なら `agent_browser_read`、見た目の証跡が必要なら `agent_browser_screenshot` を使います。
+5. 終わったら `agent_browser_close` で session を閉じます。
+
+### ログイン状態を再利用する
+
+安定した session name と `restore: true` を使います。
+
+```json
+{
+  "url": "https://example.com",
+  "session": "work-example",
+  "restore": true,
+  "headed": true
+}
+```
+
+一度ログインしたあと、同じ `session` と `restore` で開くと saved browser state を再利用できます。
+
+### saved state を確認・削除する
+
+一覧:
+
+```json
+{
+  "operation": "list"
+}
+```
+
+特定 session の state を削除:
+
+```json
+{
+  "operation": "clear",
+  "sessionName": "work-example"
+}
+```
+
+すべて削除する場合だけ、明示的に `all` を使います。
+
+```json
+{
+  "operation": "clear",
+  "all": true
+}
+```
+
+## Safety guidance
+
+- browsing scope が分かっている場合は `allowedDomains` を渡してください。
+- report、audit、後日の verification に使う抽出結果は `outputPath` に保存してください。
+- navigation、form submit、modal dismissal、failed click のあとには fresh snapshot を取り直してください。
+- text evidence には `agent_browser_read`、visual evidence には `agent_browser_screenshot` を優先してください。
+- `agent_browser_eval` は、`read` や `snapshot` では必要な state が取れない場合にだけ使ってください。
+- saved browser state を広く消す前に、対象を確認してください。
+
+## 人間への handoff（引き渡し）
+
+すべての操作を無人で任せるべきではありません。適切なタイミングで人間に操作を戻すためのツールが2つあります。
+
+- **安全な境界線** — `agent_browser_handoff` は、AI自身では対応できない・すべきでない操作（2FA/OTPの入力、CAPTCHA、本人確認）の直前でエージェントを一時停止させます。スクリーンショットで状況を記録したあと、実際のUIダイアログでブロックし、人間が「対応完了」または「中止」を選ぶまで待ちます。`agent_browser_read` / `agent_browser_snapshot` も簡易なキーワード検知を行い、こうしたチェックポイントらしき内容を出力に警告として付加します（この警告自体は停止ではなく、`agent_browser_handoff` を呼ぶきっかけとして扱ってください）。
+- **Commit前の停止** — `agent_browser_commit` は、決済・削除・投稿・送信といった「やり直しのきかない操作」専用の確認ゲートです。証跡を保存し、クリックの前に必ず人間の明示的な確認を求め、承認されない限りクリックは実行されません。対話的なUIが利用できない場合は、クリックせずにそのまま拒否します。こうした操作の最終確認クリックには `agent_browser_click` ではなく必ずこちらを使ってください。
+
+どちらのツールも対話的なセッション（TUI、またはUI対応のRPC）を必要とします。無人・headless実行では、黙って進めるのではなく拒否します。
+
+## Troubleshooting
+
+```bash
+agent-browser doctor
+```
+
+または Pi に `agent_browser_doctor` を実行させてください。
+
+よくある修正:
+
+```bash
+npm install -g agent-browser
+agent-browser install
+```
+
+## Project history
+
+package 名の判断、npm Trusted Publishing、semantic-release workflow については、言語別の project history を参照してください。
+
+- [English](docs/project-history.en.md)
+- [日本語](docs/project-history.ja-JP.md)
+
+## License
+
+MIT

package/README.md

@@ -1,5 +1,7 @@
 # pi-agent-browser
 
+English | [日本語](README.ja-JP.md)
+
 Typed browser automation tools for Pi, powered by [`agent-browser`](https://github.com/vercel-labs/agent-browser).
 
 Use this package when Pi needs to operate real web pages, extract rendered context, keep login sessions, or capture visual evidence without falling back to ad-hoc shell commands.
@@ -63,6 +65,8 @@ pi install ./pi-agent-browser
 - `agent_browser_eval` — evaluate JavaScript in the active tab
 - `agent_browser_state` — save, load, list, show, rename, clear, or clean saved state
 - `agent_browser_close` — close a browser session
+- `agent_browser_handoff` — pause and hand control to a human for 2FA, CAPTCHA, or identity verification
+- `agent_browser_commit` — confirmation gate for irreversible actions (payment, delete, publish, send)
 - `agent_browser_doctor` — run `agent-browser doctor`
 
 ## Common workflows
@@ -72,7 +76,7 @@ pi install ./pi-agent-browser
 Ask Pi:
 
 ```text
-https://example.com をブラウザ経由で読み、outputs/browser/example.md に保存して
+Read https://example.com through the browser and save the extracted text to outputs/browser/example.md.
 ```
 
 Equivalent `agent_browser_read` arguments:
@@ -154,6 +158,15 @@ Clear all saved states only when intentionally resetting everything:
 - Use `agent_browser_eval` only when `read` or `snapshot` cannot expose the required state.
 - Review saved browser state before clearing broadly.
 
+## Human handoff
+
+Not every step should run unattended. Two tools hand control back to a human at the right moments:
+
+- **Safe boundaries** — `agent_browser_handoff` pauses the agent for steps it cannot or should not perform itself: 2FA/OTP entry, CAPTCHA, identity verification. It captures a screenshot for context, then blocks on a real confirmation dialog until a human resumes or aborts. `agent_browser_read` and `agent_browser_snapshot` also run a lightweight keyword scan and flag likely checkpoints in their output as a nudge to call this tool.
+- **Stop before commit** — `agent_browser_commit` is a dedicated gate for irreversible actions: payment, deletion, publishing, sending. It captures evidence, requires explicit human confirmation before clicking, and refuses outright (without clicking) if no interactive UI is available to ask. Use it instead of `agent_browser_click` for the final confirming step of any such action.
+
+Both tools require an interactive session (TUI or RPC with UI support); in headless/unattended runs they refuse rather than proceed silently.
+
 ## Troubleshooting
 
 Run:
@@ -171,6 +184,22 @@ npm install -g agent-browser
 agent-browser install
 ```
 
+## Project history
+
+If you maintain or contribute to this package, read the project history in [English](docs/project-history.en.md) or [Japanese](docs/project-history.ja-JP.md) for the package-name decision, npm Trusted Publishing setup, and semantic-release workflow.
+
+## Release automation
+
+Releases are automated with semantic-release. Maintainers merge Conventional Commits into `main`; GitHub Actions runs `npx semantic-release@25`, determines the next version, publishes to npm through Trusted Publishing, creates the Git tag, and writes the GitHub Release.
+
+Commit message examples:
+
+- `fix: handle empty snapshot output` → patch release
+- `feat: add browser state export tool` → minor release
+- `feat!: rename tool arguments` or `BREAKING CHANGE:` → major release
+
+The npm Trusted Publisher should continue to point at `.github/workflows/publish.yml`. No long-lived `NPM_TOKEN` is required.
+
 ## License
 
 MIT

package/docs/i18n.md

@@ -0,0 +1,59 @@
+# Documentation localization policy
+
+This project keeps English as the primary README language and provides Japanese as a first translated README.
+
+The current structure follows patterns observed in trending GitHub repositories via Trendshift:
+
+- `alibaba/open-code-review` places language links at the top of `README.md` and keeps translated files such as `README.ja-JP.md`, `README.zh-CN.md`, and localized `CONTRIBUTING.*.md` files.
+- `shiyu-coder/Kronos` places a compact language switcher directly under the project title and links to multiple translated README variants.
+- `mattpocock/skills`, `earendil-works/pi`, `likec4/likec4`, and `block/buzz` keep the main README focused on the strongest user path; this is useful even when a project later adds translations.
+
+## File naming
+
+Use locale-suffixed Markdown files at the repository root for user-facing README translations:
+
+```text
+README.md          # English, default for GitHub and npm
+README.ja-JP.md    # Japanese
+```
+
+Use locale-suffixed files under `docs/` for maintainer documentation:
+
+```text
+docs/project-history.en.md
+docs/project-history.ja-JP.md
+docs/project-history.md       # language index
+```
+
+## README language switcher
+
+The top of each README should include a short language switcher:
+
+```md
+English | [日本語](README.ja-JP.md)
+```
+
+Japanese README:
+
+```md
+[English](README.md) | 日本語
+```
+
+Keep this near the title so readers can switch before scanning installation instructions.
+
+## Translation scope
+
+Translate user-facing material first:
+
+- what the package does
+- installation
+- registered tools
+- common workflows
+- safety guidance
+- troubleshooting
+
+Maintainer-only material can live in `docs/` and should be translated when it affects contribution or release work.
+
+## Maintenance rule
+
+When changing user-facing README content, update both `README.md` and `README.ja-JP.md` in the same pull request. If a section exists in only one language, make that explicit rather than leaving readers to guess whether the other file is stale.

package/docs/project-history.en.md

@@ -0,0 +1,154 @@
+# Why this package exists
+
+This note explains the decisions behind `@53able/pi-agent-browser`: why it was created, why it uses a scoped npm name, and how releases are automated.
+
+It is mainly for maintainers and contributors. If you only want to install and use the package, start with the [README](../README.md).
+
+## The problem we wanted to solve
+
+Pi can already run shell commands, so it can call [`agent-browser`](https://github.com/vercel-labs/agent-browser) directly. That works for quick experiments, but it has a weak interface for repeatable agent work:
+
+- every browser action is a shell string
+- outputs are not consistently shaped
+- saved screenshots and extracted text are easy to lose
+- login/session state needs manual handling
+- safety options such as `allowedDomains` are easy to forget
+
+The extension began as a local wrapper around `agent-browser` to solve those issues. The goal was not to replace `agent-browser`; it was to make its browser capabilities feel native inside Pi.
+
+## What was extracted
+
+The local extension exposed browser actions as typed Pi tools:
+
+- `agent_browser_open`
+- `agent_browser_read`
+- `agent_browser_snapshot`
+- `agent_browser_click`
+- `agent_browser_fill`
+- `agent_browser_screenshot`
+- `agent_browser_eval`
+- `agent_browser_state`
+- `agent_browser_close`
+- `agent_browser_doctor`
+
+This made each action explicit. Pi can see the expected arguments, return artifact paths, preserve session names, and guide the model toward safer browsing patterns.
+
+Once the local version worked, it was moved into a standalone repository so other Pi users could install it.
+
+Repository:
+
+```text
+https://github.com/53able/pi-agent-browser
+```
+
+## Why the package is scoped
+
+The unscoped npm name `pi-agent-browser` was already taken:
+
+- npm: https://www.npmjs.com/package/pi-agent-browser
+- repository metadata: `github.com/coctostan/pi-agent-browser`
+
+That package has a similar goal, but a different interface. It exposes one generic `browser` tool that accepts an `agent-browser` command string.
+
+This package uses multiple typed tools instead. The tradeoff is intentional:
+
+| Package | Interface | Best fit |
+| --- | --- | --- |
+| `pi-agent-browser` | One free-form `browser` command string | Flexible CLI-style control |
+| `@53able/pi-agent-browser` | Separate typed `agent_browser_*` tools | Structured Pi workflows, saved artifacts, session/state handling |
+
+To avoid confusion and make ownership clear, this package is published as:
+
+```text
+@53able/pi-agent-browser
+```
+
+## npm and Trusted Publishing setup
+
+The first npm release created the package:
+
+```text
+@53able/pi-agent-browser@0.1.0
+```
+
+After the package existed on npm, Trusted Publisher was configured for GitHub Actions:
+
+- npm package: `@53able/pi-agent-browser`
+- GitHub owner: `53able`
+- GitHub repository: `pi-agent-browser`
+- Workflow filename: `publish.yml`
+- Allowed action: `npm publish`
+
+This lets GitHub Actions publish through npm OIDC. The repository does not need a long-lived `NPM_TOKEN` secret for publishing.
+
+## README cleanup
+
+The first npm README mixed user-facing instructions with maintainer details such as publishing setup. That made the package page less useful for people deciding whether to install it.
+
+The README was revised to lead with user value:
+
+- rendered-page context extraction
+- stable `@ref` handles for interaction
+- screenshot and artifact capture
+- persistent browser sessions
+- safer browsing controls
+- typed Pi tools instead of raw command strings
+
+That user-focused README was released as `0.1.1`.
+
+## Current release workflow
+
+Releases now run through [`semantic-release`](https://github.com/semantic-release/semantic-release).
+
+The current flow is:
+
+1. A maintainer merges commits into `main`.
+2. GitHub Actions runs `.github/workflows/publish.yml`.
+3. The workflow runs `npx semantic-release@25`.
+4. semantic-release reads commits since the latest release tag.
+5. If a release is needed, semantic-release chooses the next version, publishes to npm, creates the Git tag, and creates a GitHub Release.
+6. npm publish uses Trusted Publishing / OIDC.
+
+The workflow deliberately avoids `actions/setup-node`'s `registry-url` option. That option can create an `.npmrc` that interferes with semantic-release's npm authentication flow.
+
+## Commit messages drive releases
+
+Use Conventional Commits on `main`.
+
+```text
+fix: handle empty snapshot output
+```
+
+Creates a patch release.
+
+```text
+feat: add browser state export tool
+```
+
+Creates a minor release.
+
+```text
+feat!: rename browser state arguments
+```
+
+Creates a major release.
+
+A commit such as `docs: ...`, `ci: ...`, or `chore: ...` can still run the workflow, but it should not publish a new npm version.
+
+## Release workflow check
+
+After the semantic-release migration, the workflow ran successfully. semantic-release reported that npm OIDC authentication worked and that no release was needed for the `ci:` commit:
+
+```text
+OIDC token exchange with the npm registry succeeded
+Found git tag v0.1.1 associated with version 0.1.1
+There are no relevant changes, so no new version is released.
+```
+
+That means the authentication and release plumbing are in place. The next `fix:`, `feat:`, or breaking-change commit on `main` should produce the corresponding npm release.
+
+## User install command
+
+```bash
+pi install npm:@53able/pi-agent-browser
+```

package/docs/project-history.ja-JP.md

@@ -0,0 +1,154 @@
+# このパッケージを作った理由
+
+この文書は、`@53able/pi-agent-browser` を作った理由、scoped package 名を選んだ理由、リリース自動化の流れを記録したものです。
+
+主な読者はメンテナとコントリビュータです。インストールして使いたいだけの場合は、まず [README](../README.md) を読んでください。
+
+## 解決したかった問題
+
+Pi は shell command を実行できるため、[`agent-browser`](https://github.com/vercel-labs/agent-browser) を直接呼び出せます。短い検証ならそれで十分です。ただし、繰り返し使うエージェントの作業としては、次の弱さがありました。
+
+- ブラウザ操作がすべて文字列コマンドになる
+- 出力の形がそろわない
+- 保存したスクリーンショットや抽出テキストを見失いやすい
+- ログイン状態や session state を手で扱う必要がある
+- `allowedDomains` などの安全設定を忘れやすい
+
+この拡張は、それらを解消するためのローカル wrapper として始まりました。`agent-browser` を置き換えることではなく、Pi の中で `agent-browser` の機能を自然に使えるようにすることが目的です。
+
+## 切り出したもの
+
+ローカル拡張では、ブラウザ操作を次の typed Pi tools として公開しました。
+
+- `agent_browser_open`
+- `agent_browser_read`
+- `agent_browser_snapshot`
+- `agent_browser_click`
+- `agent_browser_fill`
+- `agent_browser_screenshot`
+- `agent_browser_eval`
+- `agent_browser_state`
+- `agent_browser_close`
+- `agent_browser_doctor`
+
+各操作を分けることで、Pi は引数の形を把握できます。artifact path を返したり、session 名を保ったり、安全なブラウザ操作へ誘導したりしやすくなります。
+
+ローカル版が動いたあと、他の Pi ユーザーもインストールできるように standalone repository へ移しました。
+
+Repository:
+
+```text
+https://github.com/53able/pi-agent-browser
+```
+
+## scoped package にした理由
+
+unscoped な npm package 名 `pi-agent-browser` は、すでに使われていました。
+
+- npm: https://www.npmjs.com/package/pi-agent-browser
+- repository metadata: `github.com/coctostan/pi-agent-browser`
+
+既存パッケージも `agent-browser` を Pi から使うという意味では近い目的を持っています。ただし interface が違います。既存パッケージは、`agent-browser` のコマンド文字列を受け取る単一の `browser` tool を公開しています。
+
+このパッケージは、複数の typed tools を公開します。違いは次の通りです。
+
+| Package | Interface | 向いている用途 |
+| --- | --- | --- |
+| `pi-agent-browser` | 1つの自由形式 `browser` command string | CLI に近い柔軟な操作 |
+| `@53able/pi-agent-browser` | 複数の typed `agent_browser_*` tools | 構造化された Pi workflow、artifact 保存、session/state 管理 |
+
+名前の衝突を避け、所有者を明確にするため、このパッケージは次の名前で公開しています。
+
+```text
+@53able/pi-agent-browser
+```
+
+## npm と Trusted Publishing
+
+最初の npm release で package を作成しました。
+
+```text
+@53able/pi-agent-browser@0.1.0
+```
+
+package が npm 上に存在したあと、GitHub Actions 用の Trusted Publisher を設定しました。
+
+- npm package: `@53able/pi-agent-browser`
+- GitHub owner: `53able`
+- GitHub repository: `pi-agent-browser`
+- Workflow filename: `publish.yml`
+- Allowed action: `npm publish`
+
+これにより、GitHub Actions は npm OIDC 経由で publish できます。publish 用の長期 `NPM_TOKEN` secret は不要です。
+
+## README の整理
+
+最初の npm README には、利用者向けの説明と、メンテナ向けの publish 設定が混ざっていました。パッケージページを見た人にとって、インストールする価値が分かりにくい状態でした。
+
+README は、利用者のメリットが先に伝わるように整理しました。
+
+- rendered page からの context extraction
+- `@ref` による安定した操作対象
+- screenshot と artifact の保存
+- browser session の再利用
+- browsing scope を絞る安全設定
+- raw command string ではなく typed Pi tools として使えること
+
+この利用者向け README は `0.1.1` として release しました。
+
+## 現在のリリース workflow
+
+現在の release は [`semantic-release`](https://github.com/semantic-release/semantic-release) で行います。
+
+流れは次の通りです。
+
+1. メンテナが `main` に commit を merge する。
+2. GitHub Actions が `.github/workflows/publish.yml` を実行する。
+3. workflow が `npx semantic-release@25` を実行する。
+4. semantic-release が最新 release tag 以降の commit を読む。
+5. release が必要なら、次の version を決め、npm publish、Git tag 作成、GitHub Release 作成を行う。
+6. npm publish は Trusted Publishing / OIDC を使う。
+
+workflow では、`actions/setup-node` の `registry-url` option をあえて使っていません。この option は `.npmrc` を生成し、semantic-release の npm authentication flow とぶつかることがあるためです。
+
+## Commit message が release を決める
+
+`main` に入れる commit は Conventional Commits に従います。
+
+```text
+fix: handle empty snapshot output
+```
+
+patch release になります。
+
+```text
+feat: add browser state export tool
+```
+
+minor release になります。
+
+```text
+feat!: rename browser state arguments
+```
+
+major release になります。
+
+`docs: ...`、`ci: ...`、`chore: ...` のような commit でも workflow は動きます。ただし通常は npm の新しい version は publish されません。
+
+## release workflow の確認
+
+semantic-release へ移行したあと、workflow は成功しました。semantic-release は、npm OIDC authentication が成功し、`ci:` commit では release 不要だと報告しました。
+
+```text
+OIDC token exchange with the npm registry succeeded
+Found git tag v0.1.1 associated with version 0.1.1
+There are no relevant changes, so no new version is released.
+```
+
+これで認証と release の配線は確認できています。次に `fix:`、`feat:`、breaking change の commit が `main` に入ると、対応する npm release が作られる想定です。
+
+## インストール方法
+
+```bash
+pi install npm:@53able/pi-agent-browser
+```

package/docs/project-history.md

@@ -0,0 +1,6 @@
+# Project history
+
+Choose a language:
+
+- [English](project-history.en.md)
+- [日本語](project-history.ja-JP.md)

package/index.ts

@@ -9,6 +9,32 @@ const DEFAULT_TIMEOUT_MS = 120_000;
 const DEFAULT_MAX_PREVIEW_CHARS = 12_000;
 const DEFAULT_ARTIFACT_DIR = "outputs/browser";
 
+// Heuristics for content that signals a human-only checkpoint (2FA, CAPTCHA, identity
+// verification). These are advisory only — read/snapshot stay non-destructive, so we
+// annotate the result instead of blocking. Actual pausing happens in agent_browser_handoff.
+const SENSITIVE_CONTENT_PATTERNS: { label: string; pattern: RegExp }[] = [
+  {
+    label: "2FA / one-time code",
+    pattern: /\b(two[-\s]?factor|2fa|one[-\s]?time (?:code|password)|otp|verification code)\b|認証コード|二段階認証|二要素認証|ワンタイムパスワード/i,
+  },
+  {
+    label: "CAPTCHA",
+    pattern: /\b(captcha|recaptcha|hcaptcha|prove you.?re (?:not a robot|human)|are you a robot)\b|画像認証|パズル認証/i,
+  },
+  {
+    label: "identity verification",
+    pattern: /\bverify your identity\b|\bidentity verification\b|\bconfirm it.?s you\b|本人確認|身分証明/i,
+  },
+];
+
+function scanForSensitiveContent(text: string): string[] {
+  const hits: string[] = [];
+  for (const { label, pattern } of SENSITIVE_CONTENT_PATTERNS) {
+    if (pattern.test(text)) hits.push(label);
+  }
+  return hits;
+}
+
 type JsonObject = Record<string, unknown>;
 
 type RunResult = {
@@ -25,6 +51,7 @@ type RunResult = {
 type ToolDetails = RunResult & {
   artifactPaths?: string[];
   parsedJson?: unknown;
+  securityFlags?: string[];
 };
 
 function optionalString(description: string) {
@@ -119,6 +146,25 @@ const StateSchema = Type.Object(commonSchema({
   json: optionalBoolean("Forward --json for machine-readable state output."),
 }));
 
+const HandoffSchema = Type.Object(commonSchema({
+  category: Type.Union(
+    [Type.Literal("2fa"), Type.Literal("captcha"), Type.Literal("identity_verification"), Type.Literal("other")],
+    { description: "Why AI control must pause: 2fa, captcha, identity_verification, or other." },
+  ),
+  reason: Type.String({ description: "Concise description of what the human must do (e.g. 'Enter the SMS code sent to the user\\'s phone')." }),
+  screenshot: optionalBoolean("Capture a screenshot for the human before waiting. Defaults to true."),
+}));
+
+const CommitSchema = Type.Object(commonSchema({
+  selector: Type.String({ description: "Element ref from snapshot, such as @e2, or a CSS selector, for the irreversible action (e.g. final pay/post/delete button)." }),
+  summary: Type.String({ description: "Plain-language description of what this click will do, e.g. '¥12,000 の決済を確定する' or 'この投稿を完全に削除する'." }),
+  riskCategory: Type.Union(
+    [Type.Literal("payment"), Type.Literal("delete"), Type.Literal("post_publish"), Type.Literal("send"), Type.Literal("other")],
+    { description: "Category of irreversible action: payment, delete, post_publish, send, or other." },
+  ),
+  screenshot: optionalBoolean("Capture a screenshot as evidence before asking for confirmation. Defaults to true."),
+}));
+
 const DoctorSchema = Type.Object({
   fix: optionalBoolean("Run agent-browser doctor --fix."),
   timeoutMs: Type.Optional(Type.Integer({ minimum: 1, description: "Tool execution timeout in milliseconds." })),
@@ -244,10 +290,11 @@ function preview(text: string, maxChars = DEFAULT_MAX_PREVIEW_CHARS): { text: st
   return { text: text.slice(0, maxChars) + `\n\n[truncated: ${text.length - maxChars} chars omitted]`, truncated: true };
 }
 
-function buildTextResult(result: RunResult, artifactPaths: string[] = [], parsedJson?: unknown): { content: { type: "text"; text: string }[]; details: ToolDetails } {
+function buildTextResult(result: RunResult, artifactPaths: string[] = [], parsedJson?: unknown, notes: string[] = []): { content: { type: "text"; text: string }[]; details: ToolDetails } {
   const stdoutPreview = preview(result.stdout.trim());
   const stderrPreview = preview(result.stderr.trim(), 4000);
   const lines = [
+    ...notes,
     result.ok ? "agent-browser command succeeded." : "agent-browser command failed.",
     `Command: ${result.command} ${result.args.map((a) => JSON.stringify(a)).join(" ")}`,
     `Exit: ${result.exitCode ?? "null"}${result.signal ? ` signal=${result.signal}` : ""}`,
@@ -272,6 +319,15 @@ function buildTextResult(result: RunResult, artifactPaths: string[] = [], parsed
   };
 }
 
+function sensitiveContentNotes(hits: string[]): string[] {
+  if (hits.length === 0) return [];
+  return [
+    `⚠ Possible human-only checkpoint detected: ${hits.join(", ")}.`,
+    "Do not attempt to click/fill through this yourself — call agent_browser_handoff first so a human can complete it.",
+    "",
+  ];
+}
+
 function parseJsonIfRequested(result: RunResult, requested: boolean): unknown | undefined {
   if (!requested || !result.stdout.trim()) return undefined;
   try {
@@ -341,7 +397,9 @@ export default function agentBrowserExtension(pi: ExtensionAPI) {
       if (typeof params.outputPath === "string" && params.outputPath.trim()) {
         artifacts.push(await saveArtifact(resolveOutputPath(ctx, params.outputPath, "read.txt"), result.stdout));
       }
-      return buildTextResult(result, artifacts, parseJsonIfRequested(result, params.json === true));
+      const hits = scanForSensitiveContent(result.stdout);
+      const { content, details } = buildTextResult(result, artifacts, parseJsonIfRequested(result, params.json === true), sensitiveContentNotes(hits));
+      return { content, details: { ...details, securityFlags: hits } };
     },
   });
 
@@ -363,7 +421,9 @@ export default function agentBrowserExtension(pi: ExtensionAPI) {
       if (typeof params.outputPath === "string" && params.outputPath.trim()) {
         artifacts.push(await saveArtifact(resolveOutputPath(ctx, params.outputPath, "snapshot.txt"), result.stdout));
       }
-      return buildTextResult(result, artifacts);
+      const hits = scanForSensitiveContent(result.stdout);
+      const { content, details } = buildTextResult(result, artifacts, undefined, sensitiveContentNotes(hits));
+      return { content, details: { ...details, securityFlags: hits } };
     },
   });
 
@@ -506,6 +566,153 @@ export default function agentBrowserExtension(pi: ExtensionAPI) {
     },
   });
 
+  pi.registerTool({
+    name: "agent_browser_handoff",
+    label: "Agent Browser Handoff",
+    description: "Pause AI control and hand the browser to a human for steps AI cannot or must not perform: 2FA codes, CAPTCHA, identity verification. Blocks until the human confirms completion or aborts.",
+    promptSnippet: "Call agent_browser_handoff and stop acting on the page whenever a 2FA/CAPTCHA/identity-verification checkpoint blocks progress.",
+    promptGuidelines: [
+      "Never attempt to solve CAPTCHAs, guess 2FA/OTP codes, or fabricate identity-verification data yourself.",
+      "Call this tool as soon as such a checkpoint appears; do not keep clicking or filling around it.",
+      "If the tool reports no interactive UI is available, stop the task and tell the operator to rerun it interactively.",
+    ],
+    parameters: HandoffSchema,
+    async execute(_id, rawParams, signal, onUpdate, ctx) {
+      const params = rawParams as JsonObject;
+      const category = String(params.category);
+      const reason = String(params.reason);
+      const artifacts: string[] = [];
+
+      if (params.screenshot !== false) {
+        const path = resolveOutputPath(ctx, undefined, "handoff.png");
+        await mkdir(resolve(path, ".."), { recursive: true });
+        const shotArgs: string[] = [];
+        pushCommonArgs(shotArgs, params);
+        shotArgs.push("screenshot", path);
+        await runAgentBrowser(shotArgs, { signal, timeoutMs: params.timeoutMs as number | undefined, cwd: ctx.cwd, onUpdate });
+        if (existsSync(path)) artifacts.push(path);
+      }
+
+      if (!ctx.hasUI) {
+        return {
+          content: [{
+            type: "text",
+            text: [
+              `Human handoff required (${category}) but no interactive UI is available in this run mode.`,
+              `Reason: ${reason}`,
+              artifacts.length ? `Evidence: ${artifacts.join(", ")}` : undefined,
+              "Stop this task now and ask the operator to rerun it interactively so the handoff can be completed.",
+            ].filter(Boolean).join("\n"),
+          }],
+          details: { ok: false, command: "handoff", args: [category], stdout: "", stderr: "no interactive UI", exitCode: null, signal: null, durationMs: 0, artifactPaths: artifacts },
+          terminate: true,
+        };
+      }
+
+      ctx.ui.notify(`Human handoff needed (${category}): ${reason}`, "warning");
+      ctx.ui.setStatus("agent-browser-handoff", `⏸ Waiting for human: ${reason}`);
+      let choice: string | undefined;
+      try {
+        choice = await ctx.ui.select(
+          `Human handoff required (${category}): ${reason}`,
+          ["Resume - I completed this manually", "Abort this task"],
+        );
+      } finally {
+        ctx.ui.setStatus("agent-browser-handoff", undefined);
+      }
+
+      const resumed = choice === "Resume - I completed this manually";
+      return {
+        content: [{
+          type: "text",
+          text: [
+            resumed
+              ? "Human confirmed the handoff step is complete. Take a fresh agent_browser_snapshot before continuing."
+              : "Human aborted the task during handoff. Stop and report back; do not retry automatically.",
+            artifacts.length ? `Evidence captured: ${artifacts.join(", ")}` : undefined,
+          ].filter(Boolean).join("\n"),
+        }],
+        details: { ok: resumed, command: "handoff", args: [category], stdout: choice ?? "", stderr: "", exitCode: resumed ? 0 : 1, signal: null, durationMs: 0, artifactPaths: artifacts },
+        terminate: true,
+      };
+    },
+  });
+
+  pi.registerTool({
+    name: "agent_browser_commit",
+    label: "Agent Browser Commit",
+    description: "Gate for irreversible actions (payment, delete, publish/post, send). Captures evidence, requires explicit human confirmation, then performs the click. Never bypassed for unattended runs.",
+    promptSnippet: "Use agent_browser_commit instead of agent_browser_click for the final step of any irreversible action (pay, delete, publish, send).",
+    promptGuidelines: [
+      "Route the final confirming click of a payment, deletion, publish/post, or send action through this tool, never agent_browser_click.",
+      "Write summary in plain language describing exactly what will happen, including amounts or targets when known.",
+      "If this tool reports the human declined or no UI was available, stop and report back — do not retry the click through agent_browser_click.",
+    ],
+    parameters: CommitSchema,
+    async execute(_id, rawParams, signal, onUpdate, ctx) {
+      const params = rawParams as JsonObject;
+      const selector = String(params.selector);
+      const summary = String(params.summary);
+      const riskCategory = String(params.riskCategory);
+      const artifacts: string[] = [];
+
+      if (params.screenshot !== false) {
+        const path = resolveOutputPath(ctx, undefined, "commit-evidence.png");
+        await mkdir(resolve(path, ".."), { recursive: true });
+        const shotArgs: string[] = [];
+        pushCommonArgs(shotArgs, params);
+        shotArgs.push("screenshot", path);
+        await runAgentBrowser(shotArgs, { signal, timeoutMs: params.timeoutMs as number | undefined, cwd: ctx.cwd, onUpdate });
+        if (existsSync(path)) artifacts.push(path);
+      }
+
+      if (!ctx.hasUI) {
+        return {
+          content: [{
+            type: "text",
+            text: [
+              `Irreversible action blocked (${riskCategory}): no interactive UI available to confirm with a human.`,
+              `Action: ${summary}`,
+              artifacts.length ? `Evidence: ${artifacts.join(", ")}` : undefined,
+              "The click was NOT performed. Stop this task and ask the operator to rerun it interactively.",
+            ].filter(Boolean).join("\n"),
+          }],
+          details: { ok: false, command: "commit", args: [riskCategory, selector], stdout: "", stderr: "no interactive UI", exitCode: null, signal: null, durationMs: 0, artifactPaths: artifacts },
+          terminate: true,
+        };
+      }
+
+      ctx.ui.notify(`Confirmation needed for irreversible action (${riskCategory}): ${summary}`, "warning");
+      const approved = await ctx.ui.confirm(
+        `Confirm irreversible action (${riskCategory})`,
+        `${summary}\n\nTarget: ${selector}\nThis cannot be undone. Proceed?`,
+      );
+
+      if (!approved) {
+        return {
+          content: [{
+            type: "text",
+            text: [
+              "Human declined the irreversible action. The click was NOT performed.",
+              `Action: ${summary}`,
+              artifacts.length ? `Evidence captured before decline: ${artifacts.join(", ")}` : undefined,
+              "Stop and report back; do not retry this action automatically.",
+            ].filter(Boolean).join("\n"),
+          }],
+          details: { ok: false, command: "commit", args: [riskCategory, selector], stdout: "declined", stderr: "", exitCode: 1, signal: null, durationMs: 0, artifactPaths: artifacts },
+          terminate: true,
+        };
+      }
+
+      const args: string[] = [];
+      pushCommonArgs(args, params);
+      args.push("click", selector);
+      pushExtraArgs(args, params);
+      const result = await runAgentBrowser(args, { signal, timeoutMs: params.timeoutMs as number | undefined, cwd: ctx.cwd, onUpdate });
+      return buildTextResult(result, artifacts);
+    },
+  });
+
   pi.registerTool({
     name: "agent_browser_doctor",
     label: "Agent Browser Doctor",

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@53able/pi-agent-browser",
-  "version": "0.1.1",
+  "version": "0.2.0",
   "description": "Typed Pi tools for browser automation, rendered-page context extraction, screenshots, and saved session state via agent-browser.",
   "type": "module",
   "license": "MIT",
@@ -24,7 +24,9 @@
   "files": [
     "index.ts",
     "skills",
+    "docs",
     "README.md",
+    "README.ja-JP.md",
     "LICENSE",
     "CHANGELOG.md",
     "package.json"

package/skills/agent-browser/SKILL.md

@@ -27,6 +27,15 @@ Use the `agent_browser_*` tools instead of raw `bash` commands when browser stat
 - To keep login state automatically, open with a stable `session` and `restore: true`.
 - Before clearing state, list saved states with `agent_browser_state` unless the user explicitly asks for broad cleanup.
 
+## Human handoff and commit gates
+
+Two tools exist so the agent stops and hands control back to a human instead of guessing through security or irreversible steps:
+
+- `agent_browser_handoff` — call this the moment 2FA/OTP entry, a CAPTCHA, or identity verification blocks progress. It captures a screenshot, then blocks (via a real UI dialog) until a human resumes or aborts. Never try to solve CAPTCHAs, guess codes, or fabricate identity data yourself.
+- `agent_browser_commit` — use this instead of `agent_browser_click` for the final confirming click of any irreversible action: payment, deletion, publish/post, send. It captures evidence, requires explicit human confirmation, and only clicks if approved. If no interactive UI is available, it refuses and does not click.
+
+`agent_browser_read` and `agent_browser_snapshot` also run a lightweight heuristic scan for 2FA/CAPTCHA/identity-verification keywords and prepend a warning to their output when detected — treat that warning as a signal to call `agent_browser_handoff` next, not as a blocker by itself.
+
 ## Common examples
 
 Read a public page: