@53able/pi-agent-browser 0.2.0 → 0.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.ja-JP.md +6 -2
- package/README.md +6 -2
- package/index.ts +287 -2
- package/package.json +1 -1
- package/skills/agent-browser/SKILL.md +26 -0
package/README.ja-JP.md
CHANGED
|
@@ -66,6 +66,7 @@ pi install ./pi-agent-browser
|
|
|
66
66
|
- `agent_browser_state` — saved state を save / load / list / show / rename / clear / clean する
|
|
67
67
|
- `agent_browser_close` — browser session を閉じる
|
|
68
68
|
- `agent_browser_handoff` — 2FA・CAPTCHA・本人確認のために一時停止し、人間に操作を戻す
|
|
69
|
+
- `agent_browser_login_handoff` — インタラクティブなログイン(Google、OAuth など)のための無人 browser を起動する
|
|
69
70
|
- `agent_browser_commit` — 決済・削除・投稿・送信など、やり直しのきかない操作の確認ゲート
|
|
70
71
|
- `agent_browser_doctor` — `agent-browser doctor` を実行する
|
|
71
72
|
|
|
@@ -160,12 +161,15 @@ https://example.com をブラウザ経由で読み、outputs/browser/example.md
|
|
|
160
161
|
|
|
161
162
|
## 人間への handoff(引き渡し)
|
|
162
163
|
|
|
163
|
-
すべての操作を無人で任せるべきではありません。適切なタイミングで人間に操作を戻すためのツールが
|
|
164
|
+
すべての操作を無人で任せるべきではありません。適切なタイミングで人間に操作を戻すためのツールが3つあります。
|
|
164
165
|
|
|
165
166
|
- **安全な境界線** — `agent_browser_handoff` は、AI自身では対応できない・すべきでない操作(2FA/OTPの入力、CAPTCHA、本人確認)の直前でエージェントを一時停止させます。スクリーンショットで状況を記録したあと、実際のUIダイアログでブロックし、人間が「対応完了」または「中止」を選ぶまで待ちます。`agent_browser_read` / `agent_browser_snapshot` も簡易なキーワード検知を行い、こうしたチェックポイントらしき内容を出力に警告として付加します(この警告自体は停止ではなく、`agent_browser_handoff` を呼ぶきっかけとして扱ってください)。
|
|
167
|
+
- **インタラクティブなログイン** — `agent_browser_login_handoff` は、Google や OAuth など多くのプロバイダで sign-in がブロックされる問題を解決します。これは、agent-browser が使われた時点でオートメーション検出が起きるため、ログイン中に拒否されるというもの。このツールは CDP が接続されていない無人 Chrome プロセスを起動し、人間が正常にログインできるようにしてから、ログイン成功後に `agent-browser` を接続します。Google や OAuth の sign-in フローが期待されるタスクでは、`agent_browser_open` ではなく必ずこちらを使ってください。
|
|
166
168
|
- **Commit前の停止** — `agent_browser_commit` は、決済・削除・投稿・送信といった「やり直しのきかない操作」専用の確認ゲートです。証跡を保存し、クリックの前に必ず人間の明示的な確認を求め、承認されない限りクリックは実行されません。対話的なUIが利用できない場合は、クリックせずにそのまま拒否します。こうした操作の最終確認クリックには `agent_browser_click` ではなく必ずこちらを使ってください。
|
|
167
169
|
|
|
168
|
-
|
|
170
|
+
3つのツールすべてが対話的なセッション(TUI、またはUI対応のRPC)を必要とします。無人・headless実行では、黙って進めるのではなく拒否します。
|
|
171
|
+
|
|
172
|
+
`agent_browser_open` と `agent_browser_login_handoff` は、各セッションが`headed: true`で起動されたかどうかも記録します。`agent_browser_handoff`が、headedで開かれていないセッションに対して呼ばれた場合、人間への警告に加えて、可視ウィンドウを開き直すための`agent_browser_open`の引数(セッション名、`headed: true`、`restore: true`、最後に開いていたURL)をそのまま提示します — どのコマンドを打ち直せばよいか推測する必要はありません。ログインや2FA、CAPTCHA、本人確認に差し掛かる可能性があるタスクでは、最初から`headed: true`で開くことを推奨します。一部の確認チャレンジは使い切り・期限付きのため、途中で開き直すとログインからやり直しになることがあります。
|
|
169
173
|
|
|
170
174
|
## Troubleshooting
|
|
171
175
|
|
package/README.md
CHANGED
|
@@ -66,6 +66,7 @@ pi install ./pi-agent-browser
|
|
|
66
66
|
- `agent_browser_state` — save, load, list, show, rename, clear, or clean saved state
|
|
67
67
|
- `agent_browser_close` — close a browser session
|
|
68
68
|
- `agent_browser_handoff` — pause and hand control to a human for 2FA, CAPTCHA, or identity verification
|
|
69
|
+
- `agent_browser_login_handoff` — spawn an unattended browser for interactive login flows (Google, OAuth)
|
|
69
70
|
- `agent_browser_commit` — confirmation gate for irreversible actions (payment, delete, publish, send)
|
|
70
71
|
- `agent_browser_doctor` — run `agent-browser doctor`
|
|
71
72
|
|
|
@@ -160,12 +161,15 @@ Clear all saved states only when intentionally resetting everything:
|
|
|
160
161
|
|
|
161
162
|
## Human handoff
|
|
162
163
|
|
|
163
|
-
Not every step should run unattended.
|
|
164
|
+
Not every step should run unattended. Three tools hand control back to a human at the right moments:
|
|
164
165
|
|
|
165
166
|
- **Safe boundaries** — `agent_browser_handoff` pauses the agent for steps it cannot or should not perform itself: 2FA/OTP entry, CAPTCHA, identity verification. It captures a screenshot for context, then blocks on a real confirmation dialog until a human resumes or aborts. `agent_browser_read` and `agent_browser_snapshot` also run a lightweight keyword scan and flag likely checkpoints in their output as a nudge to call this tool.
|
|
167
|
+
- **Interactive login flows** — `agent_browser_login_handoff` solves the problem that Google, OAuth, and similar providers block sign-in when agent-browser is used, because automation detection happens during the login step itself. This tool spawns an unattended Chrome process (with no CDP attached) so the human can log in normally, then attaches `agent-browser` after login succeeds. Use this instead of `agent_browser_open` whenever a task is expected to hit an interactive Google or OAuth sign-in flow.
|
|
166
168
|
- **Stop before commit** — `agent_browser_commit` is a dedicated gate for irreversible actions: payment, deletion, publishing, sending. It captures evidence, requires explicit human confirmation before clicking, and refuses outright (without clicking) if no interactive UI is available to ask. Use it instead of `agent_browser_click` for the final confirming step of any such action.
|
|
167
169
|
|
|
168
|
-
|
|
170
|
+
All three tools require an interactive session (TUI or RPC with UI support); in headless/unattended runs they refuse rather than proceed silently.
|
|
171
|
+
|
|
172
|
+
`agent_browser_open` and `agent_browser_login_handoff` both remember whether each named session was launched with `headed: true`. If `agent_browser_handoff` is called for a session that was not opened headed, it automatically warns the human and includes the exact `agent_browser_open` arguments (session name, `headed: true`, `restore: true`, and the last known URL) needed to reopen a visible window — no more guessing which command to rerun. Prefer `headed: true` up front for any task that might hit a login, 2FA, CAPTCHA, or identity-verification step, since some challenges are single-use and reopening mid-flow may require restarting the login.
|
|
169
173
|
|
|
170
174
|
## Troubleshooting
|
|
171
175
|
|
package/index.ts
CHANGED
|
@@ -1,9 +1,10 @@
|
|
|
1
1
|
import type { ExtensionAPI, AgentToolUpdateCallback, ExtensionContext } from "@earendil-works/pi-coding-agent";
|
|
2
2
|
import { Type } from "typebox";
|
|
3
|
-
import { access, mkdir, writeFile } from "node:fs/promises";
|
|
3
|
+
import { access, mkdir, readFile, writeFile } from "node:fs/promises";
|
|
4
4
|
import { constants, existsSync } from "node:fs";
|
|
5
5
|
import { join, resolve, basename } from "node:path";
|
|
6
6
|
import { spawn } from "node:child_process";
|
|
7
|
+
import { createServer } from "node:net";
|
|
7
8
|
|
|
8
9
|
const DEFAULT_TIMEOUT_MS = 120_000;
|
|
9
10
|
const DEFAULT_MAX_PREVIEW_CHARS = 12_000;
|
|
@@ -35,6 +36,51 @@ function scanForSensitiveContent(text: string): string[] {
|
|
|
35
36
|
return hits;
|
|
36
37
|
}
|
|
37
38
|
|
|
39
|
+
// Tracks whether each agent-browser session was last opened headed, so
|
|
40
|
+
// agent_browser_handoff / agent_browser_commit can tell a human definitively
|
|
41
|
+
// whether a visible browser window exists for them to act in, instead of guessing.
|
|
42
|
+
const SESSION_STATE_DIR = ".pi-agent-browser";
|
|
43
|
+
const SESSION_STATE_FILE = "sessions.json";
|
|
44
|
+
const DEFAULT_SESSION_KEY = "__default__";
|
|
45
|
+
|
|
46
|
+
type SessionState = { headed: boolean; lastUrl?: string; updatedAt: string };
|
|
47
|
+
|
|
48
|
+
function sessionKey(params: JsonObject): string {
|
|
49
|
+
return typeof params.session === "string" && params.session.trim() ? params.session.trim() : DEFAULT_SESSION_KEY;
|
|
50
|
+
}
|
|
51
|
+
|
|
52
|
+
async function readSessionStates(cwd: string): Promise<Record<string, SessionState>> {
|
|
53
|
+
try {
|
|
54
|
+
const raw = await readFile(join(cwd, SESSION_STATE_DIR, SESSION_STATE_FILE), "utf8");
|
|
55
|
+
return JSON.parse(raw) as Record<string, SessionState>;
|
|
56
|
+
} catch {
|
|
57
|
+
return {};
|
|
58
|
+
}
|
|
59
|
+
}
|
|
60
|
+
|
|
61
|
+
async function writeSessionState(cwd: string, session: string, patch: Partial<Omit<SessionState, "updatedAt">>): Promise<void> {
|
|
62
|
+
const dir = join(cwd, SESSION_STATE_DIR);
|
|
63
|
+
await mkdir(dir, { recursive: true });
|
|
64
|
+
const states = await readSessionStates(cwd);
|
|
65
|
+
const previous = states[session];
|
|
66
|
+
states[session] = {
|
|
67
|
+
headed: patch.headed ?? previous?.headed ?? false,
|
|
68
|
+
lastUrl: patch.lastUrl ?? previous?.lastUrl,
|
|
69
|
+
updatedAt: new Date().toISOString(),
|
|
70
|
+
};
|
|
71
|
+
await writeFile(join(dir, SESSION_STATE_FILE), JSON.stringify(states, null, 2), "utf8");
|
|
72
|
+
}
|
|
73
|
+
|
|
74
|
+
function reopenHeadedGuidance(session: string, lastUrl: string | undefined): string[] {
|
|
75
|
+
const sessionArg = session === DEFAULT_SESSION_KEY ? {} : { session };
|
|
76
|
+
const reopenArgs = JSON.stringify({ ...sessionArg, headed: true, restore: true, ...(lastUrl ? { url: lastUrl } : {}) });
|
|
77
|
+
return [
|
|
78
|
+
`⚠ このセッション(${session === DEFAULT_SESSION_KEY ? "既定セッション" : session})はheadless(ウィンドウ非表示)で開かれています。`,
|
|
79
|
+
`人間が直接操作できるようにするには、agent_browser_open を次の引数で呼び直してください: ${reopenArgs}`,
|
|
80
|
+
"同じセッション名なのでcookie等は復元されますが、2FA/CAPTCHAなど使い切り・期限付きのチャレンジは、再度最初からやり直しになる場合があります。",
|
|
81
|
+
];
|
|
82
|
+
}
|
|
83
|
+
|
|
38
84
|
type JsonObject = Record<string, unknown>;
|
|
39
85
|
|
|
40
86
|
type RunResult = {
|
|
@@ -155,6 +201,15 @@ const HandoffSchema = Type.Object(commonSchema({
|
|
|
155
201
|
screenshot: optionalBoolean("Capture a screenshot for the human before waiting. Defaults to true."),
|
|
156
202
|
}));
|
|
157
203
|
|
|
204
|
+
const LoginHandoffSchema = Type.Object({
|
|
205
|
+
url: Type.String({ description: "Login/app URL to open in a real, non-automated browser for the human to sign in." }),
|
|
206
|
+
session: optionalString("agent-browser session name to attach as after login completes. Defaults to the default session."),
|
|
207
|
+
reason: Type.String({ description: "Human-readable reason for the login handoff (e.g. 'Googleアカウントへのログインが必要です')." }),
|
|
208
|
+
executablePath: optionalString("Override path to a real Chrome executable. Auto-detected on macOS if omitted."),
|
|
209
|
+
profileDir: optionalString("Override persistent user-data-dir for the login browser. Defaults to a per-session directory reused across runs so re-running the same session reuses the same authenticated profile."),
|
|
210
|
+
timeoutMs: Type.Optional(Type.Integer({ minimum: 1, description: "Tool execution timeout in milliseconds for the post-login attach step." })),
|
|
211
|
+
});
|
|
212
|
+
|
|
158
213
|
const CommitSchema = Type.Object(commonSchema({
|
|
159
214
|
selector: Type.String({ description: "Element ref from snapshot, such as @e2, or a CSS selector, for the irreversible action (e.g. final pay/post/delete button)." }),
|
|
160
215
|
summary: Type.String({ description: "Plain-language description of what this click will do, e.g. '¥12,000 の決済を確定する' or 'この投稿を完全に削除する'." }),
|
|
@@ -190,6 +245,61 @@ async function findAgentBrowserBin(): Promise<string> {
|
|
|
190
245
|
return "agent-browser";
|
|
191
246
|
}
|
|
192
247
|
|
|
248
|
+
function findRealChromeExecutable(): string | undefined {
|
|
249
|
+
const macPath = "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome";
|
|
250
|
+
if (existsSync(macPath)) return macPath;
|
|
251
|
+
return undefined;
|
|
252
|
+
}
|
|
253
|
+
|
|
254
|
+
async function findFreePort(): Promise<number> {
|
|
255
|
+
return await new Promise((resolvePort, reject) => {
|
|
256
|
+
const server = createServer();
|
|
257
|
+
server.once("error", reject);
|
|
258
|
+
server.listen(0, "127.0.0.1", () => {
|
|
259
|
+
const address = server.address();
|
|
260
|
+
const port = typeof address === "object" && address ? address.port : undefined;
|
|
261
|
+
server.close(() => {
|
|
262
|
+
if (port) resolvePort(port);
|
|
263
|
+
else reject(new Error("Could not determine a free port."));
|
|
264
|
+
});
|
|
265
|
+
});
|
|
266
|
+
});
|
|
267
|
+
}
|
|
268
|
+
|
|
269
|
+
function abortableDelay(ms: number, signal?: AbortSignal): Promise<void> {
|
|
270
|
+
return new Promise((resolveDelay) => {
|
|
271
|
+
if (signal?.aborted) {
|
|
272
|
+
resolveDelay();
|
|
273
|
+
return;
|
|
274
|
+
}
|
|
275
|
+
const onAbort = () => {
|
|
276
|
+
clearTimeout(timer);
|
|
277
|
+
resolveDelay();
|
|
278
|
+
};
|
|
279
|
+
const timer = setTimeout(() => {
|
|
280
|
+
signal?.removeEventListener("abort", onAbort);
|
|
281
|
+
resolveDelay();
|
|
282
|
+
}, ms);
|
|
283
|
+
signal?.addEventListener("abort", onAbort, { once: true });
|
|
284
|
+
});
|
|
285
|
+
}
|
|
286
|
+
|
|
287
|
+
async function waitForDebugPort(port: number, timeoutMs = 8000, intervalMs = 300, signal?: AbortSignal): Promise<boolean> {
|
|
288
|
+
const deadline = Date.now() + timeoutMs;
|
|
289
|
+
while (Date.now() < deadline) {
|
|
290
|
+
if (signal?.aborted) return false;
|
|
291
|
+
try {
|
|
292
|
+
const res = await fetch(`http://127.0.0.1:${port}/json/version`);
|
|
293
|
+
if (res.ok) return true;
|
|
294
|
+
} catch {
|
|
295
|
+
// Not up yet; keep polling.
|
|
296
|
+
}
|
|
297
|
+
if (signal?.aborted) return false;
|
|
298
|
+
await abortableDelay(intervalMs, signal);
|
|
299
|
+
}
|
|
300
|
+
return false;
|
|
301
|
+
}
|
|
302
|
+
|
|
193
303
|
function pushCommonArgs(args: string[], params: JsonObject): void {
|
|
194
304
|
if (typeof params.session === "string" && params.session.trim()) {
|
|
195
305
|
args.push("--session", params.session.trim());
|
|
@@ -346,6 +456,7 @@ export default function agentBrowserExtension(pi: ExtensionAPI) {
|
|
|
346
456
|
promptGuidelines: [
|
|
347
457
|
"Use agent_browser_snapshot after opening or navigating before clicking by @ref.",
|
|
348
458
|
"Pass allowedDomains for untrusted or task-scoped browsing when possible.",
|
|
459
|
+
"Prefer headed: true whenever the task might hit a login, 2FA, CAPTCHA, or identity-verification step — a human can only intervene through agent_browser_handoff if a visible window already exists.",
|
|
349
460
|
],
|
|
350
461
|
parameters: OpenSchema,
|
|
351
462
|
async execute(_id, rawParams, signal, onUpdate, ctx) {
|
|
@@ -365,6 +476,12 @@ export default function agentBrowserExtension(pi: ExtensionAPI) {
|
|
|
365
476
|
if (typeof params.url === "string" && params.url.trim()) args.push(params.url.trim());
|
|
366
477
|
pushExtraArgs(args, params);
|
|
367
478
|
const result = await runAgentBrowser(args, { signal, timeoutMs: params.timeoutMs as number | undefined, cwd: ctx.cwd, onUpdate });
|
|
479
|
+
if (result.ok) {
|
|
480
|
+
await writeSessionState(ctx.cwd, sessionKey(params), {
|
|
481
|
+
headed: params.headed === true,
|
|
482
|
+
lastUrl: typeof params.url === "string" && params.url.trim() ? params.url.trim() : undefined,
|
|
483
|
+
});
|
|
484
|
+
}
|
|
368
485
|
return buildTextResult(result);
|
|
369
486
|
},
|
|
370
487
|
});
|
|
@@ -582,6 +699,9 @@ export default function agentBrowserExtension(pi: ExtensionAPI) {
|
|
|
582
699
|
const category = String(params.category);
|
|
583
700
|
const reason = String(params.reason);
|
|
584
701
|
const artifacts: string[] = [];
|
|
702
|
+
const session = sessionKey(params);
|
|
703
|
+
const sessionState = (await readSessionStates(ctx.cwd))[session];
|
|
704
|
+
const headlessNotes = sessionState?.headed === true ? [] : reopenHeadedGuidance(session, sessionState?.lastUrl);
|
|
585
705
|
|
|
586
706
|
if (params.screenshot !== false) {
|
|
587
707
|
const path = resolveOutputPath(ctx, undefined, "handoff.png");
|
|
@@ -601,6 +721,7 @@ export default function agentBrowserExtension(pi: ExtensionAPI) {
|
|
|
601
721
|
`Human handoff required (${category}) but no interactive UI is available in this run mode.`,
|
|
602
722
|
`Reason: ${reason}`,
|
|
603
723
|
artifacts.length ? `Evidence: ${artifacts.join(", ")}` : undefined,
|
|
724
|
+
...headlessNotes,
|
|
604
725
|
"Stop this task now and ask the operator to rerun it interactively so the handoff can be completed.",
|
|
605
726
|
].filter(Boolean).join("\n"),
|
|
606
727
|
}],
|
|
@@ -609,8 +730,9 @@ export default function agentBrowserExtension(pi: ExtensionAPI) {
|
|
|
609
730
|
};
|
|
610
731
|
}
|
|
611
732
|
|
|
733
|
+
if (headlessNotes.length) ctx.ui.notify(headlessNotes.join(" "), "warning");
|
|
612
734
|
ctx.ui.notify(`Human handoff needed (${category}): ${reason}`, "warning");
|
|
613
|
-
ctx.ui.setStatus("agent-browser-handoff", `⏸ Waiting for human: ${reason}`);
|
|
735
|
+
ctx.ui.setStatus("agent-browser-handoff", headlessNotes.length ? `⏸ ブラウザ非表示 — ${reason}` : `⏸ Waiting for human: ${reason}`);
|
|
614
736
|
let choice: string | undefined;
|
|
615
737
|
try {
|
|
616
738
|
choice = await ctx.ui.select(
|
|
@@ -626,6 +748,7 @@ export default function agentBrowserExtension(pi: ExtensionAPI) {
|
|
|
626
748
|
content: [{
|
|
627
749
|
type: "text",
|
|
628
750
|
text: [
|
|
751
|
+
...headlessNotes,
|
|
629
752
|
resumed
|
|
630
753
|
? "Human confirmed the handoff step is complete. Take a fresh agent_browser_snapshot before continuing."
|
|
631
754
|
: "Human aborted the task during handoff. Stop and report back; do not retry automatically.",
|
|
@@ -638,6 +761,168 @@ export default function agentBrowserExtension(pi: ExtensionAPI) {
|
|
|
638
761
|
},
|
|
639
762
|
});
|
|
640
763
|
|
|
764
|
+
pi.registerTool({
|
|
765
|
+
name: "agent_browser_login_handoff",
|
|
766
|
+
label: "Agent Browser Login Handoff",
|
|
767
|
+
description: "Launch a real, non-CDP-attached Chrome for an interactive login (e.g. Google sign-in) that automation frameworks get blocked from, then attach agent-browser to the resulting session once the human confirms login is complete. Blocks until the human confirms or aborts.",
|
|
768
|
+
promptSnippet: "Use agent_browser_login_handoff (not agent_browser_open) whenever a task requires an interactive Google/OAuth-style login.",
|
|
769
|
+
promptGuidelines: [
|
|
770
|
+
"Never attempt an interactive Google/OAuth-style login via agent_browser_open, agent_browser_click, or agent_browser_fill — those attach CDP immediately, and Google blocks sign-in for CDP-attached sessions regardless of profile or cookies.",
|
|
771
|
+
"Call this tool instead whenever a task is expected to require signing into a Google account or a similarly automation-hostile login flow.",
|
|
772
|
+
"After this tool reports success, take a fresh agent_browser_snapshot before continuing — do not assume page state.",
|
|
773
|
+
"If this tool reports no interactive UI, a decline, or a failure, stop the task and report back; do not retry via agent_browser_open.",
|
|
774
|
+
],
|
|
775
|
+
parameters: LoginHandoffSchema,
|
|
776
|
+
async execute(_id, rawParams, signal, onUpdate, ctx) {
|
|
777
|
+
const params = rawParams as JsonObject;
|
|
778
|
+
const url = String(params.url);
|
|
779
|
+
const reason = String(params.reason);
|
|
780
|
+
const session = sessionKey(params);
|
|
781
|
+
|
|
782
|
+
const executablePath = typeof params.executablePath === "string" && params.executablePath.trim()
|
|
783
|
+
? params.executablePath.trim()
|
|
784
|
+
: findRealChromeExecutable();
|
|
785
|
+
|
|
786
|
+
if (!executablePath) {
|
|
787
|
+
return {
|
|
788
|
+
content: [{
|
|
789
|
+
type: "text",
|
|
790
|
+
text: [
|
|
791
|
+
"Could not find a real Chrome executable for the login handoff.",
|
|
792
|
+
"Auto-detection only checks the default macOS install path (/Applications/Google Chrome.app/Contents/MacOS/Google Chrome).",
|
|
793
|
+
"Pass executablePath explicitly, or install Google Chrome, and retry. Do not fall back to agent-browser's bundled test browser — it will be blocked by Google the same way.",
|
|
794
|
+
].join("\n"),
|
|
795
|
+
}],
|
|
796
|
+
details: { ok: false, command: "login-handoff", args: [], stdout: "", stderr: "no chrome executable found", exitCode: null, signal: null, durationMs: 0 },
|
|
797
|
+
terminate: true,
|
|
798
|
+
};
|
|
799
|
+
}
|
|
800
|
+
|
|
801
|
+
const profileDir = typeof params.profileDir === "string" && params.profileDir.trim()
|
|
802
|
+
? params.profileDir.trim()
|
|
803
|
+
: join(ctx.cwd, SESSION_STATE_DIR, "login-profiles", session);
|
|
804
|
+
await mkdir(profileDir, { recursive: true });
|
|
805
|
+
|
|
806
|
+
const port = await findFreePort();
|
|
807
|
+
|
|
808
|
+
let spawnError: Error | undefined;
|
|
809
|
+
const child = spawn(executablePath, [
|
|
810
|
+
`--remote-debugging-port=${port}`,
|
|
811
|
+
`--user-data-dir=${profileDir}`,
|
|
812
|
+
"--no-first-run",
|
|
813
|
+
"--no-default-browser-check",
|
|
814
|
+
url,
|
|
815
|
+
], { detached: true, stdio: "ignore" });
|
|
816
|
+
// Persistent handler: without this, an 'error' event (e.g. ENOENT for a bad
|
|
817
|
+
// executablePath) on this detached/unref'd child is unhandled and crashes the
|
|
818
|
+
// whole host process whenever it fires, even long after this tool call returns.
|
|
819
|
+
child.on("error", (error) => {
|
|
820
|
+
spawnError = error;
|
|
821
|
+
});
|
|
822
|
+
child.unref();
|
|
823
|
+
|
|
824
|
+
const debugPortWaitController = new AbortController();
|
|
825
|
+
const debugPortUp = spawnError ? false : await Promise.race([
|
|
826
|
+
waitForDebugPort(port, undefined, undefined, debugPortWaitController.signal),
|
|
827
|
+
new Promise<boolean>((resolveRace) => {
|
|
828
|
+
child.once("error", () => resolveRace(false));
|
|
829
|
+
}),
|
|
830
|
+
]);
|
|
831
|
+
// Whichever branch lost the race must stop polling promptly instead of
|
|
832
|
+
// holding the event loop open for its own full timeout in the background.
|
|
833
|
+
debugPortWaitController.abort();
|
|
834
|
+
if (spawnError || !debugPortUp) {
|
|
835
|
+
try { child.kill(); } catch { /* best-effort */ }
|
|
836
|
+
return {
|
|
837
|
+
content: [{
|
|
838
|
+
type: "text",
|
|
839
|
+
text: [
|
|
840
|
+
spawnError
|
|
841
|
+
? `Failed to launch the login browser: ${spawnError.message}`
|
|
842
|
+
: `Failed to confirm the login browser's debug port (127.0.0.1:${port}) came up in time.`,
|
|
843
|
+
`Attempted to launch: ${executablePath}`,
|
|
844
|
+
"The login browser was not confirmed usable; it has been asked to close. Stop and report back.",
|
|
845
|
+
].join("\n"),
|
|
846
|
+
}],
|
|
847
|
+
details: { ok: false, command: "login-handoff", args: [executablePath, url], stdout: "", stderr: spawnError ? spawnError.message : "debug port did not come up", exitCode: null, signal: null, durationMs: 0 },
|
|
848
|
+
terminate: true,
|
|
849
|
+
};
|
|
850
|
+
}
|
|
851
|
+
|
|
852
|
+
if (!ctx.hasUI) {
|
|
853
|
+
try { child.kill(); } catch { /* best-effort */ }
|
|
854
|
+
return {
|
|
855
|
+
content: [{
|
|
856
|
+
type: "text",
|
|
857
|
+
text: [
|
|
858
|
+
"Login handoff required but no interactive UI is available in this run mode.",
|
|
859
|
+
`Reason: ${reason}`,
|
|
860
|
+
"The login browser has been closed. Stop this task now and ask the operator to rerun it interactively so the login can be completed.",
|
|
861
|
+
].join("\n"),
|
|
862
|
+
}],
|
|
863
|
+
details: { ok: false, command: "login-handoff", args: [executablePath, url], stdout: "", stderr: "no interactive UI", exitCode: null, signal: null, durationMs: 0 },
|
|
864
|
+
terminate: true,
|
|
865
|
+
};
|
|
866
|
+
}
|
|
867
|
+
|
|
868
|
+
ctx.ui.notify(`Login handoff needed: ${reason}`, "warning");
|
|
869
|
+
ctx.ui.setStatus("agent-browser-login-handoff", `⏸ ログイン待機中 — ${reason}`);
|
|
870
|
+
let choice: string | undefined;
|
|
871
|
+
try {
|
|
872
|
+
choice = await ctx.ui.select(
|
|
873
|
+
`Login required: ${reason}`,
|
|
874
|
+
["Resume - I completed the login", "Abort this task"],
|
|
875
|
+
);
|
|
876
|
+
} finally {
|
|
877
|
+
ctx.ui.setStatus("agent-browser-login-handoff", undefined);
|
|
878
|
+
}
|
|
879
|
+
|
|
880
|
+
if (choice !== "Resume - I completed the login") {
|
|
881
|
+
try { child.kill(); } catch { /* best-effort */ }
|
|
882
|
+
return {
|
|
883
|
+
content: [{
|
|
884
|
+
type: "text",
|
|
885
|
+
text: "Human aborted the task during login handoff. Stop and report back; do not retry automatically.",
|
|
886
|
+
}],
|
|
887
|
+
details: { ok: false, command: "login-handoff", args: [executablePath, url], stdout: choice ?? "", stderr: "", exitCode: 1, signal: null, durationMs: 0 },
|
|
888
|
+
terminate: true,
|
|
889
|
+
};
|
|
890
|
+
}
|
|
891
|
+
|
|
892
|
+
const attachArgs = ["--cdp", String(port), "--session", session, "open"];
|
|
893
|
+
const result = await runAgentBrowser(attachArgs, { signal, timeoutMs: params.timeoutMs as number | undefined, cwd: ctx.cwd, onUpdate });
|
|
894
|
+
|
|
895
|
+
if (!result.ok) {
|
|
896
|
+
return {
|
|
897
|
+
content: [{
|
|
898
|
+
type: "text",
|
|
899
|
+
text: [
|
|
900
|
+
"Login browser was confirmed by the human, but attaching agent-browser via --cdp failed. Do not assume the session is usable.",
|
|
901
|
+
`Command: ${result.command} ${result.args.map((a) => JSON.stringify(a)).join(" ")}`,
|
|
902
|
+
].join("\n"),
|
|
903
|
+
}],
|
|
904
|
+
details: { ...result, artifactPaths: [] },
|
|
905
|
+
terminate: true,
|
|
906
|
+
};
|
|
907
|
+
}
|
|
908
|
+
|
|
909
|
+
await writeSessionState(ctx.cwd, session, { headed: true, lastUrl: url });
|
|
910
|
+
|
|
911
|
+
return {
|
|
912
|
+
content: [{
|
|
913
|
+
type: "text",
|
|
914
|
+
text: [
|
|
915
|
+
"Login handoff complete and agent-browser is now attached to the human's authenticated Chrome session.",
|
|
916
|
+
`Session: ${session === DEFAULT_SESSION_KEY ? "(default)" : session}`,
|
|
917
|
+
"Take a fresh agent_browser_snapshot before continuing.",
|
|
918
|
+
].join("\n"),
|
|
919
|
+
}],
|
|
920
|
+
details: { ...result, artifactPaths: [] },
|
|
921
|
+
terminate: true,
|
|
922
|
+
};
|
|
923
|
+
},
|
|
924
|
+
});
|
|
925
|
+
|
|
641
926
|
pi.registerTool({
|
|
642
927
|
name: "agent_browser_commit",
|
|
643
928
|
label: "Agent Browser Commit",
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@53able/pi-agent-browser",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.4.0",
|
|
4
4
|
"description": "Typed Pi tools for browser automation, rendered-page context extraction, screenshots, and saved session state via agent-browser.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"license": "MIT",
|
|
@@ -36,6 +36,32 @@ Two tools exist so the agent stops and hands control back to a human instead of
|
|
|
36
36
|
|
|
37
37
|
`agent_browser_read` and `agent_browser_snapshot` also run a lightweight heuristic scan for 2FA/CAPTCHA/identity-verification keywords and prepend a warning to their output when detected — treat that warning as a signal to call `agent_browser_handoff` next, not as a blocker by itself.
|
|
38
38
|
|
|
39
|
+
### If the human can't find the browser window
|
|
40
|
+
|
|
41
|
+
`agent_browser_open` records whether each named session was launched with `headed: true`. When `agent_browser_handoff` runs for a session that was not opened headed, it automatically prepends a warning naming the session and the exact `agent_browser_open` arguments (including the last known URL) needed to reopen it visibly, instead of leaving the human to guess. Prefer `headed: true` from the start for any task that might hit a login, 2FA, CAPTCHA, or identity-verification step — a human can only intervene through `agent_browser_handoff` if a visible window already exists. Note that some verification challenges are single-use or session-bound, so reopening may require restarting the login flow rather than resuming mid-challenge.
|
|
42
|
+
|
|
43
|
+
### For interactive login flows (Google, OAuth, and similar)
|
|
44
|
+
|
|
45
|
+
`agent_browser_login_handoff` solves a specific problem: Google and similar OAuth providers block sign-in when `agent-browser open` is used, because agent-browser attaches a remote debugging protocol (CDP) session the instant Chrome launches, and Google detects this automation marker and rejects the login attempt regardless of profile, cookies, or executable.
|
|
46
|
+
|
|
47
|
+
This tool spawns a plain, un-instrumented Chrome process (not through `agent-browser open`) with a remote debugging port and a dedicated persistent profile, so no CDP client is attached while the human logs in. Chrome behaves like an ordinary browser to Google during sign-in. Only after the human confirms login is complete does the tool attach `agent-browser` via CDP to that already-authenticated browser for all subsequent automated snapshot, read, and click operations.
|
|
48
|
+
|
|
49
|
+
**Important:** this is an architecture fix, not detection evasion. The tool does not hide webdriver flags or automation fingerprints — it simply avoids having a CDP session attached during the interactive login moment itself.
|
|
50
|
+
|
|
51
|
+
Use `agent_browser_login_handoff` instead of `agent_browser_open` for any task expected to hit an interactive Google, OAuth, or similar sign-in flow. Never try to push through such a login via `agent_browser_open` + click/fill — that's the exact failure mode this tool exists to avoid.
|
|
52
|
+
|
|
53
|
+
Example call:
|
|
54
|
+
|
|
55
|
+
```json
|
|
56
|
+
{
|
|
57
|
+
"url": "https://myapp.com/login",
|
|
58
|
+
"session": "google-oauth-flow",
|
|
59
|
+
"reason": "User must complete Google OAuth sign-in for app access"
|
|
60
|
+
}
|
|
61
|
+
```
|
|
62
|
+
|
|
63
|
+
After the human confirms login is complete, subsequent `agent_browser_open`, `agent_browser_snapshot`, `agent_browser_click`, and other calls with the same `session` name will reuse the authenticated browser state and operate normally. The session is marked as headed in the session-state tracking, so later calls to `agent_browser_handoff` on the same session won't false-flag it as headless.
|
|
64
|
+
|
|
39
65
|
## Common examples
|
|
40
66
|
|
|
41
67
|
Read a public page:
|