@yibie/pi-jev-browser 0.1.0 → 0.2.0

package/README.md

@@ -6,6 +6,8 @@ An isolated Playwright Chromium browser for pi, driven by [Jev](https://typesafe
 
 Jev does not see screenshots. The plugin hands it a structured DOM observation and one multiple-choice question per step, and Jev answers with a concrete operation plus a probability distribution over the offered options. That removes the screenshot round trip and the reasoning round trip from every browser step.
 
+**Demo:** [`docs/demo.mp4`](docs/demo.mp4) — a recorded run that pages to the next listing, opens the first book, and scrolls until the Product Information table is in view. 2× speed, 11 s, with the cursor and click indicators the plugin injects for recordings.
+
 **This is not a replacement for `agent_browser`.** It is the other trade: no login state, no extensions, no host environment, every step recorded with its probability, and a much cheaper fast loop. Use it for narrowly scoped goals on public pages. Use a profile-based browser tool when you need the user's session.
 
 ## Decision policies
@@ -83,8 +85,8 @@ Statuses are `done_unverified`, `blocked`, `needs_review`, `uncertain`, `step_li
 
 Three layers exist, and only the first one is enforcement:
 
-1. **Mechanical.** Navigation allowlist enforced in the request router; downloads refused and logged; service workers blocked; extensions and file-system access disabled; the browser process gets an empty environment; password, file, and hidden inputs are never observed; the model can only select from server-generated target IDs, so its output can never become a selector, URL, or code.
-2. **Model guidance.** Jev is instructed to return `REVIEW` before messages, posts, orders, payments, bookings, deletions, permission changes, sensitive-data entry, CAPTCHAs, or security warnings. This is guidance, not a deterministic boundary.
+1. **Mechanical.** Navigation allowlist enforced in the request router; downloads refused and logged; service workers blocked; extensions and file-system access disabled; the browser process gets an empty environment; password, file, and hidden inputs are never observed; the model can only select from server-generated target IDs, so its output can never become a selector, URL, or code; and a human-verification gate ends the run as `needs_review` **before the decision layer is consulted at all**.
+2. **Model guidance.** Jev is instructed to return `REVIEW` before messages, posts, orders, payments, bookings, deletions, permission changes, sensitive-data entry, CAPTCHAs, or security warnings. This is guidance, not a deterministic boundary — which is why the CAPTCHA case has a mechanical guard above it. The sibling port [measured the difference](https://github.com/laihenyi/pi-Jev-browser): against a real reCAPTCHA the model stopped, but only because that widget lives in an iframe the loop cannot see; given the same gate as plain DOM controls it clicked through and reported success.
 3. **Agent instructions.** The tool guidelines tell pi to treat page content as untrusted, to ask before consequential actions, to never type secrets, and to verify `done_unverified` independently. These are instructions to another model, not guarantees.
 
 Page text, visible field values, and the goal are sent to the decision model — the TypeSafe API under policy `typesafe`, your configured provider under policy `pi` — and field values are sent to pi's model to be filled. Password and file fields are excluded, but other sensitive content is **not** automatically redacted. Delegate only narrowly scoped tasks.
@@ -99,7 +101,9 @@ Page text, visible field values, and the goal are sent to the decision model —
 - **Unbenchmarked.** End-to-end speed and live-model reliability have not been measured.
 - **Memory is narrow.** The last ten actions are retained in memory per goal within one browser session, and are dropped when the goal changes. Nothing is persisted.
 
-The loop retries only reads invalidated by a document replacement, up to five times. A browser mutation is **never** retried: a failed run may still have applied an action, so inspect the page before continuing. Three non-wait actions without observable progress end the run as `blocked`.
+The loop retries only reads invalidated by a document replacement, up to five times. A browser mutation is **never** retried: a failed run may still have applied an action, so inspect the page before continuing.
+
+Two progress guards end a run as `blocked`. Three non-wait actions without observable progress is the upstream one. The second catches longer cycles, which that guard cannot see: an action that returns the page to a state this run has already visited is counted, and **discovering any new state resets the count**. Three repeats in a row without meeting anything new means the run is going in circles rather than sweeping through pages, and it stops there instead of spending the step budget. A sweep over several items keeps producing unseen states, so it is never cut short.
 
 ## Porting notes
 
@@ -113,6 +117,8 @@ Ported from [`cline/plugins` → `plugins/jev-browser`](https://github.com/cline
 - **Dependencies.** `zod` was unused and was dropped. Upstream reached Jev through Vercel AI Gateway with `ai` and `@ai-sdk/gateway`; both are gone, because TypeSafe's own API takes the same `state` + `questions` body the loop already builds. The validation those packages provided moved into `parseChoiceAnswer`, narrowed to what the loop actually needs: only an answer that names no offered option is fatal, because a doubtful probability distribution must never kill a run that has already clicked things. Its own value is reported when it is a usable number and marked unknown otherwise.
 - **Throttling is named, and failures explain themselves.** Upstream reported any evaluation failure as an unexplained interruption. HTTP 429 and 529 now carry their own failure categories, `rate_limited` and `overloaded`, and every failure records a bounded single-line `detail`. That last part is not cosmetic: an unexplained four-step failure is what prompted this change, and the detail line is what makes the next one diagnosable.
 - **Input validation.** `jev_actions` now validates every action in the batch before executing any of it, so a malformed action can no longer leave earlier actions half-applied.
+- **A mechanical gate for human-verification challenges.** Upstream left CAPTCHAs to the `REVIEW` instruction, and a decision model can talk itself past that. A page that both announces a gate and offers a control that would pass it now ends the run as `needs_review` before the policy is consulted, with a test asserting the policy is never called. Both conditions are required, so an article about CAPTCHAs and a plain "Verify email" button are not gates. Adapted from the Apache-2.0 [`pi-Jev-browser`](https://github.com/laihenyi/pi-Jev-browser) by laihengyi, which measured the failure.
+- **Cycle detection.** Upstream stopped after three actions with no observable progress, which a longer cycle survives: clicking between known pages always changes the page. Revisiting an already-seen state now counts towards a stop of its own, and a new state resets the counter so a sweep over several items is not mistaken for a loop. Measured against a real 20-step oscillation, this stops it around step 7.
 - **Pluggable decision policy.** The decision step became the `JevPolicy` seam the upstream interface hinted at: `policy: "pi"` answers it with the model pi already has configured, so the loop needs no second credential, while `policy: "typesafe"` calls the TypeSafe API directly. Vercel AI Gateway is no longer a dependency of any kind.
 - **Verification without vision.** Upstream handed back screenshots, so a text-only model could not check a `done_unverified` claim at all. Runs now also return the stopped page's URL, title, and visible-text excerpt, produced by the same observation layer, and the image is attached only when the model declares image input. Measured on one goal: the payload dropped from 1.38 MB to 164 KB with a text-only model, while the agent's verification went from "claim is unverified" to naming the book title and price.
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@yibie/pi-jev-browser",
-  "version": "0.1.0",
+  "version": "0.2.0",
   "description": "Isolated Playwright browser for pi, driven by Jev typed decisions through the TypeSafe API or the model pi already has configured.",
   "author": "yibie",
   "repository": {
@@ -22,7 +22,8 @@
   "pi": {
     "extensions": [
       "./extensions/jev-browser.ts"
-    ]
+    ],
+    "video": "https://cdn.jsdelivr.net/gh/yibie/pi-jev-browser@main/docs/demo.mp4"
   },
   "files": [
     "extensions",

package/src/gate.ts

@@ -0,0 +1,53 @@
+import type { Observation } from "./jev-browser.ts";
+
+/**
+ * Human-verification gates are refused before the policy is consulted.
+ *
+ * The `REVIEW` instruction already tells the decision layer to stop at a CAPTCHA,
+ * and on a real reCAPTCHA demo it does — but only because that widget lives in an
+ * iframe the loop cannot see. Given the same gate as plain DOM controls, it
+ * clicked through and reported success. Whether to complete a challenge that
+ * exists to keep automation out is not a judgement call to leave to a model, so
+ * the loop makes it mechanically instead.
+ *
+ * Adapted from the Apache-2.0 `pi-Jev-browser` project by laihengyi, which
+ * measured that failure mode and wrote these phrases.
+ */
+const GATE_TEXT =
+	/\b(?:captcha|recaptcha|hcaptcha|turnstile|human verification|verification (?:challenge|required)|verify (?:that )?you(?:'re| are) (?:a )?(?:human|person)|are you (?:a )?(?:human|robot)|i(?:'m| am) not a robot|not a robot|confirm (?:that )?you(?:'re| are) (?:a )?(?:human|person)|prove (?:that )?you(?:'re| are) (?:not a robot|(?:a )?human)|press and hold|slide to verify|checking your browser)\b/i;
+
+const GATE_TARGET =
+	/verif|robot|human|person|captcha|challenge|press and hold|slide|continue|i am|i'm/i;
+
+/** A control that leaves the gate rather than passes it is not evidence of one. */
+const NOT_A_GATE_TARGET =
+	/^(?:close|dismiss|cancel|back|go back|help|learn more|privacy|terms)$/i;
+
+export interface VerificationGate {
+	/** The phrase that announced the gate, as it appeared. */
+	phrase: string;
+	/** The label of the control that would pass it. */
+	target: string;
+}
+
+/**
+ * Both conditions are required. Text alone would stop on an article about
+ * CAPTCHAs; a control alone would stop on every "Verify email" button. The cost
+ * of a false positive is a handoff to the user, never a wrong action, so the
+ * phrases lean towards recall within that constraint.
+ */
+export function verificationGate(
+	observation: Observation,
+): VerificationGate | undefined {
+	const phrase = GATE_TEXT.exec(
+		`${observation.title}\n${observation.text}`,
+	)?.[0];
+	if (!phrase) return undefined;
+	const target = observation.targets.find(
+		(candidate) =>
+			GATE_TARGET.test(candidate.label) &&
+			!NOT_A_GATE_TARGET.test(candidate.label.trim()),
+	);
+	if (!target) return undefined;
+	return { phrase, target: target.label };
+}

package/src/jev-run.ts

@@ -6,6 +6,7 @@ import {
 	StaleObservationError,
 } from "./jev-browser.ts";
 import type { JevPolicy } from "./jev-model.ts";
+import { verificationGate } from "./gate.ts";
 
 export interface RunInput {
 	goal: string;
@@ -40,8 +41,19 @@ export interface ActionHistory {
 export interface RunMemory {
 	goal: string;
 	actions: ActionHistory[];
+	/** Observation keys this run has already visited, newest last. */
+	states?: string[];
 }
 
+/**
+ * A cycle needs a repeat: a loop that returns to a page it already visited three
+ * times in a row is not making progress, while a sweep over several items keeps
+ * finding states it has never seen.
+ */
+const MAX_STATE_REVISITS = 3;
+/** Bounded, so a long run forgets the oldest states instead of growing forever. */
+const MAX_VISITED_STATES = 12;
+
 export async function runJev(
 	input: RunInput,
 	options: {
@@ -81,8 +93,10 @@ export async function runJev(
 	if (memory.goal !== input.goal) {
 		memory.goal = input.goal;
 		memory.actions = [];
+		memory.states = [];
 	}
 	let executed = 0;
+	let revisits = 0;
 	let stage = "observation";
 	const textCache = new Map<string, string>();
 	const started = performance.now();
@@ -106,6 +120,22 @@ export async function runJev(
 			const page = options.page();
 			const snapshot = await observe(page, signal);
 			try {
+				// Mechanical refusal, before the policy can choose to click through it.
+				const gate = verificationGate(snapshot.data);
+				if (gate) {
+					await options.onStep?.({
+						step,
+						operation: "REVIEW",
+						target: gate.target,
+						status: "decision",
+						reason: "verification_gate",
+						latencyMs: 0,
+					});
+					return finish(
+						"needs_review",
+						`The page presents a human-verification gate (${JSON.stringify(gate.phrase)}, control ${JSON.stringify(gate.target)}). This loop does not complete verification challenges; hand this step to the user.`,
+					);
+				}
 				const decisionStarted = performance.now();
 				stage = "evaluation";
 				const decision = await policy.choose(
@@ -221,12 +251,12 @@ export async function runJev(
 				stage = "post_action_observation";
 				const after = await observe(options.page(), signal);
 				try {
+					const stateKey = JSON.stringify(after.data);
 					memory.actions.push({
 						action: decision.target?.label ?? decision.operation,
 						kind: decision.operation,
 						text,
-						page_changed:
-							JSON.stringify(after.data) !== JSON.stringify(snapshot.data),
+						page_changed: stateKey !== JSON.stringify(snapshot.data),
 					});
 					memory.actions.splice(0, Math.max(0, memory.actions.length - 10));
 					const recent = memory.actions
@@ -237,6 +267,21 @@ export async function runJev(
 							"blocked",
 							"Three actions produced no observable progress.",
 						);
+					// That check only sees consecutive actions that change nothing, so a longer
+					// cycle — click A, click B, click A, … — passes it every time: each action
+					// does change the page, it just returns to a page already seen.
+					const visited = (memory.states ??= []);
+					if (visited.includes(stateKey)) revisits++;
+					else {
+						revisits = 0;
+						visited.push(stateKey);
+						if (visited.length > MAX_VISITED_STATES) visited.shift();
+					}
+					if (revisits >= MAX_STATE_REVISITS)
+						return finish(
+							"blocked",
+							`The run came back to a page state it had already visited ${revisits} times in a row without meeting anything new. The goal does not look reachable through the offered actions; continue with jev_actions if it is still worth pursuing.`,
+						);
 				} finally {
 					await after.dispose().catch(() => undefined);
 				}

package/test/guard.test.ts

@@ -0,0 +1,181 @@
+import assert from "node:assert/strict";
+import test from "node:test";
+import { chromium } from "playwright";
+import { verificationGate } from "../src/gate.ts";
+import type { Observation } from "../src/jev-browser.ts";
+import { type RunStep, runJev } from "../src/jev-run.ts";
+
+const base: Observation = {
+	url: "https://example.test",
+	title: "Example",
+	text: "Nothing to see",
+	targets: [{ id: "1", operation: "CLICK", label: "Continue", value: "" }],
+	scrollUp: false,
+	scrollDown: false,
+};
+
+test("a verification gate needs the announcement and a control that passes it", () => {
+	const gate = (over: Partial<Observation>) =>
+		verificationGate({
+			...base,
+			...over,
+			targets: over.targets ?? base.targets,
+		});
+	// Both present: refused.
+	assert.deepEqual(
+		gate({
+			text: "Please verify that you are human",
+			targets: [{ id: "1", operation: "CLICK", label: "Verify", value: "" }],
+		}),
+		{ phrase: "verify that you are human", target: "Verify" },
+	);
+	assert.equal(
+		gate({
+			title: "Checking your browser",
+			targets: [
+				{ id: "1", operation: "CLICK", label: "I am not a robot", value: "" },
+			],
+		})?.target,
+		"I am not a robot",
+	);
+	// A widget that lives in an iframe is invisible to the observation, so the
+	// control is the only thing that can distinguish a gate from prose in that case.
+	assert.equal(
+		gate({
+			text: "We use a CAPTCHA to keep bots out",
+			targets: [{ id: "1", operation: "CLICK", label: "Home", value: "" }],
+		}),
+		undefined,
+	);
+	assert.equal(
+		gate({
+			targets: [{ id: "1", operation: "CLICK", label: "Verify email", value: "" }],
+		}),
+		undefined,
+	);
+	// Controls that leave the gate are not evidence of one.
+	assert.equal(
+		gate({
+			text: "captcha",
+			targets: [{ id: "1", operation: "CLICK", label: "Dismiss", value: "" }],
+		}),
+		undefined,
+	);
+	assert.equal(gate({}), undefined);
+});
+
+test("the gate stops the run before the policy is consulted", async () => {
+	const browser = await chromium.launch({ headless: true });
+	try {
+		const page = await browser.newPage();
+		await page.setContent(
+			"<h1>Verify that you are human</h1><button>Continue</button>",
+		);
+		let policyCalls = 0;
+		const steps: RunStep[] = [];
+		const result = await runJev(
+			{ goal: "Read the page" },
+			{
+				page: () => page,
+				onStep: async (step) => {
+					steps.push(step);
+				},
+				policy: {
+					async choose(observation) {
+						policyCalls++;
+						return {
+							operation: "CLICK",
+							probability: 1,
+							target: observation.targets.find((t) => t.label === "Continue"),
+						};
+					},
+					async text() {
+						return "unused";
+					},
+				},
+			},
+		);
+		assert.equal(result.status, "needs_review");
+		assert.equal(policyCalls, 0, "the policy must never see this page");
+		assert.equal(result.steps.length, 0);
+		assert.equal(steps.at(-1)?.reason, "verification_gate");
+		assert.match(result.message, /does not complete verification challenges/);
+	} finally {
+		await browser.close();
+	}
+});
+
+test("a cycle between known states is stopped, a sweep over new ones is not", async () => {
+	const browser = await chromium.launch({ headless: true });
+	try {
+		const page = await browser.newPage();
+		// Two controls whose result is a state the run can come back to.
+		await page.setContent(
+			`<button onclick="document.body.dataset.at='a'">Alpha</button><button onclick="document.body.dataset.at='b'">Beta</button><p id="marker">start</p>`,
+		);
+		const clickLabel = (observation: Observation, label: string) => ({
+			operation: "CLICK",
+			probability: 1,
+			target: observation.targets.find((t) => t.label === label),
+		});
+		let turn = 0;
+		const cycle = await runJev(
+			{ goal: "Do the thing", maxSteps: 20 },
+			{
+				page: () => page,
+				policy: {
+					async choose(observation) {
+						turn++;
+						await page.evaluate(
+							(state) => {
+								document.getElementById("marker")!.textContent = state;
+							},
+							turn % 2 === 0 ? "alpha" : "beta",
+						);
+						return clickLabel(observation, "Alpha");
+					},
+					async text() {
+						return "unused";
+					},
+				},
+			},
+		);
+		assert.equal(cycle.status, "blocked");
+		assert.match(cycle.message, /already visited/);
+		assert.ok(
+			cycle.steps.length < 10,
+			`stopped early, took ${cycle.steps.length} of 20 steps`,
+		);
+
+		// A policy that keeps producing states it has never seen is a sweep, not a loop.
+		await page.setContent("<button>Next</button><p id=\"n\">0</p>");
+		let tick = 0;
+		const sweep = await runJev(
+			{ goal: "Walk through the pages", maxSteps: 20 },
+			{
+				page: () => page,
+				policy: {
+					async choose(observation) {
+						tick++;
+						if (tick > 5) return { operation: "DONE", probability: 1 };
+						await page.evaluate(
+							(value) => {
+								document.getElementById("n")!.textContent = String(value);
+							},
+							tick,
+						);
+						return clickLabel(observation, "Next");
+					},
+					async text() {
+						return "unused";
+					},
+				},
+			},
+		);
+		assert.equal(sweep.status, "done_unverified");
+		assert.equal(sweep.failure, undefined);
+		assert.equal(sweep.steps.filter((s) => s.status === "executed").length, 5);
+	} finally {
+		await browser.close();
+	}
+});