pi-lean-dimension 0.2.3 → 0.3.0

package/README.md

@@ -1,6 +1,6 @@
 # pi-lean-dimension
 
-> Web browsing and search tools for [Pi](https://github.com/earendil-works/pi-coding-agent), the AI coding agent.
+> Web browsing and search tools for [Pi](https://github.com/earendil-works/pi), the AI coding agent.
 
 A monorepo housing three Pi extension packages that give your AI agent the
 ability to browse the web interactively, fetch static pages as Markdown, and

package/node_modules/pi-lean-portal/AGENTS.md

@@ -39,16 +39,11 @@ Capabilities (`PluginCapabilities`) advertise quirks. The router checks them at
 ## Browser launch hook
 
 The portal's own backends launch their browser directly and drive
-their own page — there is **no external-attach path**. The
-`getAttachEndpoint()` interface method, the `AttachEndpoint` union,
-`core/shared/cdp-endpoint.ts`, the `launchServer()` / `connect()`
-hop, and the `_cdpEndpoint` / `_wsEndpoint` / `_browserServer` /
-`_reconnectBrowser()` scaffolding were all removed (see
+their own page — there is **no external-attach path** and no
+post-launch hook for third-party subclasses (see
 [`docs/decisions/miniwob-and-host-setup.md`](../../docs/decisions/miniwob-and-host-setup.md)).
-There is no post-launch hook for third-party subclasses and no
-`connectOverCDP?` interface hook — a host-owns-browser ("Mode B")
-path was considered and dropped as YAGNI; either will be re-added
-alongside a real consumer that needs it.
+A host-owns-browser path ("Mode B") was considered and dropped as
+YAGNI; either will be re-added alongside a real consumer that needs it.
 
 ## Stealth backends (user-managed)
 
@@ -140,22 +135,20 @@ as class attributes on your subclass:
 |------|---------|-----------------|
 | `_fingerprint_managed_context` | `False` | `create_browser_context()` skips hardcoded `viewport`/`user_agent`; lets the fingerprint package set them. |
 | `_eval_prefix` | `""` | Prepended to every `page.evaluate` expression in `do_evaluate` (e.g. Camoufox's `"mw:"` routes writes to the main world). |
-| `_scroll_via_wheel` | `False` | `do_scroll` uses `page.mouse.wheel` instead of `page.evaluate("window.scrollBy")` (avoids eval-write under isolated-world stealth). |
+| `_scroll_via_wheel` | `False` | `do_scroll` uses `page.mouse.wheel` instead of `page.evaluate("window.scrollBy")`. Legacy Camoufox (``135.0.1-beta.24``) needed this; current binary (``152.0.4-beta.28``) is the reverse (wheel no-ops, eval scrollBy works), so Camoufox leaves the default `False`. |
 | `_skip_default_viewport` | `False` | Skips Playwright's `Browser.setDefaultViewport` CDP call (Camoufox binary rejects its `isMobile` prop). |
 | `_skip_networkidle` | `False` | Nav-settle uses `load` instead of `networkidle` (patched binaries don't fire `networkidle` reliably). |
-| `_wrap_mw_eval_in_eval` | `False` | `do_evaluate` rewrites the expression as `eval(<JSON-string of expression>)` before prepending `_eval_prefix`, so multi-statement scripts survive Camoufox's `let _s = (${script})` main-world wrapper (see prose below). |
+| `_wrap_mw_eval_in_eval` | `False` | `do_evaluate` rewrites the expression as `eval(<JSON-string of expression>)` before prepending `_eval_prefix`, so multi-statement scripts survive Camoufox's `let _s = (${script})` main-world wrapper. |
+| `_settle_budget_ms` | `400` | Poll budget (ms) for `_wait_for_navigation_settle` when no `framenavigated` event fires. Stealth backends with high Juggler event latency (e.g. Camoufox) override to 2000. |
+| `_url_stability_settle` | `False` | When True, the no-nav poll confirms URL is stable at a new value for 150 ms before exiting, rather than a fixed budget. |
 | `_csp_safe_readonly_via_init_script` | `False` | `do_evaluate(read_only=True)` (the EXTRACTOR_SCRIPT) reads its JSON result from a `<meta id="__pi-extract">` tag that `create_browser_context` registers as a `context.add_init_script` (isolated world, CSP-free) at `DOMContentLoaded`, instead of `page.evaluate`. For patched-Firefox stealth binaries that route `page.evaluate` through `eval()` in the page's main world (CSP-subject) — Camoufox is NOT affected (its binary keeps Juggler's CSP-free isolated-world). The adapter plumbs the script via the `browser.init` config key `readOnlyExtractorScript`. Stale across SPA route changes (no new load) — fine for navigate→inspect. |
 
 All flags default off → `chromium-py` / `firefox-py` behavior is
-bit-identical to a pre-stealth install. A dropped v2 `_context_factory`
-flag (dispatching to a `_camoufox_new_context` helper) was removed
-when `camoufox.NewContext` turned out to be broken on the current
-binary (`Protocol error (Browser.setDefaultViewport)` from the same
-`isMobile` rejection `_skip_default_viewport` handles). Camoufox
-injects the fingerprint at **browser launch** via `camoufox.NewBrowser`,
+bit-identical to a pre-stealth install. Camoufox injects the
+fingerprint at **browser launch** via `camoufox.NewBrowser`,
 so standard `browser.new_context()` with
-`_fingerprint_managed_context = True` is correct — do not re-attempt
-`NewContext`.
+`_fingerprint_managed_context = True` is correct — do not
+re-attempt `NewContext`.
 
 ### Camoufox: the shipped example template
 
@@ -204,14 +197,13 @@ All tool calls dispatch through the router. Key responsibilities:
 
 ## Known Constraints & Debt
 
-- **Console capture in Python backends** — Both `chromium-py` and `firefox-py` inherit console capture (500-entry ring buffer) and dialog auto-dismissal from `PlaywrightBridge._setup_page_session()` in `python-base`. The base `BrowserBridge` (an `abc.ABC`) does not define `_setup_page_session`; future Python plugins that subclass `BrowserBridge` directly must supply their own session setup (subclasses of `PlaywrightBridge` inherit it).
+- **Console capture in Python backends** — Both `chromium-py` and `firefox-py` inherit console capture (500-entry ring buffer) and dialog auto-dismissal from `PlaywrightBridge._setup_page_session()` in `python-base`. All Python browser backends must subclass `PlaywrightBridge` to get session setup and JSON-RPC dispatch.
 - **AbortSignal not supported on Python bridge** — the router passes `signal` through unconditionally (no capability check). The Python adapter accepts and silently ignores the signal. `supportsAbortSignal` is advertised but unenforced.
 - **Sessions are per taskId** — mapped to `browser-NNN` keys via `_sessionKeys`/`_sessionCounter` in `core/shared/task-id.ts`. Created on first navigate, cleaned up on `session_shutdown`.
-- **Python shared-context machinery removed (B1)** — the `browser.newPage`/`browser.closePage` RPC routes, `_profile_contexts` ref-counting, and `ensure_profile_session`/`remove_profile_session` methods were removed from both the base `BrowserBridge` and `ChromiumPyBridge`. Named profiles now use disk persistence (load-on-navigate via `storageState`) matching the TS Chromium plugin. Both backends use `ensure_session(task_id, config)` for all sessions.
+- **Python shared-context machinery removed** — Named profiles now use disk persistence (load-on-navigate via `storageState`) matching the TS Chromium plugin. Both backends use `ensure_session(task_id, config)` for all sessions.
 - **Python bridge reuses BrowserContexts across navigations** — `ensure_session()` returns the existing session on re-navigate (unlike the TS Chromium plugin which creates a fresh context per navigate). This means in-process cookies survive re-navigation without explicit save, but also means `storageState` from the router is ignored on re-navigate (the context already exists). The Python adapter's `_persistState()` saves current cookies to disk before the navigate RPC for cross-process persistence.
 - **`_persistState()` helper in both backends** — both `PlaywrightPluginBase._persistState` (direct `context.storageState()` call) and `PythonPluginAdapter._persistState` (JSON-RPC `browser.getStorageState` retrieval) delegate to the shared `persistSessionState()` helper in `core/shared/storage-state.ts`, which owns the `session?.persistState` gate, the `saveStorageState()` call, and the warn-and-swallow error path. Called both from `cleanup()` and — on re-navigate — from `getOrCreateContext()` (Chromium) or `navigate()` (Python) before the old context is closed/reused.
 - **Role-based locators only**: never XPath/CSS — always `getByRole()` via `buildLocator()` with positional `.nth()` for duplicates. The `INTERACTIVE_ROLES` set defines which roles get @e refs.
-- **All URLs go through `url-safety.ts`** — blocks localhost, private IPs (10.x, 172.16-31.x, 192.168.x, 169.254.169.254), dangerous schemes (file:, ftp:, data:, javascript:, vbscript:), and heuristically detects secrets in URLs.
 - **Screenshot**: JPEG 80% quality, viewport constrained to 1280px wide, returns data URI.
 - **Accessibility tree parser is single-pass, no-cap**: both TypeScript (`core/shared/accessibility-tree.ts`) and Python (`backends/python-base/pi_browser_bridge/accessibility.py`) use an identical single-pass algorithm — every interactive element gets an @e ref, no dialog prioritization, no element cap. Full ARIA trees beyond truncation are cached to disk via `snapshot-cache.ts`.
 - **Bot detection has three tiers**: checked against page title (challenge phrases), body text (challenge phrases + CDN patterns), and raw HTML (CAPTCHA widget embed codes). Both the TypeScript (`core/shared/bot-detection.ts`) and Python (`python-base/pi_browser_bridge/bot_detection.py`) backends share the same HTML-level signal set.
@@ -220,7 +212,7 @@ All tool calls dispatch through the router. Key responsibilities:
 - **`browser-inspect`** (`core/shared/dom-extractor.ts`): runs inline JS via `page.evaluate()`. Requires `getElementCache()` on the plugin. Text output truncated at ~2500 chars by default; pass `maxChars=0` for full. Keyword filtering via `query` parameter (case-insensitive substring on text, href, src).
 - **`parentRef` on `AriaCachedNode`**: enables `subtree=...` queries in `browser-inspect`. Set by depth-based parent stack in `parseSnapshot()`'s single pass. Dialogs become parent of interior elements.
 - **`dialogDetected` is resolved from element cache**: computed from the parsed `ElementCache` via `Array.some()` matching `role="dialog"` or `role="alertdialog"`. Not affected by snapshot truncation (unlike the old string-scan approach).
-- **Guide staleness**: no builtin site guides shipped — entirely user-authored via `~/.pi/agent/pi-lean-portal/web-guides/*.md`. Guides carry `updated` date and `currentDate` timestamp in output.
+- **Guide staleness**: no builtin site guides shipped — entirely user-authored via `~/.pi/agent/pi-lean-portal/web-guides/*.md`. User `.md` files override builtins by name collision (the file name minus `.md` becomes the guide key; a user `bot-detection.md` shadows the builtin `bot-detection` pattern). Guides carry `updated` date and `currentDate` timestamp in output.
 - **Learn mode toggle**: `/web learn` enables `web-learn` tool; `/web on` removes it. Agent never calls `web-learn` unprompted. Default is off on fresh sessions.
 - **Navigation settle** (`core/shared/nav-settle.ts`): after click or press, detects page navigation via a `framenavigated` listener and waits for `load + networkidle` (capped, errors swallowed) before reading URL/title/snapshot. Replaces the old fixed `waitForTimeout(300)` pattern that caused URL/DOM mismatches. Framework-agnostic via a lightweight `NavigationSettlePage` interface for testability.
 - **Stealth backends are user-managed, not shipped** — they live under `~/.pi/agent/pi-lean-portal/user-backends/`, are never in the npm tarball, and the extension never auto-downloads them. The user-side install burden is real: a per-engine venv, a ~100 MB patched-binary fetch, and an explicit `settings.json` entry with an **absolute** `pythonPath`. See `contributed/README.md` for the install flow.

package/node_modules/pi-lean-portal/README.md

@@ -61,7 +61,7 @@ The browser tools are **enabled by default**. You can:
 
 ## Extending it
 
-Beyond the toggle, two surfaces are user-extensible rather than hardcoded:
+Beyond the toggle, three surfaces are extensible or shared rather than hardcoded:
 
 - **Navigation guides** — `web-learn` saves site-specific playbooks that
   auto-match by domain and resurface in later sessions. See
@@ -71,6 +71,14 @@ Beyond the toggle, two surfaces are user-extensible rather than hardcoded:
   and drive a patched engine like [Camoufox](https://github.com/daijro/camoufox)
   yourself. The full flow lives in [Backend Architecture](#backend-architecture)
   and [`contributed/README.md`](./contributed/README.md).
+- **Toolset toggles** — the `/web` toggle is powered by the shared
+  [`pi-tool-masking`](https://github.com/coreyryanhanson/pi-tool-masking/) peer dep,
+  which handles active-set masking, sibling-tool union math, and the
+  `TOOLSET_EVENTS` protocol that keeps the `browser`/`search` status-bar
+  glyphs in sync across extensions. It's published as a general toggle
+  mechanism — any extension that hides/shows its own tools can build on the
+  same scheme so masking and glyph events stay consistent across the whole
+  agent, rather than each toggle reinventing its own.
 
 ---
 
@@ -265,6 +273,28 @@ HTML to Markdown. It does **not** create a browser session.
 | `pagination` | On-demand | Next buttons, infinite scroll, pages |
 | `search` | On-demand | Search boxes, comboboxes, result lists |
 
+### Overriding Built-in Guides
+
+A same-named `.md` file in `~/.pi/agent/pi-lean-portal/web-guides/` (e.g.
+`bot-detection.md`) **shadows the builtin entirely** — the whole guide is
+replaced, not field-merged. To keep a pattern guide firing, include
+`trigger.signal: botDetected` (or `dialogDetected`) in the frontmatter;
+omitting it disables the trigger.
+
+```yaml
+---
+category: pattern
+trigger.signal: botDetected
+icon: 🤖
+shortName: my bot guide
+---
+Custom guidance text for bot challenges.
+```
+
+Site guides and pattern guides live in **disjoint namespaces** — a site guide
+for `www.botdetection.com` does not collide with the `bot-detection` pattern
+guide; both fire when applicable.
+
 ### Viewing Guides
 
 ```text

package/node_modules/pi-lean-portal/__tests__/browser-toggle-profile.test.ts

@@ -10,7 +10,8 @@
 
 import { describe, it, expect, beforeEach, vi, type Mock } from "vitest";
 import { existsSync, readdirSync, statSync } from "node:fs";
-import type { ExtensionAPI } from "@earendil-works/pi-coding-agent";
+import { EventEmitter } from "node:events";
+import type { ExtensionAPI, EventBus } from "@earendil-works/pi-coding-agent";
 
 // Mock node:fs to control the filesystem for profile tests
 vi.mock("node:fs", () => ({
@@ -30,6 +31,7 @@ import { listProfiles, formatProfileList } from "../browser-profile.js";
 // ─── Helpers ─────────────────────────────────────────────────────
 
 function mockPi(tools?: string[]): ExtensionAPI {
+	const emitter = new EventEmitter();
 	const allTools = (tools ?? []).map((name) => ({ name }));
 	// Return a full active set by default
 	const active = tools ?? [];
@@ -40,6 +42,15 @@ function mockPi(tools?: string[]): ExtensionAPI {
 		appendEntry: vi.fn(),
 		registerCommand: vi.fn(),
 		on: vi.fn(),
+		get events() {
+			return {
+				emit: (channel: string, data: unknown) => emitter.emit(channel, data),
+				on: (channel: string, handler: (data: unknown) => void) => {
+					emitter.on(channel, handler);
+					return () => emitter.off(channel, handler);
+				},
+			} as EventBus;
+		},
 	} as unknown as ExtensionAPI;
 }