pi-read-page 0.1.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Shuqian
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,233 @@
+# pi-read-page
+
+Let [pi](https://github.com/earendil-works/pi-coding-agent) read webpages through your local browser and return Markdown.
+
+## What it provides
+
+- One read-only Agent tool: `read-page`.
+- Local Chrome/Chromium rendering.
+- Manual handoff for login/captcha/blocked states.
+- Markdown output with pagination and cache.
+- Defensive defaults for untrusted webpages and private-network access.
+
+## Requirements
+
+- pi.
+- A local Chrome/Chromium browser.
+- Bun only if you are developing or running tests locally.
+
+`pi-read-page` uses `playwright-core`; it does not download a browser. By default it launches the `chrome` channel. Set `READ_PAGE_CHROME_PATH` or `READ_PAGE_BROWSER_CHANNEL` if needed.
+
+## Installation
+
+Install from npm:
+
+```bash
+pi install npm:pi-read-page
+```
+
+Try it for one pi run without installing:
+
+```bash
+pi -e npm:pi-read-page
+```
+
+Install from GitHub if you want the latest repository version:
+
+```bash
+pi install https://github.com/Sukitly/pi-read-page
+```
+
+Use a local checkout:
+
+```bash
+git clone https://github.com/Sukitly/pi-read-page.git
+cd pi-read-page
+bun install
+pi -e .
+```
+
+## Usage
+
+Ask pi to read a URL:
+
+```text
+Read https://example.com
+```
+
+The extension registers one Agent-facing tool:
+
+```text
+read-page(url, offset?, limit?, refresh?, preserveQuery?)
+```
+
+Parameters:
+
+| Parameter | Default | Description |
+| --- | --- | --- |
+| `url` | required | HTTP or HTTPS URL to read. |
+| `offset` | `1` | 1-based line offset for pagination. |
+| `limit` | `300` | Number of lines to return. Maximum `1000`. |
+| `refresh` | `false` | Force browser re-extraction and overwrite cache. |
+| `preserveQuery` | `false` | Preserve URL query parameters. By default query params are stripped for canonical cache keys. |
+
+Use the returned `Next offset` to continue reading long pages.
+
+## How extraction works
+
+```text
+URL normalization and private-network policy
+  -> headed Playwright browser
+  -> DOMContentLoaded + network idle wait
+  -> final URL private-network policy
+  -> read-only lazy-load scroll
+  -> open shadow-root flattening
+  -> URL absolutization
+  -> Defuddle HTML/Markdown extraction
+  -> confidence and handoff detection
+  -> local cache write
+  -> paginated Markdown output
+```
+
+If the page appears to require a real user action, pi shows a confirmation prompt and leaves the headed browser open. Complete the login/captcha/manual navigation in that browser, then confirm in pi. The same browser page is settled and extracted again. After the tool call completes, the page and browser context are closed.
+
+## Cache
+
+Successful browser extractions are cached under:
+
+```text
+~/.pi/agent/caches/read-page
+```
+
+Cache behavior:
+
+- Normal TTL: 30 days.
+- User-action TTL: 1 day.
+- Cache files: `content.md` and `meta.json`.
+- Writes are atomic.
+- Cached Markdown is sha256-verified on load.
+- If refresh/extraction fails and a cache entry exists, the tool returns cached content with an explicit `refresh-failed-fresh` or `stale-fallback` status.
+
+## Security model
+
+`read-page` treats webpages as untrusted external content.
+
+- The output includes a security notice and document boundary.
+- The Agent is instructed not to follow instructions from the page unless the user explicitly asks.
+- Private/local hosts and IPs are blocked by default.
+- Browser automation is read-only: it may navigate, wait, scroll, extract DOM, and cache content.
+- The extension does not expose browser mutation/control tools to the Agent.
+- User handoff is only used for actionable captcha, blocked/interstitial, or explicit login-wall states.
+
+To intentionally allow private/local network URLs:
+
+```bash
+READ_PAGE_ALLOW_PRIVATE_NETWORK=1 pi
+```
+
+## Configuration
+
+Optional environment variables:
+
+| Variable | Default | Description |
+| --- | --- | --- |
+| `READ_PAGE_CHROME_PATH` | unset | Explicit Chrome/Chromium executable path. |
+| `READ_PAGE_BROWSER_CHANNEL` | `chrome` | Playwright browser channel. |
+| `READ_PAGE_PROFILE_DIR` | `~/.pi/agent/read-page/browser-profile` | Persistent browser profile directory. |
+| `READ_PAGE_DISABLE_TEMP_PROFILE_FALLBACK` | unset | Set to `1` to fail instead of using a temporary profile when the persistent profile is locked. |
+| `READ_PAGE_ALLOW_PRIVATE_NETWORK` | unset | Set to `1` to allow private/local network access. |
+| `READ_PAGE_PARSE_TIMEOUT_MS` | `8000` | Defuddle parse timeout before sync fallback. |
+| `READ_PAGE_DEFUDDLE_ASYNC` | unset | Set to `1` to allow Defuddle third-party async extraction. |
+| `READ_PAGE_DEFUDDLE_DEBUG` | unset | Set to `1` to include Defuddle debug information. |
+
+## Development
+
+Install dependencies:
+
+```bash
+bun install
+```
+
+Run deterministic checks:
+
+```bash
+bun run lint
+bun test
+```
+
+Run the browser integration test:
+
+```bash
+bun run integration -- https://example.com
+```
+
+The integration test opens a real browser, extracts the page, prints extraction metadata, and closes the browser context.
+
+## Publishing
+
+Pi package catalog entries are discovered from public npm packages with the `pi-package` keyword.
+
+Before publishing:
+
+```bash
+bun run lint
+bun test
+npm pack --dry-run
+```
+
+Publish:
+
+```bash
+npm login
+npm publish --access public
+```
+
+After publishing, install with:
+
+```bash
+pi install npm:pi-read-page
+```
+
+## Project layout
+
+```text
+extensions/pi-read-page.ts      extension entrypoint
+src/tools/read-page.ts          tool orchestration, output formatting, TUI rendering
+src/browser/                    browser lifecycle, extraction, handoff, confidence
+src/cache/cache.ts              cache, pagination, checksums
+src/security/url-policy.ts      URL normalization and private-network policy
+test/                           deterministic unit tests
+scripts/integration-read-page.ts browser integration runner
+```
+
+## Troubleshooting
+
+### Chrome is not found
+
+Install Google Chrome/Chromium, or set:
+
+```bash
+READ_PAGE_CHROME_PATH=/path/to/chrome pi
+```
+
+### Login state is missing
+
+By default the extension uses a persistent profile at:
+
+```text
+~/.pi/agent/read-page/browser-profile
+```
+
+If that profile is already locked by another browser process, `read-page` falls back to a temporary profile. The tool output will include a warning when this happens.
+
+### Query parameters were removed
+
+Set `preserveQuery: true` when query parameters are required for the page content, such as search results, filters, or app/detail pages.
+
+### Localhost or private IP is blocked
+
+This is intentional. Use `READ_PAGE_ALLOW_PRIVATE_NETWORK=1` only when you explicitly want to read local/private services.
+
+## License
+
+MIT. See [LICENSE](LICENSE).

package/extensions/pi-read-page.ts

@@ -0,0 +1,11 @@
+import type { ExtensionAPI } from "@earendil-works/pi-coding-agent";
+import { closeBrowser } from "../src/browser/browser-manager";
+import { registerReadPageTool } from "../src/tools/read-page";
+
+export default function readPageExtension(pi: ExtensionAPI) {
+  registerReadPageTool(pi);
+
+  pi.on("session_shutdown", async () => {
+    await closeBrowser();
+  });
+}

package/package.json

@@ -0,0 +1,65 @@
+{
+  "name": "pi-read-page",
+  "version": "0.1.0",
+  "description": "Read webpages through a local browser and return Markdown for Pi coding agent.",
+  "type": "module",
+  "license": "MIT",
+  "author": "Sukitly",
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/Sukitly/pi-read-page.git"
+  },
+  "homepage": "https://github.com/Sukitly/pi-read-page#readme",
+  "bugs": {
+    "url": "https://github.com/Sukitly/pi-read-page/issues"
+  },
+  "keywords": [
+    "pi-package",
+    "pi-extension",
+    "pi",
+    "read-page",
+    "browser",
+    "markdown"
+  ],
+  "files": [
+    "extensions",
+    "src",
+    "README.md",
+    "LICENSE",
+    "package.json"
+  ],
+  "publishConfig": {
+    "access": "public"
+  },
+  "pi": {
+    "extensions": [
+      "./extensions/pi-read-page.ts"
+    ]
+  },
+  "scripts": {
+    "typecheck": "tsc --noEmit",
+    "test": "vitest run",
+    "integration": "tsx scripts/integration-read-page.ts",
+    "lint": "bun run typecheck && bunx --bun @biomejs/biome check --error-on-warnings --write ."
+  },
+  "dependencies": {
+    "defuddle": "^0.18.1",
+    "linkedom": "^0.18.12",
+    "playwright-core": "^1.60.0"
+  },
+  "peerDependencies": {
+    "@earendil-works/pi-coding-agent": "*",
+    "@earendil-works/pi-tui": "*",
+    "typebox": "*"
+  },
+  "devDependencies": {
+    "@biomejs/biome": "2.5.0",
+    "@earendil-works/pi-coding-agent": "^0.79.2",
+    "@earendil-works/pi-tui": "^0.79.2",
+    "@types/node": "^25.9.3",
+    "tsx": "^4.22.4",
+    "typebox": "1.1.38",
+    "typescript": "^6.0.3",
+    "vitest": "^4.1.8"
+  }
+}

package/src/browser/browser-manager.ts

@@ -0,0 +1,329 @@
+import { mkdir, mkdtemp, rm } from "node:fs/promises";
+import { homedir, tmpdir } from "node:os";
+import { join, resolve } from "node:path";
+import { type BrowserContext, chromium, type Page } from "playwright-core";
+import { assertHttpUrlAllowed, isHttpLikeUrl } from "../security/url-policy";
+
+type ManagedBrowserContext = {
+  context: BrowserContext;
+  profileDir: string;
+  temporaryProfileDir?: string;
+};
+
+type BrowserAutomation = Pick<typeof chromium, "launchPersistentContext">;
+
+let browserAutomation: BrowserAutomation = chromium;
+let managedContext: ManagedBrowserContext | undefined;
+let managedContextPromise: Promise<ManagedBrowserContext> | undefined;
+let contextGeneration = 0;
+
+function expandHome(path: string): string {
+  if (path === "~") return homedir();
+  if (path.startsWith("~/")) return resolve(homedir(), path.slice(2));
+  return path;
+}
+
+function defaultProfileDir(): string {
+  return resolve(homedir(), ".pi", "agent", "read-page", "browser-profile");
+}
+
+async function getContext(signal?: AbortSignal): Promise<BrowserContext> {
+  throwIfAborted(signal, "read-page aborted before opening browser");
+  if (managedContext) return managedContext.context;
+
+  const generation = contextGeneration;
+  if (!managedContextPromise) managedContextPromise = createManagedContext();
+  const startup = managedContextPromise;
+
+  try {
+    const created = await abortable(
+      startup,
+      signal,
+      "read-page aborted while starting browser",
+      closeManagedContext,
+    );
+
+    if (generation !== contextGeneration) {
+      throw new Error("read-page browser context closed during startup");
+    }
+
+    managedContext = created;
+    return created.context;
+  } catch (error) {
+    if (managedContextPromise === startup) managedContextPromise = undefined;
+    throw error;
+  }
+}
+
+async function createManagedContext(): Promise<ManagedBrowserContext> {
+  const profileDir = expandHome(
+    process.env.READ_PAGE_PROFILE_DIR || defaultProfileDir(),
+  );
+  await mkdir(profileDir, { recursive: true });
+
+  try {
+    return {
+      context: await launchPersistent(profileDir),
+      profileDir,
+    };
+  } catch (error) {
+    if (
+      !isProfileInUseError(error) ||
+      process.env.READ_PAGE_DISABLE_TEMP_PROFILE_FALLBACK === "1"
+    ) {
+      throw error;
+    }
+
+    const temporaryProfileDir = await mkdtemp(
+      join(tmpdir(), "read-page-profile-"),
+    );
+    try {
+      return {
+        context: await launchPersistent(temporaryProfileDir),
+        profileDir: temporaryProfileDir,
+        temporaryProfileDir,
+      };
+    } catch (tempError) {
+      await removeTemporaryProfile(temporaryProfileDir);
+      throw tempError;
+    }
+  }
+}
+
+async function launchPersistent(profileDir: string): Promise<BrowserContext> {
+  const browserContext = await browserAutomation.launchPersistentContext(
+    profileDir,
+    {
+      headless: false,
+      channel: process.env.READ_PAGE_BROWSER_CHANNEL || "chrome",
+      executablePath: process.env.READ_PAGE_CHROME_PATH || undefined,
+      viewport: null,
+      args: ["--disable-blink-features=AutomationControlled"],
+    },
+  );
+
+  try {
+    await installNetworkPolicy(browserContext);
+    return browserContext;
+  } catch (error) {
+    await browserContext.close().catch(() => undefined);
+    throw error;
+  }
+}
+
+async function installNetworkPolicy(
+  browserContext: BrowserContext,
+): Promise<void> {
+  await browserContext.route("**/*", async (route) => {
+    const url = route.request().url();
+    if (!isHttpLikeUrl(url)) {
+      await route.continue();
+      return;
+    }
+
+    try {
+      await assertHttpUrlAllowed(url);
+      await route.continue();
+    } catch {
+      await route.abort("blockedbyclient");
+    }
+  });
+}
+
+function isProfileInUseError(error: unknown): boolean {
+  const message = error instanceof Error ? error.message : String(error);
+  return /existing browser session|profile is already in use|user data directory is already in use/i.test(
+    message,
+  );
+}
+
+export function setBrowserAutomationForTest(
+  automation: BrowserAutomation | undefined,
+): void {
+  browserAutomation = automation ?? chromium;
+}
+
+export function getBrowserRuntimeInfo() {
+  return {
+    profileDir: managedContext?.profileDir,
+    usingTemporaryProfile: managedContext?.temporaryProfileDir !== undefined,
+  };
+}
+
+export async function closeBrowser(): Promise<void> {
+  contextGeneration += 1;
+  const current = managedContext;
+  const startup = managedContextPromise;
+  managedContext = undefined;
+  managedContextPromise = undefined;
+
+  if (current) await closeManagedContext(current);
+  if (!startup) return;
+
+  const created = await startup.catch(() => undefined);
+  if (created && created.context !== current?.context) {
+    await closeManagedContext(created);
+  }
+}
+
+export async function openPage(
+  url: string,
+  signal?: AbortSignal,
+): Promise<Page> {
+  throwIfAborted(signal, "read-page aborted before opening browser");
+
+  await abortable(
+    assertHttpUrlAllowed(url),
+    signal,
+    "read-page aborted while validating URL",
+  );
+  const browserContext = await getContext(signal);
+  const page = await abortable(
+    browserContext.newPage(),
+    signal,
+    "read-page aborted while opening page",
+    async (createdPage) => {
+      await createdPage.close().catch(() => undefined);
+    },
+  );
+
+  let shouldClosePage = true;
+  try {
+    await abortable(
+      page.goto(url, { waitUntil: "domcontentloaded", timeout: 45_000 }),
+      signal,
+      "read-page aborted while navigating page",
+    );
+    await abortable(
+      assertHttpUrlAllowed(page.url()),
+      signal,
+      "read-page aborted while validating final URL",
+    );
+    await settlePage(page, signal);
+    await abortable(
+      assertHttpUrlAllowed(page.url()),
+      signal,
+      "read-page aborted while validating settled URL",
+    );
+    shouldClosePage = false;
+    return page;
+  } finally {
+    if (shouldClosePage) await page.close().catch(() => undefined);
+  }
+}
+
+export async function settlePage(
+  page: Page,
+  signal?: AbortSignal,
+): Promise<void> {
+  throwIfAborted(signal, "read-page aborted while waiting for page");
+
+  await abortable(
+    page.waitForLoadState("networkidle", { timeout: 8_000 }),
+    signal,
+    "read-page aborted while waiting for page",
+  ).catch((error) => {
+    if (isAbortError(error)) throw error;
+  });
+  await abortable(
+    page.waitForTimeout(750),
+    signal,
+    "read-page aborted while waiting for page",
+  );
+
+  // Read-only lazy-load trigger. No clicks, no typing, no submission.
+  await abortable(
+    page.evaluate(async () => {
+      const delay = (ms: number) =>
+        new Promise((resolve) => setTimeout(resolve, ms));
+      const maxY = Math.max(
+        document.body.scrollHeight,
+        document.documentElement.scrollHeight,
+      );
+      const step = Math.max(600, Math.floor(window.innerHeight * 0.8));
+      for (let y = 0; y < maxY; y += step) {
+        window.scrollTo(0, y);
+        await delay(80);
+      }
+      window.scrollTo(0, 0);
+    }),
+    signal,
+    "read-page aborted while preparing page",
+  ).catch((error) => {
+    if (isAbortError(error)) throw error;
+  });
+
+  await abortable(
+    page.waitForTimeout(300),
+    signal,
+    "read-page aborted while waiting for page",
+  );
+}
+
+async function closeManagedContext(
+  browserContext: ManagedBrowserContext,
+): Promise<void> {
+  await browserContext.context.close().catch(() => undefined);
+  if (browserContext.temporaryProfileDir) {
+    await removeTemporaryProfile(browserContext.temporaryProfileDir);
+  }
+}
+
+async function removeTemporaryProfile(profileDir: string): Promise<void> {
+  await rm(profileDir, { recursive: true, force: true }).catch(() => undefined);
+}
+
+function throwIfAborted(
+  signal: AbortSignal | undefined,
+  message: string,
+): void {
+  if (signal?.aborted) throw abortError(message);
+}
+
+async function abortable<T>(
+  promise: Promise<T>,
+  signal: AbortSignal | undefined,
+  message: string,
+  cleanup?: (value: T) => Promise<void> | void,
+): Promise<T> {
+  if (!signal) return promise;
+
+  let aborted = signal.aborted;
+  let removeAbortListener: () => void = () => undefined;
+  const trackedPromise = promise.then((value) => {
+    if (aborted && cleanup) {
+      void Promise.resolve(cleanup(value)).catch(() => undefined);
+    }
+    return value;
+  });
+  void trackedPromise.catch(() => undefined);
+
+  if (aborted) throw abortError(message);
+
+  const abortPromise = new Promise<never>((_resolve, reject) => {
+    const onAbort = () => {
+      aborted = true;
+      reject(abortError(message));
+    };
+    signal.addEventListener("abort", onAbort, { once: true });
+    removeAbortListener = () => {
+      signal.removeEventListener("abort", onAbort);
+    };
+  });
+
+  try {
+    return await Promise.race([trackedPromise, abortPromise]);
+  } finally {
+    removeAbortListener();
+  }
+}
+
+function abortError(message: string): Error {
+  const error = new Error(message);
+  error.name = "AbortError";
+  return error;
+}
+
+function isAbortError(error: unknown): boolean {
+  return error instanceof Error && error.name === "AbortError";
+}