@underactive/pi-topping-web-tools 0.1.0

package/CHANGELOG.md

@@ -0,0 +1,29 @@
+# Changelog
+
+## [Unreleased]
+
+## [0.1.0] - 2026-08-17
+
+### Added
+
+- Initial bundle: `fetch_markdown` and `web_browser` pi extensions with a shared preapproved-host allowlist, merged from the standalone `pi-fetch-markdown-tool`, `pi-web-browser-tool`, and `pi-web-permissions` packages.
+- `web_browser`: `scroll` action (selector-based `scrollIntoView` or pixel-delta mouse wheel).
+- `web_browser`: `drag` action for drag-and-drop between two selectors.
+- `web_browser`: `upload_file` action to set files on an `<input type=file>`.
+- `web_browser`: `set_dialog_behavior` and `get_dialog_logs` actions to configure and inspect handling of JS `alert`/`confirm`/`prompt` dialogs (dismissed by default).
+- `web_browser`: `list_tabs` and `switch_tab` actions for multi-tab sessions; popups (`target=_blank`, `window.open`) automatically become the active tab.
+- `web_browser`: `frame` parameter on selector-based actions (click, type, hover, select_option, screenshot, get_text, wait_for, get_accessibility_snapshot, scroll, upload_file, drag) to target elements inside an `<iframe>`.
+- `web_browser`: selectors now also accept Playwright's `role=` and `text=` selector engines, not just CSS.
+- `fetch_markdown`: `offset` parameter to paginate past the 100K-character truncation limit; truncated responses report the offset to continue from.
+- `fetch_markdown`: readability article extraction (`@mozilla/readability` + `linkedom`) on HTML pages by default, stripping nav/footer boilerplate; the page title is restored as the leading heading, `raw: true` opts out, and non-article pages fall back to full-page conversion.
+- `fetch_markdown`: conditional revalidation of expired cache entries via `If-None-Match`/`If-Modified-Since`; a 304 reuses the cached markdown and refreshes the TTL.
+- `pdf_extract`: new tool extracting the text layer from remote (`url`) or local (`path`) PDFs via `unpdf`, which `fetch_markdown` and `web_fetch` treat as undisplayable binary content. Output carries `--- Page N ---` markers, `pages` selects a range (`"3"`, `"1-5"`, `"1,4,7-9"`), `offset` paginates past the 100K-character limit, and results are wrapped in `<untrusted-content>`. Local paths resolve through `realpath` before permission checks so symlinks cannot escape the working directory, and extraction is bounded by 25MB, 2000 pages, and a 30s timeout. Adds the `/clear-pdf-extract-cache` command.
+
+### Changed
+
+- `fetch_markdown`: raised the maximum HTTP body size from 10 MB to 25 MB so very large single-page documents (e.g. the WHATWG HTML spec) can be fetched and paginated with `offset`.
+- `fetch_markdown`: renamed the internal `sliceMarkdown` helper to `sliceContent` and the `getWithPermittedRedirects` header parameter to `extraHeaders`, so `pdf_extract` can reuse the same content-windowing and redirect logic.
+
+### Fixed
+
+- `fetch_markdown`: tool errors rendered as `Received NaNMB (undefined undefined)`; error results now fall back to the error text.

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Eric Sison
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,175 @@
+# @underactive/pi-topping-web-tools
+
+Web tools bundle for [pi](https://github.com/earendil-works/pi): keyless URL-to-markdown fetching, PDF text extraction, and headless Playwright browser automation, sharing one preapproved-host allowlist and permission model.
+
+## Install
+
+```bash
+pi install npm:@underactive/pi-topping-web-tools
+```
+
+The `web_browser` tool additionally requires Playwright's Chromium (~150MB):
+
+```bash
+npx playwright install chromium
+```
+
+## Why this extension
+
+Web work has two distinct needs: clean context from public sources and browser-level verification of the live experience. This bundle gives pi both, so an agent can move from a documentation question to an interactive check without stitching together unrelated extensions. Public-page retrieval is keyless, while browser automation remains available for the cases where a static fetch cannot answer the question.
+
+- **Keep research focused.** `fetch_markdown` converts public documentation, READMEs, and articles into Markdown, extracting the primary article instead of a page's surrounding navigation. Its cache makes iterative questions against the same source faster.
+- **Reach PDF-only sources.** `pdf_extract` reads the text layer of specifications, datasheets, and papers that markdown fetchers treat as undisplayable binary content, and that pi's `read` tool cannot parse.
+- **Validate the rendered product.** `web_browser` runs a fresh headless Chromium context for client-rendered pages, local development servers, and user journeys that static HTML cannot represent. It can inspect the accessibility tree, interact with controls, and surface console and failed-network diagnostics.
+- **Use the appropriate tool, not the heavier tool by default.** Fetch static public material first; move to browser automation when JavaScript, visual state, or user input is part of the question.
+- **Keep control over access.** The tools share a host allowlist and confirmation workflow; fetched page text is explicitly marked as untrusted, and browser sessions do not reuse the system browser profile.
+
+## Examples
+
+Give pi a task like the following; the extension supplies the web capabilities needed to complete it.
+
+### Research a dependency before changing code
+
+> Read `https://react.dev/reference/react/useEffect`. Summarize the cleanup behavior, identify the parts relevant to subscriptions, and cite the sections I should review before changing this component.
+
+`fetch_markdown` returns the page as focused Markdown, making it useful for documentation research without spending context on site chrome. Repeated follow-up questions can reuse the cached response.
+
+### Verify a local user journey
+
+> Open `http://localhost:3000`, set a 390×844 viewport, add an item to the cart, and report any failed network requests or console errors. Save a screenshot of the final state.
+
+`web_browser` can drive the local application as a user would, then collect the diagnostics that explain a failed interaction rather than only reporting that it failed.
+
+### Investigate a JavaScript-heavy page
+
+> Review the migration guide at `https://example.com/migration`. If the important content is not present in the static page, inspect the live page in the browser and return the breaking changes that affect this project.
+
+Start with `fetch_markdown` for efficient static content; use `web_browser` when the page depends on client-side rendering or requires interaction. The two tools let the agent choose the lowest-cost path that still produces reliable evidence.
+
+## Tools
+
+| Tool | Description |
+|------|-------------|
+| `fetch_markdown` | Fetch a public URL and return its content as markdown (cached, keyless) |
+| `pdf_extract` | Extract the text layer from a remote or local PDF, with page markers |
+| `web_browser` | Headless Chromium automation — navigate, click, type, screenshot, evaluate, logs, cookies |
+
+## `fetch_markdown`
+
+Complements pi's built-in search-backed `web_fetch` with a lightweight, keyless fetch path for public documentation, READMEs, and articles:
+
+- **Native fetch** — no API keys, no provider routing
+- **HTML → markdown** — [turndown](https://github.com/mixmark-io/turndown) conversion for readable agent context
+- **Readability extraction** — [@mozilla/readability](https://github.com/mozilla/readability) (via [linkedom](https://github.com/WebReflection/linkedom)) strips nav/footer/sidebar boilerplate from article pages by default; pass `raw: true` for the full page. Falls back to full-page conversion when a page doesn't look like an article
+- **15-minute LRU cache with revalidation** — faster repeat lookups; expired entries are revalidated with `If-None-Match`/`If-Modified-Since` when the server sent `ETag`/`Last-Modified`, so unchanged pages cost a 304 instead of a full download
+- **Strict redirect handling** — redirects that change host, scheme, or port, or that leave a preapproved path, return the redirect URL instead of being followed; call again to follow (`www.` prefixes are treated as the same host)
+- **Host confirmation** — prompts for non-preapproved hosts (session-scoped allow/deny)
+
+Usage notes:
+
+- HTTP URLs are upgraded to HTTPS automatically
+- Content is returned in 100K-character windows; truncated responses report the `offset` to pass for the next window
+- For GitHub repo content, prefer `web_fetch` or the `gh` CLI
+- JS-rendered SPAs may return empty markdown (static HTML only)
+- Authenticated or private pages (Google Docs, Confluence, Jira) are not supported
+
+Command: `/clear-fetch-markdown-cache` — clear the in-memory URL cache.
+
+## `pdf_extract`
+
+PDFs are otherwise unreachable: `fetch_markdown` and pi's `web_fetch` classify `application/pdf` as binary, and the `read` tool cannot parse it. This tool extracts the text layer via [unpdf](https://github.com/unjs/unpdf) (a serverless build of Mozilla's PDF.js, keyless and with no native dependencies).
+
+- **Remote or local** — pass `url` for a remote PDF or `path` for an absolute path or `file://` URL; provide exactly one
+- **Page markers** — output is delimited with `--- Page N ---` so pages can be cited
+- **Page selection** — `pages` accepts `"3"`, `"1-5"`, or `"1,4,7-9"`; defaults to every page
+- **15-minute LRU cache** — the whole parse is cached, so a later `pages` or `offset` call reuses it; local files are re-read when their modification time changes
+- **Host and file confirmation** — remote URLs use the shared allowlist; local files inside the working directory are read without prompting, anything outside prompts (session-scoped allow/deny)
+
+Usage notes:
+
+- Content is returned in 100K-character windows; truncated responses report the `offset` to pass for the next window
+- Scanned or image-only PDFs have no text layer and report that explicitly — there is no OCR
+- Limits: 25MB download, 2000 pages, and a 30s extraction timeout
+
+Command: `/clear-pdf-extract-cache` — clear the in-memory PDF text cache.
+
+## `web_browser`
+
+Single tool with an `action` parameter:
+
+| Action | Parameters | Description |
+|--------|------------|-------------|
+| `navigate` | `url` | Open a URL or `file://` path (HTTP upgraded to HTTPS except localhost/.local/private IPs) |
+| `screenshot` | `fullPage?`, `selector?`, `toFile?` | Capture PNG screenshot (base64 image, or temp file path when `toFile=true`) |
+| `click` | `selector` | Click a CSS selector |
+| `type` | `selector`, `text`, `mode?` | Fill (`mode=fill`, default) or key-by-key type (`mode=press`) into a selector |
+| `hover` | `selector` | Hover over a CSS selector (tooltips, hover-menus) |
+| `press` | `key` | Press a keyboard key globally (Enter, Escape, Tab, etc.) |
+| `select_option` | `selector`, `values` | Select option(s) in a `<select>` dropdown (comma-separated values) |
+| `set_viewport` | `width`, `height` | Resize the browser viewport (persists until changed or browser closes) |
+| `evaluate` | `script` | Run JavaScript in the page context (re-checks permission for the current page URL) |
+| `get_content` | — | Return raw page HTML |
+| `get_text` | `selector?` | Return visible text (layout-aware innerText; default selector `body`) |
+| `get_markdown` | — | Return page content as markdown (headings, lists, links) |
+| `wait_for` | `selector?`, `state?`, `networkidle?` | Wait for selector state and/or network idle |
+| `get_accessibility_snapshot` | `selector?` | Return ARIA accessibility tree (YAML) |
+| `get_console_logs` | — | Drain captured console/page errors |
+| `get_network_logs` | — | Drain captured failed requests and 4xx/5xx responses |
+| `get_cookies` | — | Return cookie metadata as JSON (name, domain, path, flags; values are redacted) |
+| `set_cookies` | `cookies` | Set cookies (each needs `url` or `domain`+`path`) |
+| `go_back` / `go_forward` / `reload` | — | Browser history navigation / reload |
+| `scroll` | `selector?`, `frame?`, `deltaX?`, `deltaY?` | Scroll an element into view (`selector`) or scroll by pixel deltas (default `deltaY` = viewport height) |
+| `drag` | `selector`, `targetSelector`, `frame?` | Drag-and-drop from `selector` to `targetSelector` |
+| `upload_file` | `selector`, `files`, `frame?` | Set files on an `<input type=file>` (absolute paths) |
+| `set_dialog_behavior` | `dialogAction`, `promptText?` | Configure how future JS dialogs (alert/confirm/prompt) are resolved for the rest of the session (default: dismiss) |
+| `get_dialog_logs` | — | Drain captured dialogs (type, message, default value, resolution) |
+| `list_tabs` | — | List open tabs (index, URL, title, active) |
+| `switch_tab` | `index` | Switch the active tab (from `list_tabs`) |
+| `close` | — | Close the browser |
+
+Optional `timeout` (ms, default 30000) applies to navigation and to selector-based actions (click, type, hover, select_option, screenshot, get_text, get_markdown, wait_for, get_accessibility_snapshot, scroll, drag, upload_file). Other actions use Playwright defaults.
+
+Selectors are CSS selectors, or Playwright's `role=` and `text=` selector engines (e.g. `role=button[name='Submit']`, `text=Sign in`). Pass `frame` (a CSS selector for the containing `<iframe>`) to scope click/type/hover/select_option/screenshot/get_text/wait_for/get_accessibility_snapshot/scroll/upload_file/drag into that frame.
+
+Opening a link with `target=_blank` or `window.open` creates a popup that automatically becomes the active tab — subsequent actions target it until you `switch_tab` back. Use `list_tabs` to see all open tabs.
+
+JS dialogs (`alert`/`confirm`/`prompt`) are dismissed by default so they never block automation; call `set_dialog_behavior` before the action that triggers one if you need `accept` instead (and, for `prompt`, a `promptText` value).
+
+Commands:
+
+- `/browser` — show open/closed status, current URL, title, tab count, console/network/dialog counts, and session-approved host count
+- `/browser-close` — force close the browser
+- `/browser-screenshot` — save a full-page screenshot to a temp PNG file
+
+Status line: a footer segment (`browser: example.com`) appears while the browser is open.
+
+## Security model (shared)
+
+- **Preapproved hosts**: common documentation and dev sites are allowed without prompting. The allowlist lives in `src/permissions.ts` and is shared by both tools.
+- **User confirmation**: other hosts prompt Allow once / Allow for this session / Deny.
+- **URL validation**: all tools reject embedded credentials and overlong URLs. `fetch_markdown` and `pdf_extract` additionally reject loopback, private, and link-local hosts and any non-HTTP(S) scheme on a `url`. `web_browser` accepts `localhost`, private IPs, and `file://` paths so it can drive local dev servers and local HTML — each still prompts for confirmation unless session-approved.
+- **Local file resolution**: `pdf_extract` resolves a `path` through `realpath` before any check, so a symlink pointing outside the working directory is treated as outside and prompts. Directories, FIFOs, and devices are rejected, and the size limit is enforced from `stat` before any bytes are read.
+- **Isolated browser context**: each session uses a fresh Chromium context (no shared profile/cookies with your system browser).
+- **Untrusted content boundary**: `fetch_markdown` results are wrapped in `<untrusted-content url="…">` tags, and `pdf_extract` results in `<untrusted-content source="…">`, so the model treats fetched page and document text as data, not instructions.
+- **PDF resource limits**: extraction is bounded by download size, page count, and a timeout. Because the bundled PDF.js runs on the event loop rather than a worker, the timeout bounds extraction across its await points but cannot interrupt a fully synchronous parse — the size and page caps are the primary defence.
+
+## Limitations
+
+- `web_browser` requires the Chromium download.
+- Screenshots consume significant context on vision models — prefer `get_text`, `get_markdown`, or `get_accessibility_snapshot`.
+- Network logs capture only failed requests and 4xx/5xx responses (not all traffic).
+- `pdf_extract` reads only an existing text layer; scanned documents need OCR, which is out of scope.
+
+### Popups and navigation
+
+Popups inherit the same security model as same-tab link clicks: opening a `target=_blank` link or calling `window.open` is not gated by the host allowlist, since it's JS-initiated navigation rather than a tool-driven `navigate` call. `evaluate` remains gated per active page URL — the permission check follows the active-tab cursor automatically, so switching to a popup and calling `evaluate` re-checks permission for that popup's URL.
+
+## Development
+
+Requires Node >= 22.19 (tests run TypeScript directly via `node --test`).
+
+```bash
+npm install
+npm run typecheck
+npm test          # browser tests require Chromium
+```

package/package.json

@@ -0,0 +1,63 @@
+{
+	"name": "@underactive/pi-topping-web-tools",
+	"version": "0.1.0",
+	"description": "Pi extensions for the web: fetch_markdown (URL to markdown, cached, keyless), pdf_extract (PDF text extraction) and web_browser (headless Playwright automation), sharing a preapproved-host allowlist.",
+	"keywords": ["pi-package", "pi-extension"],
+	"license": "MIT",
+	"author": "Eric Sison",
+	"repository": {
+		"type": "git",
+		"url": "git+https://github.com/underactive/pi-topping-web-tools.git"
+	},
+	"bugs": {
+		"url": "https://github.com/underactive/pi-topping-web-tools/issues"
+	},
+	"homepage": "https://github.com/underactive/pi-topping-web-tools#readme",
+	"type": "module",
+	"engines": {
+		"node": ">=22.19.0"
+	},
+	"publishConfig": {
+		"access": "public"
+	},
+	"files": [
+		"src/**/*.ts",
+		"README.md",
+		"LICENSE",
+		"CHANGELOG.md"
+	],
+	"scripts": {
+		"typecheck": "tsc --noEmit",
+		"test": "node --test test/*.test.ts"
+	},
+	"pi": {
+		"extensions": [
+			"./src/fetch-markdown.ts",
+			"./src/pdf-extract.ts",
+			"./src/web-browser/index.ts"
+		]
+	},
+	"dependencies": {
+		"@mozilla/readability": "^0.6.0",
+		"linkedom": "^0.18.13",
+		"lru-cache": "^11.0.0",
+		"playwright": "^1.52.0",
+		"turndown": "^7.2.0",
+		"unpdf": "^1.8.1"
+	},
+	"peerDependencies": {
+		"@earendil-works/pi-ai": "*",
+		"@earendil-works/pi-coding-agent": "*",
+		"@earendil-works/pi-tui": "*",
+		"typebox": "*"
+	},
+	"devDependencies": {
+		"@earendil-works/pi-ai": "0.82.1",
+		"@earendil-works/pi-coding-agent": "0.82.1",
+		"@earendil-works/pi-tui": "0.82.1",
+		"@types/node": "^24.0.0",
+		"@types/turndown": "^5.0.5",
+		"typebox": "^1.3.3",
+		"typescript": "^5.9.3"
+	}
+}