@gavin-hu/my-pi-agent 0.18.0 → 0.19.0

package/{extensions/web-access/README.md → docs/extensions/web-access.md}

@@ -3,53 +3,57 @@
 One extension, two tools, keyless and dependency-free (native `fetch`, no
 `curl`): `web_search` runs general web search through a pluggable provider
 (keyless DuckDuckGo by default; optional SearXNG or Brave), and `web_fetch`
-reads a URL as text. Both are `direct` and active by default, and
-read the `web-access` section of the shared `bytefly.json`.
+reads a URL as text. Both are `direct`, active on registration, and read the
+`web-access` section of the shared `bytefly.json`.
 
-```
+## Capabilities
+
+Status records what a reader can rely on: **Shipped** is in a tagged release, **Unreleased** is complete in the current tree but not yet tagged, and **Partial** has one released part and one unreleased part. Rows are ordered by user value, highest first.
+
+| Capability | How | What you get | Status |
+|---|---|---|---|
+| Web search | `web_search` | Titles, URLs, snippets, and a direct answer when the provider has one. | Shipped |
+| Page reading | `web_fetch` | A URL's readable text, with links kept as `[text](url)`. | Shipped |
+| Structured bodies | `web_fetch` | A JSON, CSV, or XML body arrives as an aligned table or an indented tree instead of the raw blob. | Unreleased |
+| Write-style fetch | `web_fetch` (`method`, `headers`, `body`) | POST with custom headers and a bounded body, so form and GraphQL endpoints are reachable; a body implies POST. | Shipped |
+| Paged output | `web_fetch`, `startIndex`, `maxChars` | A long page arrives in budgeted slices and names the offset that continues it. | Shipped |
+| Passage search | `web_fetch`, `find`, `contextChars` | Matching passages instead of the whole page, with a note when a cap hid the rest. | Shipped |
+| Private-address refusal | `web_fetch`, `fetch.allowPrivateHosts` | Loopback, private, link-local, and metadata targets are refused on the first request and on every redirect hop. | Shipped |
+| Charset-aware decoding | `web_fetch` | A GBK/GB18030 or `<meta charset>` page reads as text instead of replacement characters. | Shipped |
+| Page cache | `web_fetch`, `refresh`, `fetch.cacheEnabled` | Paging and `find` reuse one fetch, and `refresh: true` bypasses a stale entry. | Shipped |
+| PDF extraction | `web_fetch`, `fetch.pdfEnabled` | A PDF becomes text once the optional `unpdf` package is installed. | Shipped |
+| Provider choice | `search.provider`, `search.endpoint`, `search.apiKeyEnv` | The keyless default works with no configuration; SearXNG or Brave add full result lists. | Shipped |
+| Structured result | `nextIndex`, `matchesTruncated`, `cached` | A caller can continue paging and tell a capped match set from a complete one without parsing the text. | Shipped |
+| Politeness throttle | `minIntervalMs` | Search requests are spaced instead of hammering a provider on a rapid series of calls. | Shipped |
+
+## Quickstart
+
+```bash
 pi --extension ./extensions/web-access    # load just this extension
 pi -e .                                   # load the whole @gavin-hu/my-pi-agent package
 pi install ./                             # or install the package
 ```
 
-## Tool rendering
+```text
+web_search({ query: "pi coding agent" })  # keyless, no configuration
+web_fetch({ url: "https://pi.dev/" })     # the page as readable text
+```
 
-How the `web_search` and `web_fetch` calls and their results look in the transcript — the call line as a headline and the body indented under it:
+## Tools
 
 <img src="../../assets/tools/web_search.png" alt="The web_search tool's call and result: the call line and an indented body with the query and the ranked results" width="100%">
 
 <img src="../../assets/tools/web_fetch.png" alt="The web_fetch tool's call and result: the call line and an indented body with the URL, title, and fetched text preview" width="100%">
 
-## What it does
-
-- Registers `web_search`, general web search through a pluggable provider:
-  keyless DuckDuckGo by default, or a self-hosted SearXNG instance / keyed Brave
-  search. Returns titles, URLs, snippets, and a direct answer when available,
-  always with no configuration required.
-- Registers `web_fetch`, which fetches a URL (GET or POST) and
-  returns readable text with links kept as `[text](url)`.
-- Decodes by the byte-order mark, the declared `Content-Type` charset, or the
-  page's own `<meta charset>`, so a GBK/GB18030 page reads as text instead of
-  replacement characters.
-- Pages long output and reports the next `startIndex` (`nextIndex` in
-  `structuredContent`); `find` returns matching passages instead of the whole
-  page, and says when the match cap hid further hits.
-- Re-validates every redirect hop against private/internal addresses, and
-  refuses transport headers and oversized bodies.
-- Optionally extracts PDFs (`unpdf`); the package loads lazily and is never
-  required.
-- Caches fetched pages in one in-process cache per registration, so paging and
-  `find` do not refetch; the cache is cleared on `session_start` and
-  `session_shutdown`.
-
-## Tools
+Both tools are `direct` (the default) and active on registration; neither needs
+configuration to run.
 
 | Tool | What it does |
 |---|---|
 | `web_search` | General web search; keyless by default, SearXNG or Brave optional. |
 | `web_fetch` | Fetch a URL and return readable text, pageable and searchable. |
 
-```
+```text
 web_search({ query: "pi coding agent" })
 web_fetch({ url: "https://pi.dev/" })
 web_fetch({ url: "https://pi.dev/", find: ["documentation", "extensions"] })
@@ -68,7 +72,7 @@ flight it draws a railed `Searching… 8s` or `Fetching example.com… 3s` line,
 a finished call appends its duration. A failed page or a host-rejected call
 renders one sanitized `Error:` line in the error role.
 
-```
+```text
 web_search "pi coding agent" (max 8)
 
 via searxng · 8 results · 2.1s
@@ -80,7 +84,7 @@ Answer: Pi is a coding agent harness …
 +3 more (ctrl+o to expand)
 ```
 
-```
+```text
 web_fetch https://pi.dev/
 Pi — a coding-agent component · 200 · 12k chars (cached) · 3.4s
 ```
@@ -90,9 +94,10 @@ which URL supplied the text.
 
 ## Configuration
 
-Merged from the `web-access` section of the global `bytefly.json` and the
-project `<cwd>/.pi/bytefly.json` (project wins). Missing or malformed documents
-and unknown keys are ignored.
+The `web-access` section of the shared `bytefly.json`: `search` configures
+`web_search`, `fetch` configures `web_fetch`. Resolution rules are in
+[`../configuration.md#web-access--search-and-fetch`](../configuration.md#web-access--search-and-fetch),
+which also lists every key with its default and accepted values.
 
 ```jsonc
 // bytefly.json → "web-access"
@@ -183,7 +188,7 @@ whether it came from an explicit choice, the `auto` rule, or the keyless default
 the API-key variable when the provider takes one and no literal `apiKey` is set,
 and whether that variable is present; and, only on an explicit `refresh`, whether
 the configured `endpoint` answers. The key is never reported, only its variable
-name. See [`bytefly`](../bytefly/README.md#what-it-does) for the report.
+name. See [`bytefly`](./bytefly.md#capabilities) for the report.
 
 | Parameter | Type | Notes |
 |---|---|---|
@@ -199,11 +204,16 @@ web_search({ query: "python list comprehension", maxResults: 3 })
 ## `web_fetch`
 
 GETs (or POSTs) an http(s) URL and returns readable text. HTML becomes
-plain text with links kept as `[text](url)`; JSON/text/XML is returned as-is;
-PDFs are text-extracted when the optional `unpdf` package is installed; other
-binary content is reported by type and size. Redirects are followed manually and
-**every hop is re-validated**, so a public URL cannot redirect into a private
-address. Loopback, private, link-local, and cloud-metadata addresses are refused.
+plain text with links kept as `[text](url)`; a JSON, CSV, or XML body becomes a
+table or an indented tree ([Structured bodies](#structured-bodies)); other textual
+content is returned as-is; PDFs are text-extracted when the optional `unpdf`
+package is installed; other binary content is reported by type and size. Bodies
+are decoded by the
+byte-order mark, the declared `Content-Type` charset, or the page's own
+`<meta charset>`, so a GBK/GB18030 page reads as text instead of replacement
+characters. Redirects are followed manually and **every hop is re-validated**, so
+a public URL cannot redirect into a private address. Loopback, private,
+link-local, and cloud-metadata addresses are refused.
 
 | Parameter | Type | Notes |
 |---|---|---|
@@ -237,6 +247,32 @@ web_fetch({ url: "https://pi.dev/", find: ["documentation", "extensions"] })
 web_fetch({ url: "https://api.example/graphql", method: "POST", headers: { "Content-Type": "application/json" }, body: '{"query":"{ viewer { login } }"}' })
 ```
 
+### Structured bodies
+
+A JSON, CSV, or XML body is rendered as text instead of the blob it downloaded: an
+array of records becomes a table with the union of their keys and padded columns,
+an object becomes indented `key: value` lines, comma-separated rows become the same
+table, and XML becomes an indented tree with attributes as `@name`. The declared
+content type decides first; a body that declares nothing usable (`text/plain`) is
+sniffed only when it has a shape — an opening brace, a uniform comma count, a root
+tag — and anything that does not parse keeps its raw form, so a malformed document
+is never made harder to read than it already was.
+
+```text
+URL: https://api.example/users
+Status: 200 (application/json)
+
+id  name   role
+1   Ada    admin
+2   Grace  user
+```
+
+Rendering happens once per fetch, before the result is cached, so `startIndex`,
+`find`, and a second call all read the same string. Two bounds keep it honest: a
+body past `MAX_SOURCE_CHARS` is returned raw, because a paged raw body beats a
+rendered one, and the rendered text stops at `MAX_RENDERED_CHARS` with a note saying
+so — both constants live in `fetch/structured.ts`.
+
 ### Optional dependencies
 
 PDF extraction needs a package that is not installed by default; it is loaded
@@ -259,6 +295,11 @@ the content type and size.
 - **Heuristic extraction**: `web_fetch` strips scripts/styles/nav/footer and
   prefers `<main>`/`<article>`, but is not a full readability engine. PDF
   extraction is optional (`unpdf`) and off unless installed.
+- **A structured body is rendered by a reader, not a parser.** JSON, CSV, and XML
+  are read well enough to show a shape — CSV by RFC 4180 quoting, XML by balancing
+  tags — but a body it cannot parse keeps its raw form rather than a partial one,
+  and the `text/plain` sniff is a heuristic: a page of prose with a uniform comma
+  in every line reads as a one-row table.
 - **`find` output is budgeted.** With `maxMatches: 50` and `contextChars: 2000`
   the passages alone would exceed `maxOutputChars`; the result shows the passages
   that fit and names the knob to change.
@@ -296,11 +337,11 @@ the content type and size.
   `config.ts`, `schema.ts`, `types.ts`, `format.ts`, and `tool.ts`, and do not
   import each other; `index.ts` only registers. Shared text and HTML helpers
   live at the top level (`text.ts`, `html.ts`, `transcript.ts`, `http.ts`).
-- **Shared transport.** [`http.ts`](./http.ts) holds the fetch runner
+- **Shared transport.** [`http.ts`](../../extensions/web-access/http.ts) holds the fetch runner
   (`HttpRunner`, `createFetchRunner`, HTTP error types), used by both halves and
   injectable for tests. It reads the body through a byte-counting stream, so
   `maxBytes` stops a chunked or compressed response before it is buffered whole.
-- **Charset-aware decoding.** [`charset.ts`](./charset.ts) is the one place a
+- **Charset-aware decoding.** [`charset.ts`](../../extensions/web-access/charset.ts) is the one place a
   byte body becomes text: BOM first, then the declared `Content-Type` charset,
   then the page's own `<meta charset>`, then UTF-8. An unknown label falls back
   to UTF-8 instead of failing the fetch.
@@ -311,9 +352,9 @@ the content type and size.
   lazily and falls back to a binary note when it is absent.
 - **Per-hop SSRF validation.** Redirects are followed with `redirect: "manual"`
   under one shared deadline, re-resolving and re-checking each `Location`.
-- **Pluggable search providers.** [`registry.ts`](./search/registry.ts) holds a
+- **Pluggable search providers.** [`registry.ts`](../../extensions/web-access/search/registry.ts) holds a
   value-only provider list (`duckduckgo`, `searxng`, `brave`) and a pure
-  `resolveProvider`; [`search.ts`](./search/search.ts) owns selection, key
+  `resolveProvider`; [`search.ts`](../../extensions/web-access/search/search.ts) owns selection, key
   resolution, a per-instance throttle, and error mapping. A provider with no
   answer and no results reports `provider: "none"`.
 - **Find-in-page.** `exact` is case-sensitive, `insensitive` lowercases both
@@ -328,17 +369,17 @@ the content type and size.
   truncation note against `maxChars`, so a paged page always carries its
   `startIndex` hint inside the limit. `formatMatches` applies the same rule to
   `find`: it includes the passages that fit and names the knob that hid the rest.
-- **Code points without a copy.** [`text.ts`](./text.ts) counts and slices by
+- **Code points without a copy.** [`text.ts`](../../extensions/web-access/text.ts) counts and slices by
   walking the string in place, so a multi-megabyte page is never expanded into an
   array of one-character strings on every call.
 - **One JSON boundary per concern.** `search/json.ts` owns `requestJson` (status
   check and parse error) and `collectResults` (title + http(s) URL + de-dupe), so
   the three providers cannot drift on either.
-- **Page cache.** [`fetch/cache.ts`](./fetch/cache.ts) is a factory
+- **Page cache.** [`fetch/cache.ts`](../../extensions/web-access/fetch/cache.ts) is a factory
   (`createPageCache`) holding an LRU/TTL map keyed by requested URL, so paging and
   find are a single fetch; entries evict by count and total bytes, and one
   instance belongs to one registration.
-- **Transcript sanitization and reuse.** [`transcript.ts`](./transcript.ts)
+- **Transcript sanitization and reuse.** [`transcript.ts`](../../extensions/web-access/transcript.ts)
   holds `oneLine`, shared by both tools to collapse untrusted web text to one
   sanitized, optionally clipped line; the renderers reuse
   `context.lastComponent` instead of allocating each render.
@@ -357,6 +398,7 @@ the content type and size.
 | `search/tool.ts` | `web_search` definition, activation, rendering |
 | `search/provider.ts` | Provider interface (`SearchProvider`, `ProviderContext`) |
 | `search/registry.ts` | Value-only provider list, `resolveProvider`, labels |
+| `search/facts.ts` | What this session resolved to, published for `doctor` (`providerFacts`) |
 | `search/config.ts` | Search config type, defaults, clamping, API-key resolution |
 | `search/schema.ts` | Params/output schema, `resolveRequest()` |
 | `search/search.ts` | Provider resolution, throttle, and outcome mapping |
@@ -368,10 +410,10 @@ the content type and size.
 | `fetch/tool.ts` | `web_fetch` definition, activation, rendering |
 | `fetch/config.ts` | Fetch config type, defaults, clamping |
 | `fetch/schema.ts` | Params/output schema, `resolveRequest()` |
-| `fetch/page.ts` | Fetch + redirect-following + content-type routing |
+| `fetch/page.ts` | Fetch + redirect-following + content-type routing; PDF extraction is injected via `FetchDeps` from the shared `lib/pdf.ts` |
+| `fetch/structured.ts` | JSON/CSV/XML body → readable text, or `undefined` to keep the raw body |
 | `fetch/ssrf.ts` | URL/host validation and private-address blocking |
 | `fetch/extract.ts` | HTML → title + readable text |
-| `fetch/page.ts` | Fetch + redirect-following + content-type routing; PDF extraction is injected via `FetchDeps` from the shared `lib/pdf.ts` |
 | `fetch/find.ts` | Passage search (exact/insensitive/fuzzy) |
 | `fetch/cache.ts` | Per-registration LRU/TTL page cache (`createPageCache`) |
 | `fetch/format.ts`, `fetch/types.ts` | Result text and data types |
@@ -386,7 +428,8 @@ charset decoding (declared, `<meta>`, and BOM precedence), a streamed body over
 the byte cap, redirect re-validation (per-hop SSRF with literal IPs),
 method/header/body validation, the nested config merge and section isolation,
 extraction, formatting/paging through the tool (including a regression test that
-a truncated page keeps its `startIndex` hint), find and its cap/budget notices,
+a truncated page keeps its `startIndex` hint), structured rendering per format and
+its malformed fallbacks, find and its cap/budget notices,
 the per-registration cache, the SSRF guard, the injected PDF dependency, and both
 tools' registration/execution. Tests inject the runner, throttle, and PDF extractor
 through the tools' dependencies, so no network or PDF library is needed; the