@plurnk/plurnk-schemes-http 1.3.12 → 1.5.0

package/.env.defaults

@@ -1,69 +1,41 @@
 # @plurnk/plurnk-schemes-http — .env.defaults (operator config + defaults floor).
 #
 # Every knob this package reads, family-namespaced PLURNK_SCHEMES_HTTP_*. This
-# file IS the operator documentation, and the daemon assembles every installed
-# member's .env.defaults into one operative floor at boot — set-if-unset under
-# the operator's env (service SPEC §operator-config-env-defaults, schemes#31).
+# file is the canonical value registry; {§http-config} owns policy. The daemon
+# assembles every installed member's .env.defaults into one operative floor at
+# boot, set-if-unset under the operator's env {§operator-config-env-defaults}.
 # UNCOMMENTED keys are real defaults the floor sets. COMMENTED keys are modes
 # whose ABSENCE is meaningful (unset = the documented behavior) — the floor
 # must not set them. Standalone (no daemon), the required numerics fail hard
 # when unset, naming the var.
 #
-# ── Required (render path) ───────────────────────────────────────────────────
-# Navigation/render timeout in ms (page.goto networkidle budget).
+# ── Required (acquisition path) ─────────────────────────────────────────────
+# HTTP request timeout in ms.
 PLURNK_SCHEMES_HTTP_FETCH_TIMEOUT=30000
-# Salvage threshold in body-text chars: a networkidle timeout whose DOM already
-# holds at least this much innerText is the chatty-page case (long-poll, ad
-# refresh) — the render is kept; below it the timeout propagates.
-PLURNK_SCHEMES_HTTP_SALVAGE_MIN_BODY_CHARS=200
-# Warm-chromium idle teardown in ms: the browser process closes after this long
-# with no render activity, and relaunches on the next render.
-PLURNK_SCHEMES_HTTP_IDLE_TIMEOUT=900000
 
 # ── Defaults (floor-set; override to change) ────────────────────────────────
-# Playwright BrowserType method. "launch" uses the compatible bundled Chromium
-# unless CHANNEL or EXECUTABLE_PATH selects another local browser. "connect"
-# uses a Playwright protocol endpoint; "connectOverCDP" attaches to a running
-# Chromium browser; "disabled" explicitly removes rendered HTML retrieval.
-PLURNK_SCHEMES_HTTP_PLAYWRIGHT_METHOD=launch
-# Browser launch/connection timeout in ms.
-PLURNK_SCHEMES_HTTP_PLAYWRIGHT_TIMEOUT=30000
-# Headless launch. Set 0 for a visible operator-managed browser session.
-PLURNK_SCHEMES_HTTP_PLAYWRIGHT_HEADLESS=1
-# Playwright's Chromium sandbox launch option. Playwright defaults this off;
-# set 1 only where the host/container supports Chromium user namespaces.
-PLURNK_SCHEMES_HTTP_PLAYWRIGHT_CHROMIUM_SANDBOX=0
-# Mobile device emulation on the render path (Pixel-5-class viewport + UA):
-# responsive sites serve a lighter, less-chrome layout — a better generation
-# hint. "0" renders desktop (escape hatch for sites that serve degraded or
-# blocked mobile content). Required read: unset crashes at render, naming the var.
-PLURNK_SCHEMES_HTTP_MOBILE=1
+# Tavily Extract extraction depth ("basic" or "advanced").
+PLURNK_SCHEMES_HTTP_TAVILY_DEPTH=basic
+# Tavily Extract request timeout in ms.
+PLURNK_SCHEMES_HTTP_TAVILY_TIMEOUT_MS=30000
 # Per-URL freshness TTL in ms: a re-read of a URL fetched within this window
 # serves the stored copy with ZERO round-trips (no conditional GET). This makes
 # a just-materialized web entry immediately readable like any other entry.
 # 0 opts into revalidation on every read via ETag/304.
 # Required read: unset crashes, naming the var.
 PLURNK_SCHEMES_HTTP_TTL_MS=300000
-# Max HTTP redirect hops the SSRF-guarded fetch follows (WebFetcher). Each hop is
-# re-guarded — a public URL redirecting into private space is refused. 0 follows
-# none (strictest). Required read: unset crashes, naming the var.
+# Max HTTP redirect hops automatic WebFetcher acquisition follows. Every hop is
+# checked; 0 follows none. Required read: unset crashes, naming the var.
 PLURNK_SCHEMES_HTTP_REDIRECTS=5
 # Maximum buffered characters in one incomplete SSE line/event. Exceeding this
 # bound terminates the remote stream instead of allowing unbounded memory growth.
 PLURNK_SCHEMES_HTTP_SSE_MAX_BUFFER_CHARS=1048576
+# Maximum characters retained from a caught fetch, extract, or transport
+# diagnostic in a model-facing Problem detail. Complete errors remain in daemon
+# diagnostics.
+PLURNK_SCHEMES_HTTP_ERROR_DETAIL_LIMIT=512
 
 # ── Optional modes (absence IS the mode — never floor-set) ──────────────────
-# Required by connect/connectOverCDP: the Playwright or CDP endpoint.
-# PLURNK_SCHEMES_HTTP_PLAYWRIGHT_ENDPOINT=http://localhost:9222
-# Optional launch channel: chromium, chrome, chrome-beta, chrome-dev,
-# chrome-canary, msedge, msedge-beta, msedge-dev, or msedge-canary.
-# PLURNK_SCHEMES_HTTP_PLAYWRIGHT_CHANNEL=chrome
-# Optional exact browser executable for launch. Mutually exclusive with CHANNEL.
-# PLURNK_SCHEMES_HTTP_PLAYWRIGHT_EXECUTABLE_PATH=/usr/bin/chromium
-# Chromium JS heap cap in MB (--max-old-space-size). Unset → chromium default.
-# PLURNK_SCHEMES_HTTP_CHROMIUM_HEAP_MB=2048
-
-# Native Playwright installation controls remain authoritative and pass through
-# unchanged. These examples are intentionally unset.
-# PLAYWRIGHT_BROWSERS_PATH=/var/cache/ms-playwright
-# PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD=1
+# Optional Tavily API key for HTML-to-Markdown extraction. When omitted/unset,
+# the installed HTML mimetype projection is used as the local floor.
+# TAVILY_API_KEY=

package/README.md

@@ -1,41 +1,92 @@
 # plurnk-schemes-http
 
-`http(s)://` URI scheme handler for the [plurnk](https://github.com/plurnk/plurnk-service) agent runtime. The **first greenfield `@plurnk/plurnk-schemes-*` sibling** — authored entirely against the DB-free capability contract ([`@plurnk/plurnk-schemes`](https://github.com/plurnk/plurnk-service/tree/main/plurnk-schemes) `SchemeCtx`), importing zero plurnk-service internals.
+HTTP(S) request/response and WebSocket scheme handlers for the
+[plurnk](https://github.com/plurnk/plurnk-service) agent runtime. The package is
+authored against the DB-free
+[`@plurnk/plurnk-schemes`](https://github.com/plurnk/plurnk-service/tree/main/plurnk-schemes)
+`SchemeCtx` contract.
 
-## What it does
+## HTTP operations
 
-Lets the model treat any web URL as an addressable, streamable resource:
+| Operation                         | Behavior                                                                                  |
+| --------------------------------- | ----------------------------------------------------------------------------------------- |
+| Unscoped `READ(http(s)://…)`      | Fetch or reuse/revalidate a GET representation, then stream the selected response channel |
+| Scoped `READ(http(s)://…)<scope>` | Apply the standard entry READ to an already-materialized response without refetching      |
+| `FIND(http(s)://…):matcher`       | Materialize an exact URL when required, then use the universal entry query and matcher    |
+| `SEND[200](http(s)://…):body:`    | POST the body and stream the response                                                     |
+| `EDIT(http(s)://…):body:`         | PUT a whole-resource replacement; line-scoped HTTP edits are invalid                      |
+| `KILL(http(s)://…)`               | DELETE the remote resource and stream the response                                        |
+| `SEND[499](http(s)://…)`          | Cancel the routed in-flight subscription                                                  |
+| `SEND[410](http(s)://…)`          | Delete the local stored response entry                                                    |
 
-| Op | Behavior |
-|---|---|
-| `READ(http(s)://host/path)` | Without `<scope>`, fetch/revalidate the URL and stream its readable body. With `<scope>`, read that range from the materialized response without refetching. |
-| `FIND(http(s)://host/path):matcher` | Materialize an exact URL when necessary, then query it through the standard entry FIND contract. Glob/regex scopes query already-materialized web entries. |
-| `SEND[200](http(s)://…)` | Request with a body (POST); response streams back the same way. |
-| `SEND[499](http(s)://…)` | Cancel an in-flight request (abort the fetch). |
-| `SEND[410](http(s)://…)` | Delete the cached response entry. |
+A path-pattern FIND surveys already-materialized web entries; it does not crawl
+or discover the remote web. Exact matcher FIND shares the standard flat
+location, weighting, pagination, and status contract.
 
-Response status + headers land in the `header` channel; the body in `body` (the default).
+## HTTP channels
 
-## Channels
+| Channel  | Content                                                                    |
+| -------- | -------------------------------------------------------------------------- |
+| `body`   | Text, derived Unicode, typed binary marker, SSE data, or HTML-page Markdown |
+| `header` | Origin, acquisition, materializer, projection, provider, and usage evidence |
+| `html`   | Original server-source HTML when the origin supplies it                    |
 
-- `body` — response payload (default channel).
-- `header` — `HTTP <status> <statusText>` line + response headers.
+A fragmentless operation publishes only `body`; auxiliary channels remain
+durable and can be addressed explicitly. Remote HTTP status is stored in
+`header`; the PLURNK operation result reports the streaming lifecycle.
+Binary responses use an installed bounded readable projection when available;
+only derived Unicode becomes durable. Otherwise direct operations preserve a
+typed empty marker and return `415`, while exact query preparation prunes the
+unreadable result from automatic search ingestion. Input above the common
+projection ceiling returns `413`.
 
 ## Design
 
-- **Streaming via the capability `subscriptions` lifecycle** (`open` → `notifyChunk` → `close`). `open()` returns the worker+teardown-composed `AbortSignal`; a `SubscriptionHandle` is registered so the engine routes `SEND[499]` cancellation to the in-flight `fetch`.
-- **Batteries-included rendering** — installation provisions Playwright's compatible Chromium and daemon boot verifies it. The shipped `.env.defaults` also exposes Playwright `launch`, `connect`, and `connectOverCDP`: launch may select an installed browser channel or executable, while connection methods target an operator-managed browser. Rendering may be explicitly disabled; incompatible choices fail instead of falling back silently.
-- **DB-free** — reaches the substrate only through `ctx` capabilities (`subscriptions`, `entries`), never a raw DB handle (plurnk-schemes SPEC §5). This is what the keystone capability ctx made possible.
+- WebFetcher checks automatic byte targets and redirects; direct HTTP and
+  WebSocket retain their explicit-target authority.
+- Generic GETs negotiate origin Markdown first. When the origin returns HTML,
+  configured Tavily produces `body`; otherwise the installed HTML reader is the
+  local route. Recoverable Tavily failures use that same reader as a `203`
+  recovery floor. Hard provider failures do not silently change producers.
+- `body`, `header`, and `html` settle independently. The selected channel
+  determines operation success, so raw server source and evidence remain usable
+  when body production fails, and a provider body can survive origin transport
+  failure without fabricating HTML.
+- GET representations carry method, acquisition-time, and single-variant cache
+  metadata; HTML adds materializer-route evidence, and local derivation adds
+  projection identity. Request metadata, `Vary`, `no-store`, expired freshness,
+  or a materializer/reader change prevents reuse; operator TTL is a separate
+  ceiling. Stale page composites are fully reacquired rather than restored by
+  an origin `304`.
+- HTTP(S) and WS(S) entry identity retains protocol, authority, path, ordered
+  query, and explicit empty query; a fragment selects a channel.
+- Handler-owned socket state follows the shared readiness, drain,
+  and aggregate-shutdown lifecycle.
+- All storage and streaming work uses `SchemeCtx` capabilities rather than a
+  raw database handle.
+
+The same package registers `wss` (with `ws` routing to it) for
+workspace-scoped full-duplex connections. See [`docs/wss.md`](docs/wss.md) for
+its operation surface and current transport limits.
 
 ## Install
 
-```
-npm i @plurnk/plurnk-schemes-http && plurnk start
+Requires Node.js 26 or newer.
+
+```sh
+npm i @plurnk/plurnk-schemes-http
+plurnk start
 ```
 
-Plugin discovery registers it at boot (`package.json#plurnk.kind === "scheme"`).
-No browser setup is required for the default installation.
+The shipped `.env.defaults` is the canonical operator configuration registry.
+To enable Tavily Extract for generic public HTML materialization, set
+`TAVILY_API_KEY` in `.env`. Depth and client timeout are controlled by
+`PLURNK_SCHEMES_HTTP_TAVILY_DEPTH` and
+`PLURNK_SCHEMES_HTTP_TAVILY_TIMEOUT_MS`. Without Tavily,
+`@plurnk/plurnk-mimetypes-text-html` supplies the local body.
 
-## Tests
+## Verify
 
-`test:lint` (tsc) + `test:unit` (conformant `SchemeCtx` stub + mock `fetch`).
+```sh
+npm test
+```