@plurnk/plurnk-schemes-http 1.4.0 → 1.6.0

package/.env.defaults

@@ -9,35 +9,15 @@
 # must not set them. Standalone (no daemon), the required numerics fail hard
 # when unset, naming the var.
 #
-# ── Required (render path) ───────────────────────────────────────────────────
-# Navigation/render timeout in ms (page.goto networkidle budget).
+# ── Required (acquisition path) ─────────────────────────────────────────────
+# HTTP request timeout in ms.
 PLURNK_SCHEMES_HTTP_FETCH_TIMEOUT=30000
-# Salvage threshold in body-text chars: a networkidle timeout whose DOM already
-# holds at least this much innerText is the chatty-page case (long-poll, ad
-# refresh) — the render is kept; below it the timeout propagates.
-PLURNK_SCHEMES_HTTP_SALVAGE_MIN_BODY_CHARS=200
-# Warm-chromium idle teardown in ms: the browser process closes after this long
-# with no render activity, and relaunches on the next render.
-PLURNK_SCHEMES_HTTP_IDLE_TIMEOUT=900000
 
 # ── Defaults (floor-set; override to change) ────────────────────────────────
-# Playwright BrowserType method. "launch" uses an operator-provisioned compatible
-# Chromium unless CHANNEL or EXECUTABLE_PATH selects another local browser. "connect"
-# uses a Playwright protocol endpoint; "connectOverCDP" attaches to a running
-# Chromium browser; "disabled" explicitly removes rendered HTML retrieval.
-PLURNK_SCHEMES_HTTP_PLAYWRIGHT_METHOD=launch
-# Browser launch/connection timeout in ms.
-PLURNK_SCHEMES_HTTP_PLAYWRIGHT_TIMEOUT=30000
-# Headless launch. Set 0 for a visible operator-managed browser session.
-PLURNK_SCHEMES_HTTP_PLAYWRIGHT_HEADLESS=1
-# Playwright's Chromium sandbox launch option. Playwright defaults this off;
-# set 1 only where the host/container supports Chromium user namespaces.
-PLURNK_SCHEMES_HTTP_PLAYWRIGHT_CHROMIUM_SANDBOX=0
-# Mobile device emulation on the render path (Pixel-5-class viewport + UA):
-# responsive sites serve a lighter, less-chrome layout — a better generation
-# hint. "0" renders desktop (escape hatch for sites that serve degraded or
-# blocked mobile content). Required read: unset crashes at render, naming the var.
-PLURNK_SCHEMES_HTTP_MOBILE=1
+# Tavily Extract extraction depth ("basic" or "advanced").
+PLURNK_SCHEMES_HTTP_TAVILY_DEPTH=basic
+# Tavily Extract request timeout in ms.
+PLURNK_SCHEMES_HTTP_TAVILY_TIMEOUT_MS=30000
 # Per-URL freshness TTL in ms: a re-read of a URL fetched within this window
 # serves the stored copy with ZERO round-trips (no conditional GET). This makes
 # a just-materialized web entry immediately readable like any other entry.
@@ -50,22 +30,12 @@ PLURNK_SCHEMES_HTTP_REDIRECTS=5
 # Maximum buffered characters in one incomplete SSE line/event. Exceeding this
 # bound terminates the remote stream instead of allowing unbounded memory growth.
 PLURNK_SCHEMES_HTTP_SSE_MAX_BUFFER_CHARS=1048576
-# Maximum characters retained from a caught fetch, render, or transport
+# Maximum characters retained from a caught fetch, extract, or transport
 # diagnostic in a model-facing Problem detail. Complete errors remain in daemon
 # diagnostics.
 PLURNK_SCHEMES_HTTP_ERROR_DETAIL_LIMIT=512
 
 # ── Optional modes (absence IS the mode — never floor-set) ──────────────────
-# Required by connect/connectOverCDP: the Playwright or CDP endpoint.
-# PLURNK_SCHEMES_HTTP_PLAYWRIGHT_ENDPOINT=http://localhost:9222
-# Optional launch channel: chromium, chrome, chrome-beta, chrome-dev,
-# chrome-canary, msedge, msedge-beta, msedge-dev, or msedge-canary.
-# PLURNK_SCHEMES_HTTP_PLAYWRIGHT_CHANNEL=chrome
-# Optional exact browser executable for launch. Mutually exclusive with CHANNEL.
-# PLURNK_SCHEMES_HTTP_PLAYWRIGHT_EXECUTABLE_PATH=/usr/bin/chromium
-# Chromium JS heap cap in MB (--max-old-space-size). Unset → chromium default.
-# PLURNK_SCHEMES_HTTP_CHROMIUM_HEAP_MB=2048
-
-# Native Playwright browser-location controls remain authoritative. This example
-# is intentionally unset; the package never supplies its own path.
-# PLAYWRIGHT_BROWSERS_PATH=/var/cache/ms-playwright
+# Optional Tavily API key for HTML-to-Markdown extraction. When omitted/unset,
+# the installed HTML mimetype projection is used as the local floor.
+# TAVILY_API_KEY=

package/README.md

@@ -8,28 +8,28 @@ authored against the DB-free
 
 ## HTTP operations
 
-| Operation                         | Behavior                                                                                  |
-| --------------------------------- | ----------------------------------------------------------------------------------------- |
-| Unscoped `READ(http(s)://…)`      | Fetch or reuse/revalidate a GET representation, then stream the selected response channel |
-| Scoped `READ(http(s)://…)<scope>` | Apply the standard entry READ to an already-materialized response without refetching      |
-| `FIND(http(s)://…):matcher`       | Materialize an exact URL when required, then use the universal entry query and matcher    |
-| `SEND[200](http(s)://…):body:`    | POST the body and stream the response                                                     |
-| `EDIT(http(s)://…):body:`         | PUT a whole-resource replacement; line-scoped HTTP edits are invalid                      |
-| `KILL(http(s)://…)`               | DELETE the remote resource and stream the response                                        |
-| `SEND[499](http(s)://…)`          | Cancel the routed in-flight subscription                                                  |
-| `SEND[410](http(s)://…)`          | Delete the local stored response entry                                                    |
+| Operation                                      | Behavior                                                                                  |
+| ---------------------------------------------- | ----------------------------------------------------------------------------------------- |
+| `## READ0 (http(s)://…)`                       | Fetch or reuse/revalidate a GET representation, then stream the selected response channel |
+| `## READ0 (http(s)://…) <scope>`               | Apply the standard entry READ to an already-materialized response without refetching      |
+| `## FIND0 (http(s)://…)` with matcher body     | Materialize an exact URL when required, then use the universal entry query and matcher    |
+| `## SEND0 [200] (http(s)://…)` with body       | POST the body and stream the response                                                     |
+| `## EDIT0 (http(s)://…)` with body             | PUT a whole-resource replacement; line-scoped HTTP edits are invalid                      |
+| `## KILL0 (http(s)://…)`                       | DELETE the remote resource and stream the response                                        |
+| `## SEND0 [499] (http(s)://…)`                 | Cancel the routed in-flight subscription                                                  |
+| `## SEND0 [410] (http(s)://…)`                 | Delete the local stored response entry                                                    |
 
 A path-pattern FIND surveys already-materialized web entries; it does not crawl
-or discover the remote web. Exact FIND and matcher READ share the standard
-catalog, matcher evidence, weighting, pagination, and status contract.
+or discover the remote web. Exact matcher FIND shares the standard flat
+location, weighting, pagination, and status contract.
 
 ## HTTP channels
 
-| Channel  | Content                                                                 |
-| -------- | ----------------------------------------------------------------------- |
-| `body`   | Text, derived Unicode, typed binary marker, SSE data, or readable HTML  |
-| `header` | HTTP status line, headers, and package acquisition metadata             |
-| `html`   | Faithful HTML used to produce the readable body                         |
+| Channel  | Content                                                                    |
+| -------- | -------------------------------------------------------------------------- |
+| `body`   | Text, derived Unicode, typed binary marker, SSE data, or HTML-page Markdown |
+| `header` | Origin, acquisition, materializer, projection, provider, and usage evidence |
+| `html`   | Original server-source HTML when the origin supplies it                    |
 
 A fragmentless operation publishes only `body`; auxiliary channels remain
 durable and can be addressed explicitly. Remote HTTP status is stored in
@@ -42,17 +42,25 @@ projection ceiling returns `413`.
 
 ## Design
 
-- WebFetcher checks automatic byte targets and redirects; direct HTTP,
-  Playwright, and WebSocket use their ordinary configured transports.
-- Direct GET HTML is rendered through Playwright, projected into `body`, and
-  retained faithfully in `html`.
+- WebFetcher checks automatic byte targets and redirects; direct HTTP and
+  WebSocket retain their explicit-target authority.
+- Generic GETs negotiate origin Markdown first. When the origin returns HTML,
+  configured Tavily produces `body`; otherwise the installed HTML reader is the
+  local route. Recoverable Tavily failures use that same reader as a `203`
+  recovery floor. Hard provider failures do not silently change producers.
+- `body`, `header`, and `html` settle independently. The selected channel
+  determines operation success, so raw server source and evidence remain usable
+  when body production fails, and a provider body can survive origin transport
+  failure without fabricating HTML.
 - GET representations carry method, acquisition-time, and single-variant cache
-  metadata; derived representations also carry projection identity. Explicit
-  request metadata, `Vary`, `no-store`, expired origin freshness, or a reader
-  change prevents unsafe reuse; the operator TTL remains an independent ceiling.
+  metadata; HTML adds materializer-route evidence, and local derivation adds
+  projection identity. Request metadata, `Vary`, `no-store`, expired freshness,
+  or a materializer/reader change prevents reuse; operator TTL is a separate
+  ceiling. Stale page composites are fully reacquired rather than restored by
+  an origin `304`.
 - HTTP(S) and WS(S) entry identity retains protocol, authority, path, ordered
   query, and explicit empty query; a fragment selects a channel.
-- Handler-owned browser and socket state follows the shared readiness, drain,
+- Handler-owned socket state follows the shared readiness, drain,
   and aggregate-shutdown lifecycle.
 - All storage and streaming work uses `SchemeCtx` capabilities rather than a
   raw database handle.
@@ -70,19 +78,15 @@ npm i @plurnk/plurnk-schemes-http
 plurnk start
 ```
 
-The Playwright client is included. Provision a compatible local browser
-separately, select an external Playwright/CDP endpoint, or disable rendering.
 The shipped `.env.defaults` is the canonical operator configuration registry.
-
-For local `launch`:
-
-```sh
-npx playwright install chromium
-```
+To enable Tavily Extract for generic public HTML materialization, set
+`TAVILY_API_KEY` in `.env`. Depth and client timeout are controlled by
+`PLURNK_SCHEMES_HTTP_TAVILY_DEPTH` and
+`PLURNK_SCHEMES_HTTP_TAVILY_TIMEOUT_MS`. Without Tavily,
+`@plurnk/plurnk-mimetypes-text-html` supplies the local body.
 
 ## Verify
 
 ```sh
 npm test
-npm run test:intg
 ```