@plurnk/plurnk-schemes-http 1.6.1 → 1.8.0

package/.env.defaults

@@ -14,10 +14,6 @@
 PLURNK_SCHEMES_HTTP_FETCH_TIMEOUT=30000
 
 # ── Defaults (floor-set; override to change) ────────────────────────────────
-# Tavily Extract extraction depth ("basic" or "advanced").
-PLURNK_SCHEMES_HTTP_TAVILY_DEPTH=basic
-# Tavily Extract request timeout in ms.
-PLURNK_SCHEMES_HTTP_TAVILY_TIMEOUT_MS=30000
 # Per-URL freshness TTL in ms: a re-read of a URL fetched within this window
 # serves the stored copy with ZERO round-trips (no conditional GET). This makes
 # a just-materialized web entry immediately readable like any other entry.
@@ -36,6 +32,9 @@ PLURNK_SCHEMES_HTTP_SSE_MAX_BUFFER_CHARS=1048576
 PLURNK_SCHEMES_HTTP_ERROR_DETAIL_LIMIT=512
 
 # ── Optional modes (absence IS the mode — never floor-set) ──────────────────
-# Optional Tavily API key for HTML-to-Markdown extraction. When omitted/unset,
-# the installed HTML mimetype projection is used as the local floor.
-# TAVILY_API_KEY=
+# The documented third-party materializer showcase: install the
+# @plurnk/plurnk-schemes-http-tavily plugin, then select it and provide its
+# key. Absence (unset) of either keeps the installed HTML projection as the
+# local floor; the plugin documents its own depth/timeout knobs.
+# PLURNK_SCHEMES_HTTP_MATERIALIZER=tavily-extract
+# TAVILY_API_KEY=...

package/README.md

@@ -45,9 +45,10 @@ projection ceiling returns `413`.
 - WebFetcher checks automatic byte targets and redirects; direct HTTP and
   WebSocket retain their explicit-target authority.
 - Generic GETs negotiate origin Markdown first. When the origin returns HTML,
-  configured Tavily produces `body`; otherwise the installed HTML reader is the
-  local route. Recoverable Tavily failures use that same reader as a `203`
-  recovery floor. Hard provider failures do not silently change producers.
+  a selected materializer plugin ({§http-materializer-plugins}) produces
+  `body`; otherwise the installed HTML reader is the local route. Recoverable
+  materializer failures use that same reader as a `203` recovery floor. Hard
+  provider failures do not silently change producers.
 - `body`, `header`, and `html` settle independently. The selected channel
   determines operation success, so raw server source and evidence remain usable
   when body production fails, and a provider body can survive origin transport
@@ -79,11 +80,11 @@ plurnk start
 ```
 
 The shipped `.env.defaults` is the canonical operator configuration registry.
-To enable Tavily Extract for generic public HTML materialization, set
-`TAVILY_API_KEY` in `.env`. Depth and client timeout are controlled by
-`PLURNK_SCHEMES_HTTP_TAVILY_DEPTH` and
-`PLURNK_SCHEMES_HTTP_TAVILY_TIMEOUT_MS`. Without Tavily,
-`@plurnk/plurnk-mimetypes-text-html` supplies the local body.
+To select a page-materializer plugin for generic public HTML materialization,
+install it and set `PLURNK_SCHEMES_HTTP_MATERIALIZER=<id>` in `.env` (the
+`@plurnk/plurnk-schemes-http-tavily` showcase plugin supplies `tavily-extract`).
+Without a selection, `@plurnk/plurnk-mimetypes-text-html` supplies the local
+body.
 
 ## Verify
 

package/SPEC.md

@@ -1,8 +1,9 @@
 # plurnk-schemes-http — Specification
 
 This package owns the `http(s)://` request/response scheme, the `ws(s)://`
-full-duplex scheme, plus automatic entry-acquisition, Tavily Extract, and local
-HTML projection foundations. Both handlers implement the DB-free `SchemeCtx` author contract.
+full-duplex scheme, plus automatic entry-acquisition, pluggable page
+materializers, and the local HTML projection foundation. Both handlers implement
+the DB-free `SchemeCtx` author contract.
 
 ## §http-manifest §1 HTTP manifest
 
@@ -66,13 +67,13 @@ flowchart TD
     origin -->|text/markdown| markdown["Use origin Markdown as body<br/>and request an HTML variant"]
     origin -->|server HTML| source["Retain exact server HTML"]
     origin -->|transport unavailable<br/>after public admission| noSource["No HTML source"]
-    source --> eligible{"Generic public request<br/>with Tavily configured?"}
+    source --> eligible{"Generic public request<br/>with a materializer selected?"}
     noSource --> eligible
     eligible -->|no| local["Installed HTML projection"]
-    eligible -->|yes| tavily["Tavily Extract Markdown"]
-    tavily -->|success| providerBody["Tavily body"]
-    tavily -->|recoverable failure<br/>and source exists| recovery["Local projection; body 203"]
-    tavily -->|hard failure or<br/>no recovery source| bodyError["Body errored"]
+    eligible -->|yes| mat["Materializer Markdown"]
+    mat -->|success| providerBody["Materializer body"]
+    mat -->|recoverable failure<br/>and source exists| recovery["Local projection; body 203"]
+    mat -->|hard failure or<br/>no recovery source| bodyError["Body errored"]
     local -->|present, including empty| localBody["Local body"]
     local -->|absent or no source| bodyError
     markdown --> settle["Write complete canonical channels<br/>and durable producer results"]
@@ -93,7 +94,7 @@ but cold and warm representations pass through the same core projection.
 | Response                                | `body`                                                   | Auxiliary materialization                                 | Completion                                  |
 | --------------------------------------- | -------------------------------------------------------- | --------------------------------------------------------- | ------------------------------------------- |
 | Negotiated origin Markdown              | Exact origin Markdown                                    | Independently acquired server source in `html` when found | Static representation, then core projection |
-| HTML-page production                    | Tavily Markdown, local Markdown, or independent error    | Exact server source in `html` when origin supplied it     | Static representation, then core projection |
+| HTML-page production                    | Materializer Markdown, local Markdown, or independent error | Exact server source in `html` when origin supplied it     | Static representation, then core projection |
 | `text/event-stream`                     | One `data` value plus newline per `text/plain` chunk     | Initial response in `header`                              | `102`; origin close settles subscription    |
 | No response body                        | Present empty text                                       | Response and package metadata in `header`                 | Static representation; body READ is `204`   |
 | Origin HTTP `4xx`/`5xx`                 | Preserve available origin or independently produced text | Exact response evidence; origin-backed channels errored   | Selected channel's durable outcome          |
@@ -138,52 +139,51 @@ mutation responses retain incremental streaming.
 | Configured binary with reader               | Apply the bounded byte projection           | Derived Unicode plus source type and identity                       |
 | Configured binary without reader            | Cancel without retaining bytes              | `null`; the consumer applies its absence policy                    |
 | Binary input exceeds the common bound       | Cancel and preserve the typed cause         | `WebMaterializationError` caused by `ProjectionInputLimitError`     |
-| Negotiated origin `text/markdown`           | Accept it without Tavily                    | Exact Markdown in `body`; independently requested source in `html`  |
-| Eligible HTML with Tavily configured        | Use Tavily as the body producer             | Tavily Markdown in `body`; origin server source in `html`           |
-| Ineligible HTML or Tavily not configured    | Use the installed HTML projection           | Local Markdown in `body`; origin server source in `html`            |
-| Recoverable Tavily failure with source HTML | Use the installed projection as recovery    | Local Markdown in `body` with terminal `203`; source in `html`       |
-| Hard Tavily failure                         | Preserve evidence; do not run local recovery | `body` errored; independently successful `html`/`header` survive    |
-| Tavily success after origin transport loss  | Preserve provider body and origin failure   | `body` static; `html` errored                                       |
+| Negotiated origin `text/markdown`           | Accept it without the materializer          | Exact Markdown in `body`; independently requested source in `html`  |
+| Eligible HTML with a materializer selected  | Use the materializer as the body producer   | Materializer Markdown in `body`; origin server source in `html`     |
+| Ineligible HTML or no materializer selected | Use the installed HTML projection           | Local Markdown in `body`; origin server source in `html`            |
+| Recoverable materializer failure with HTML  | Use the installed projection as recovery    | Local Markdown in `body` with terminal `203`; source in `html`       |
+| Hard materializer failure                   | Preserve evidence; do not run local recovery | `body` errored; independently successful `html`/`header` survive    |
+| Materializer success after origin loss      | Preserve provider body and origin failure   | `body` static; `html` errored                                       |
 | Projection implementation throws            | Stop and preserve the cause                 | `WebMaterializationError` with stage `projection`                   |
 
 A projection object is present even when its content is `""`; only `null`
 denotes absence. A materialization exception retains its original `cause`; it
 never enters the absence channel.
 
-#### §tavily-extract Tavily Extract boundary
-
-Tavily is eligible only for a credential-free generic request whose target has
-been admitted as public. Authored request metadata—including an authored
-`Accept` field—makes the request ineligible. The package-generated Markdown
-negotiation, browser-compatible `User-Agent`, and conditional cache fields are
-transport mechanics, not authored metadata. No target headers or origin
-credentials cross the provider boundary.
+#### §http-materializer-plugins Materializer plugins
+
+`PLURNK_SCHEMES_HTTP_MATERIALIZER` selects one discovered `http-materializer`
+plugin by id; unset means the installed HTML projection is the only body
+producer. A materializer package declares
+`plurnk: { kind: "http-materializer", materializers: [{ id, module }] }` and
+exports one `HttpMaterializer` per entry under the executor family's
+discovery, trust, and one-flat-id-namespace rules ({§plugin-discovery}). The
+selected materializer is consulted only for a credential-free generic request
+whose target has been admitted as public. Authored request metadata—including
+an authored `Accept` field—makes the request ineligible. The package-generated
+Markdown negotiation, browser-compatible `User-Agent`, and conditional cache
+fields are transport mechanics, not authored metadata. No target headers or
+origin credentials cross the materializer boundary.
 
 When no `Accept` is authored, origin acquisition offers
 `text/markdown, text/html;q=0.9, */*;q=0.1`. An origin `text/markdown`
-representation wins and Tavily is skipped; a second origin request with
-`Accept: text/html` may populate `html`. An authored `Accept` value is sent
-unchanged and makes any returned HTML use the local projection route.
-
-One provider request is `POST https://api.tavily.com/extract` with bearer
-authentication and exactly one URL, configured `basic` or `advanced` depth,
-`format: "markdown"`, and `include_usage: true`. A package-owned abort timeout
-bounds the call. Success requires Markdown plus `request_id` and
-`usage.credits`; those facts remain durable in `header`.
-
-| Provider outcome                         | Classification | Body behavior when server HTML exists                  |
-| ---------------------------------------- | -------------- | ------------------------------------------------------ |
-| Success with required evidence           | Success        | Use Tavily Markdown                                    |
-| Caller cancellation                      | Cancelled      | Return exact `499`; no recovery                        |
-| Client timeout or transport failure      | Recoverable    | Local projection, terminal `203`                       |
-| `429`, `5xx`, or `failed_results`        | Recoverable    | Local projection, terminal `203`                       |
-| `401` or `403`                           | Hard           | `tavily-authentication-failed`; no local projection    |
-| Other `4xx`                              | Hard           | `tavily-provider-rejected`; no local projection        |
-| Malformed success or missing evidence    | Hard           | `tavily-invalid-response`; no local projection         |
-
-Without server HTML, no local recovery input exists. A Tavily success can still
-close `body`, but every Tavily failure remains the body failure and `html`
-closes errored.
+representation wins and the materializer is skipped; a second origin request
+with `Accept: text/html` may populate `html`. An authored `Accept` value is
+sent unchanged and makes any returned HTML use the local projection route.
+
+The materializer's `eligible(url)` returns its identity string when it will
+serve the URL (its credentials and supported targets are its own); `extract`
+returns `success` (its sanitized Markdown body plus its own evidence headers),
+`recoverable` (local projection fallback with the materializer's evidence and
+terminal `203`), or `hard` (an exact materializer-owned Problem; no local
+recovery). A materializer `extract` throw is the materialization failure, never
+a silent fallback. A selected id with no installed provider fails hard rather
+than degrading silently.
+
+Without server HTML, no local recovery input exists. A materializer success can
+still close `body`, but every materializer failure remains the body failure and
+`html` closes errored.
 
 #### §http-channel-outcomes Channel outcomes
 
@@ -201,9 +201,19 @@ representation. A missing source variant is an explicit empty `errored`
 channel, never an absent fact that later cache use can reinterpret as successful
 empty content. A direct non-success HTTP response likewise remains available as
 evidence, while each origin-backed channel carries an exact
-`http-response-status` producer Problem. A Tavily-produced body and the
+`http-response-status` producer Problem. A materializer-produced body and the
 acquisition `header` remain independent of an unavailable origin source.
 
+§http-llms-txt **Origin llms.txt companions.** After a successful generic GET
+materialization, the scheme opportunistically acquires `<origin>/llms.txt`
+once per origin per TTL window and materializes it as its own origin entry
+(`https://host/llms.txt`), surfaceable by FIND. A missing companion, any
+non-2xx, binary content, or a transport failure is quiet: the companion is
+never fabricated, the failed probe is remembered (not retried per READ), and
+the piggyback never fails the READ that carried it. The companion itself
+never recurses. This is lazy by construction — one extra request only when
+an origin is first being read.
+
 ## §http-status §4 HTTP status mapping
 
 | Outcome                                                       | Operation status                                         |
@@ -214,7 +224,7 @@ acquisition `header` remain independent of an unavailable origin source.
 | Selected origin-backed channel received HTTP `4xx`/`5xx`      | Exact durable `http-response-status` Problem             |
 | Selected HTML-page channel fails                              | That channel's exact durable producer Problem            |
 | Local HTML projection is absent                               | `422` (`no-readable-projection`)                         |
-| Recoverable Tavily failure uses local projection              | Readable body with durable status `203`                  |
+| Recoverable materializer failure uses local projection        | Readable body with durable status `203`                  |
 | Finite textual or projected-binary response                   | Universal READ result                                    |
 | Finite empty text response                                    | `204`                                                    |
 | Finite binary response has no readable projection             | `415` (`binary-response-unsupported`)                    |
@@ -235,8 +245,8 @@ acquisition `header` remain independent of an unavailable origin source.
 An HTTP error status is still a successfully acquired direct response: its
 content and response evidence are preserved, and origin-backed channels carry
 that exact status as durable producer evidence. HTML still enters page
-production, so a successful Tavily `body` can remain readable while the origin
-`html` records the HTTP error. Automatic WebFetcher acquisition instead treats
+production, so a successful materializer `body` can remain readable while the
+origin `html` records the HTTP error. Automatic WebFetcher acquisition instead treats
 a non-2xx response as unavailable. Handler failures use RFC 9457 Problem Details.
 Caught direct-acquisition diagnostics are bounded by
 `PLURNK_SCHEMES_HTTP_ERROR_DETAIL_LIMIT` in model-facing detail while complete
@@ -260,7 +270,7 @@ handler does not sniff or guess unknown bytes.
 | ----------------- | --------------------------------------------------------------------------------------------------------- |
 | Platform          | Node ≥26 native fetch, streams, abort signals, decoding, DNS, and `WebSocket`                             |
 | SSE               | `eventsource-parser` for bounded WHATWG event-stream framing                                              |
-| Tavily Extract    | Optional Tavily Extract API primitive when `TAVILY_API_KEY` is present                                    |
+| Materializer plugins | Discovered `http-materializer` packages selected by `PLURNK_SCHEMES_HTTP_MATERIALIZER` {§http-materializer-plugins} |
 
 ### §http-config Operator configuration
 
@@ -268,9 +278,7 @@ handler does not sniff or guess unknown bytes.
 | --------------------- | -------------------------------------------------------------------------------------------------------- |
 | Canonical registry    | Shipped `.env.defaults`; the daemon assembles it as a set-if-unset floor {§operator-config-env-defaults} |
 | Required values       | Missing or invalid required values fail at readiness/owning read; code carries no hidden fallback        |
-| Tavily API Key        | Optional `# TAVILY_API_KEY=`; enables Tavily Extract for eligible public HTML                            |
-| Tavily Extract Depth  | `PLURNK_SCHEMES_HTTP_TAVILY_DEPTH=basic`; only `basic` or `advanced`                                     |
-| Tavily client timeout | Positive `PLURNK_SCHEMES_HTTP_TAVILY_TIMEOUT_MS=30000`                                                   |
+| Materializer selection | `PLURNK_SCHEMES_HTTP_MATERIALIZER=` names a discovered materializer id; unset = local projection only |
 
 ### §automatic-fetch-check Automatic acquisition URL check
 
@@ -279,7 +287,7 @@ acquisition, it accepts credential-free HTTP(S) targets only when every
 resolved address is ordinary globally reachable unicast, and repeats the check
 before every manually followed redirect. A refused or unresolvable target is
 the ordinary unavailable `null` result. A transport failure after public
-admission may still enter Tavily page production.
+admission may still enter materializer page production.
 
 Direct HTTP operations and WebSocket connections do not use this check. Loopback,
 private, and link-local destinations are therefore valid explicit targets.
@@ -296,12 +304,12 @@ check therefore makes no DNS-rebinding or total-egress claim.
 
 | Concern        | Contract                                                                                                   |
 | -------------- | ---------------------------------------------------------------------------------------------------------- |
-| Direct gate    | Explicit targets use native origin transport; only a generic public request grants Tavily authority          |
+| Direct gate    | Explicit targets use native origin transport; only a generic public request grants materializer authority          |
 | Automatic gate | Target and redirects require public admission; accepted generic HTML follows the same page producer       |
-| Body owner     | Origin Markdown wins; otherwise configured eligible Tavily is structural, with one local projection floor  |
+| Body owner     | Origin Markdown wins; otherwise the configured eligible materializer is structural, with one local projection floor  |
 | Source owner   | `html` is exact server-source HTML; it is never provider-generated or DOM-generated                         |
 | Projection     | A present projection, including `""`, becomes `body`; `null` alone means absence                           |
-| Cancellation   | One caller signal spans origin, auxiliary origin, projection, and Tavily work                              |
+| Cancellation   | One caller signal spans origin, auxiliary origin, projection, and materializer work                              |
 
 ### §host-rewrite Acquisition target rewrite
 
@@ -317,7 +325,7 @@ are checked under {§automatic-fetch-check} only when `WebFetcher` acquires them
 Acquired responses append package-owned `x-plurnk-request-method`,
 `x-plurnk-fetched-at`, and `x-plurnk-cache-variant` fields after origin headers.
 Page bodies append `x-plurnk-materializer-id`; local projections append
-`x-plurnk-projection-id`; Tavily appends status, route, request, usage, timing,
+`x-plurnk-projection-id`; The materializer appends its own evidence headers;
 and bounded failure evidence. Only package metadata after the acquisition stamp
 is authoritative.
 Plurnk stores one representation per canonical URL rather than a variant set:
@@ -388,16 +396,16 @@ different source representation.
 
 A stored derived representation requires the current projection identity. A
 stored page body also requires its current route: negotiated origin Markdown,
-local projection while Tavily is unconfigured, metadata-ineligible local
-projection, configured Tavily depth/version, or the corresponding recoverable
-local-fallback route. Enabling Tavily or changing depth invalidates the affected
-route.
+local projection while no materializer is selected, metadata-ineligible local
+projection, the configured materializer id, or the corresponding recoverable
+local-fallback route. Selecting or changing the materializer invalidates the
+affected route.
 
 Once a page representation leaves its fresh window, Plurnk performs complete
 reacquisition without old origin validators. Origin `304` can certify origin
-bytes, but it cannot certify a composite that may also contain a fresh Tavily
-extraction, a newly negotiated Markdown representation, or an independently
-acquired HTML variant. Projection or materializer mismatch likewise invalidates
+bytes, but it cannot certify a composite that may also contain a fresh
+materializer extraction, a newly negotiated Markdown representation, or an
+independently acquired HTML variant. Projection or materializer mismatch likewise invalidates
 body, TTL, and validators together.
 
 POST, PUT, and DELETE responses retain their method marker but cannot satisfy a
@@ -435,7 +443,8 @@ operation is cancelled.
 Top-level `null` is an automatic-acquisition liveness value rather than a thrown
 failure. Caller cancellation is not liveness: a pre-aborted caller fails before
 acquisition and a caller abort wins with its exact reason. Package-owned origin
-and Tavily timeouts are ordinary classified acquisition outcomes. WebFetcher
+timeouts are ordinary classified acquisition outcomes; materializer timeouts
+are the materializer's own classified outcomes. WebFetcher
 owns no entry identity, registry selection, selected-channel policy, or query
 policy; its consumers supply those boundaries and the projection capability.
 

package/dist/Http.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"Http.d.ts","sourceRoot":"","sources":["../src/Http.ts"],"names":[],"mappings":"AAKA,OAAO,KAAK,EACR,SAAS,EAGT,iBAAiB,EACjB,cAAc,EACd,aAAa,EACb,gCAAgC,EAChC,+BAA+B,EAC/B,aAAa,EACb,qBAAqB,EACrB,aAAa,EAOhB,MAAM,wBAAwB,CAAC;AAuMhC,MAAM,CAAC,OAAO,OAAO,IAAK,YAAW,aAAa;;IAC9C,MAAM,CAAC,QAAQ,EAAE,cAAc,CA2B7B;IAIF,cAGC;IAEK,KAAK,IAAI,OAAO,CAAC,IAAI,CAAC,CAE3B;IAKK,qBAAqB,CACvB,OAAO,EAAE,gCAAgC,EACzC,GAAG,EAAE,SAAS,GACf,OAAO,CAAC,+BAA+B,CAAC,CAS1C;IAkNK,SAAS,CAAC,UAAU,EAAE,SAAS,qBAAqB,EAAE,EAAE,GAAG,EAAE,SAAS,GAAG,OAAO,CAAC,iBAAiB,CAAC,CAqCxG;IAEK,IAAI,CAAC,SAAS,EAAE,qBAAqB,EAAE,GAAG,EAAE,SAAS,GAAG,OAAO,CAAC,iBAAiB,CAAC,CAEvF;IAIK,IAAI,CAAC,SAAS,EAAE,aAAa,EAAE,GAAG,EAAE,SAAS,GAAG,OAAO,CAAC,iBAAiB,CAAC,CAS/E;IAQK,IAAI,CAAC,SAAS,EAAE,aAAa,EAAE,GAAG,EAAE,SAAS,GAAG,OAAO,CAAC,iBAAiB,CAAC,CAoC/E;CAypBJ"}
+{"version":3,"file":"Http.d.ts","sourceRoot":"","sources":["../src/Http.ts"],"names":[],"mappings":"AAKA,OAAO,KAAK,EACR,SAAS,EAGT,iBAAiB,EACjB,cAAc,EACd,aAAa,EACb,gCAAgC,EAChC,+BAA+B,EAC/B,aAAa,EACb,qBAAqB,EACrB,aAAa,EAOhB,MAAM,wBAAwB,CAAC;AAuMhC,MAAM,CAAC,OAAO,OAAO,IAAK,YAAW,aAAa;;IAC9C,MAAM,CAAC,QAAQ,EAAE,cAAc,CA2B7B;IAQF,cAGC;IAEK,KAAK,IAAI,OAAO,CAAC,IAAI,CAAC,CAE3B;IAKK,qBAAqB,CACvB,OAAO,EAAE,gCAAgC,EACzC,GAAG,EAAE,SAAS,GACf,OAAO,CAAC,+BAA+B,CAAC,CAS1C;IAsRK,SAAS,CAAC,UAAU,EAAE,SAAS,qBAAqB,EAAE,EAAE,GAAG,EAAE,SAAS,GAAG,OAAO,CAAC,iBAAiB,CAAC,CAqCxG;IAEK,IAAI,CAAC,SAAS,EAAE,qBAAqB,EAAE,GAAG,EAAE,SAAS,GAAG,OAAO,CAAC,iBAAiB,CAAC,CAEvF;IAIK,IAAI,CAAC,SAAS,EAAE,aAAa,EAAE,GAAG,EAAE,SAAS,GAAG,OAAO,CAAC,iBAAiB,CAAC,CAS/E;IAQK,IAAI,CAAC,SAAS,EAAE,aAAa,EAAE,GAAG,EAAE,SAAS,GAAG,OAAO,CAAC,iBAAiB,CAAC,CAoC/E;CA8qBJ"}

package/dist/Http.js

@@ -6,7 +6,7 @@ import { createParser } from "eventsource-parser";
 import { MimetypeClassifier, NetworkAddress, ProjectionInputLimitError, Results, } from "@plurnk/plurnk-schemes";
 import { readFile } from "node:fs/promises";
 import ErrorDetail from "./ErrorDetail.js";
-import WebFetcher, { DEFAULT_WEB_UA, CACHE_VARIANT_HEADER, MATERIALIZER_ID_HEADER, PROJECTION_ID_HEADER, TAVILY_CREDITS_HEADER, TAVILY_REQUEST_ID_HEADER, cacheVariantEvidence, classifyCacheVariant, WebMaterializationError, } from "./WebFetcher.js";
+import WebFetcher, { DEFAULT_WEB_UA, CACHE_VARIANT_HEADER, MATERIALIZER_ID_HEADER, PROJECTION_ID_HEADER, cacheVariantEvidence, classifyCacheVariant, WebMaterializationError, } from "./WebFetcher.js";
 import { responseMimetype } from "./ContentType.js";
 import { requireNonNegativeIntegerEnv as requireNumEnv } from "./Config.js";
 // The channel the response body streams into, and the header metadata channel.
@@ -135,6 +135,7 @@ const httpDate = (value, now = Date.now()) => {
 // The package-shipped model teaching is loaded verbatim and fails hard if absent.
 // The relative path is identical from src/ during development and dist/ after build.
 const documentation = await readFile(new URL("../docs/http.md", import.meta.url), "utf-8");
+const LLMS_TEXT_ATTEMPT_TTL_MS = 3_600_000;
 class Http {
     static manifest = {
         name: "http",
@@ -166,6 +167,10 @@ class Http {
     };
     #errorDetailLimit;
     #webFetcher;
+    // {§http-llms-txt} — one opportunistic origin-companion attempt per TTL
+    // window; success and failure share the timer so a 404 does not re-probe
+    // on every READ.
+    #llmsTextAttempts = new Map();
     constructor() {
         this.#errorDetailLimit = ErrorDetail.configuredLimit();
         this.#webFetcher = new WebFetcher();
@@ -264,7 +269,49 @@ class Http {
                 });
             }
             const responseHeaders = fetched.responseHeaders ?? [];
-            if (!_a.#revalidationCorresponds(cachedHeader, new Headers(responseHeaders.map(([name, value]) => [name, value])))) {
+            if (_a.#revalidationCorresponds(cachedHeader, new Headers(responseHeaders.map(([name, value]) => [name, value])))) {
+                const channels = {
+                    ...cached.channels,
+                    [HEADER]: {
+                        ...cached.channels[HEADER],
+                        content: _a.#refreshAfter304(cachedHeader, responseHeaders, requestHeaders),
+                    },
+                };
+                const written = await ctx.entries.write(pathname, {
+                    channels,
+                });
+                if (Results.isErrorStatus(written.status))
+                    return _a.#passthrough(written);
+                return { status: 200 };
+            }
+            // {§revalidation} — a genuinely mismatched 304 (different opaque
+            // tags): the conditional is the problem, so fall back to one
+            // unconditional GET and acquire normally instead of surfacing
+            // an unrecoverable 502. A second 304 has no way out and is the
+            // honest failure.
+            try {
+                fetched = await this.#webFetcher.fetch(url, {
+                    signal: ctx.signal,
+                    headers: requestHeaders,
+                    guarded: false,
+                    acceptHttpErrors: true,
+                    preserveUnavailable: true,
+                });
+            }
+            catch (error) {
+                if (ctx.signal?.aborted === true && error === ctx.signal.reason) {
+                    return _a.#cancelled(url, "GET");
+                }
+                throw error;
+            }
+            if (fetched === null) {
+                return _a.#bad(404, "http", "not-materialized", `The URL ${url} could not be materialized.`, {
+                    target: url,
+                    stage: "acquisition",
+                    retryable: true,
+                });
+            }
+            if (fetched.status === 304) {
                 return _a.#bad(502, "http", "fetch-failed", `HTTP GET ${url} returned 304 without identifying the stored representation nominated for revalidation.`, {
                     target: url,
                     method: "GET",
@@ -272,19 +319,7 @@ class Http {
                     retryable: true,
                 });
             }
-            const channels = {
-                ...cached.channels,
-                [HEADER]: {
-                    ...cached.channels[HEADER],
-                    content: _a.#refreshAfter304(cachedHeader, responseHeaders, requestHeaders),
-                },
-            };
-            const written = await ctx.entries.write(pathname, {
-                channels,
-            });
-            if (Results.isErrorStatus(written.status))
-                return _a.#passthrough(written);
-            return { status: 200 };
+            // Non-304: fall through to ordinary acquisition below.
         }
         if (fetched.mimetype === "text/event-stream"
             && fetched.response?.body !== null
@@ -318,8 +353,46 @@ class Http {
         });
         if (Results.isErrorStatus(written.status))
             return _a.#passthrough(written);
+        await this.#piggybackLlmsText(address, ctx);
         return { status: 200 };
     }
+    // {§http-llms-txt} — after a successful generic GET materialization,
+    // acquire <origin>/llms.txt once per TTL window and materialize it as its
+    // own https entry. Any failure is quiet: the companion never fails the
+    // READ that piggybacked it, and the companion itself never recurses.
+    async #piggybackLlmsText(address, ctx) {
+        const url = new URL(address.url);
+        if (url.pathname === "/llms.txt")
+            return;
+        const origin = url.origin;
+        const last = this.#llmsTextAttempts.get(origin);
+        if (last !== undefined && Date.now() - last < LLMS_TEXT_ATTEMPT_TTL_MS)
+            return;
+        this.#llmsTextAttempts.set(origin, Date.now());
+        const llmsUrl = `${origin}/llms.txt`;
+        try {
+            const fetched = await this.#webFetcher.fetch(llmsUrl, {
+                signal: ctx.signal,
+                guarded: false,
+                acceptHttpErrors: true,
+                preserveUnavailable: true,
+            });
+            if (fetched === null)
+                return;
+            // A missing companion (404) or any non-2xx is quiet — no entry.
+            if ((fetched.status ?? 200) >= 400)
+                return;
+            const materialized = await WebFetcher.materialize(fetched, ctx.projection, ctx.signal);
+            if (materialized === null)
+                return;
+            await ctx.entries.write(`/${url.host}/llms.txt`, {
+                channels: WebFetcher.materializedChannels(materialized, { url: llmsUrl, method: "GET" }),
+            });
+        }
+        catch {
+            // Quiet by contract: 404, guard rejection, network error, binary.
+        }
+    }
     async #openEventStream(address, fetched, ctx) {
         const response = fetched.response;
         if (response?.body === null || response?.body === undefined) {
@@ -860,11 +933,9 @@ class Http {
         const isHtml = MimetypeClassifier.isHtml(sourceMimetype);
         const materializerIdentity = _a.#materializerIdentity(header);
         if (materializerIdentity !== undefined) {
-            if (!WebFetcher.materializerCurrent(materializerIdentity))
-                return false;
-            if (materializerIdentity.startsWith("tavily-extract:")
-                || materializerIdentity === "origin-markdown:v1")
-                return true;
+            // {§http-materializer-plugins} — a stored body produced by the current
+            // configured materializer (or the built-in producers) is current.
+            return WebFetcher.materializerCurrent(materializerIdentity);
         }
         if (isHtml && materializerIdentity === undefined)
             return false;
@@ -886,15 +957,34 @@ class Http {
             ...retained.map(({ name, value }) => [name, value]),
             ...updates,
         ];
-        const packageEvidence = [
-            MATERIALIZER_ID_HEADER,
-            PROJECTION_ID_HEADER,
-            TAVILY_REQUEST_ID_HEADER,
-            TAVILY_CREDITS_HEADER,
-        ].flatMap((field) => {
+        // Materializer/provider evidence headers are plugin-owned (unknown names);
+        // a 304 does not re-materialize the stored body, so its evidence is
+        // retained verbatim beside the rebuilt framework fields. Framework-owned
+        // x-plurnk-* fields (cache-variant, materializer/projection ids, method,
+        // fetched-at stamp) are rebuilt or omitted, never duplicated stale.
+        const rebuilt = [MATERIALIZER_ID_HEADER, PROJECTION_ID_HEADER].flatMap((field) => {
             const value = _a.#packageHeaderValue(header, field);
             return value === undefined ? [] : [`${field}: ${value}`];
         });
+        const frameworkOwned = new Set([
+            CACHE_VARIANT_HEADER.toLowerCase(),
+            MATERIALIZER_ID_HEADER.toLowerCase(),
+            PROJECTION_ID_HEADER.toLowerCase(),
+            REQUEST_METHOD.toLowerCase(),
+            FETCHED_AT.toLowerCase(),
+        ]);
+        const packageEvidence = [
+            ...rebuilt,
+            ...lines.slice(1).filter((line) => {
+                const colon = line.indexOf(":");
+                if (colon <= 0)
+                    return false;
+                const name = line.slice(0, colon).toLowerCase();
+                return name.startsWith("x-plurnk-")
+                    && !frameworkOwned.has(name)
+                    && !updatedNames.has(name);
+            }),
+        ];
         return [
             lines[0] ?? "HTTP 200 OK",
             ...retained.map(({ line }) => line),
@@ -916,8 +1006,11 @@ class Http {
                 : null;
             if (responseTag === null || storedTag === null)
                 return false;
-            return responseTag.opaque === storedTag.opaque
-                && (responseTag.weak || !storedTag.weak);
+            // {§revalidation} — If-None-Match correspondence is a weak
+            // comparison (RFC 9110 §8.8.3.2): opaque tags match regardless of
+            // the W/ prefix, so an origin that upgrades a stored weak etag to
+            // a strong etag on the 304 still corresponds.
+            return responseTag.opaque === storedTag.opaque;
         }
         const responseModifiedValue = responseHeaders.get("last-modified");
         const storedModifiedValues = _a.#originValues(fields, "last-modified");