@plurnk/plurnk-schemes-http 2.0.5 → 2.0.6
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +4 -4
- package/SPEC.md +122 -606
- package/dist/Guard.d.ts +4 -0
- package/dist/Guard.d.ts.map +1 -1
- package/dist/Guard.js +43 -5
- package/dist/Guard.js.map +1 -1
- package/dist/Http.d.ts +1 -1
- package/dist/Http.d.ts.map +1 -1
- package/dist/Http.js +22 -619
- package/dist/Http.js.map +1 -1
- package/dist/HttpCache.d.ts +10 -0
- package/dist/HttpCache.d.ts.map +1 -0
- package/dist/HttpCache.js +384 -0
- package/dist/HttpCache.js.map +1 -0
- package/dist/HttpGet.d.ts +5 -26
- package/dist/HttpGet.d.ts.map +1 -1
- package/dist/HttpGet.js +30 -63
- package/dist/HttpGet.js.map +1 -1
- package/dist/HttpRequester.d.ts +4 -15
- package/dist/HttpRequester.d.ts.map +1 -1
- package/dist/HttpRequester.js +31 -49
- package/dist/HttpRequester.js.map +1 -1
- package/dist/HttpShared.d.ts +42 -0
- package/dist/HttpShared.d.ts.map +1 -0
- package/dist/HttpShared.js +244 -0
- package/dist/HttpShared.js.map +1 -0
- package/dist/WebFetcher.d.ts +2 -7
- package/dist/WebFetcher.d.ts.map +1 -1
- package/dist/WebFetcher.js +14 -26
- package/dist/WebFetcher.js.map +1 -1
- package/dist/Ws.d.ts.map +1 -1
- package/dist/Ws.js +11 -7
- package/dist/Ws.js.map +1 -1
- package/dist/index.d.ts +0 -4
- package/dist/index.d.ts.map +1 -1
- package/dist/index.js +2 -5
- package/dist/index.js.map +1 -1
- package/docs/https.md +2 -2
- package/docs/wss.md +1 -1
- package/package.json +5 -5
- package/dist/Config.d.ts +0 -5
- package/dist/Config.d.ts.map +0 -1
- package/dist/Config.js +0 -13
- package/dist/Config.js.map +0 -1
- package/dist/ContentType.d.ts +0 -2
- package/dist/ContentType.d.ts.map +0 -1
- package/dist/ContentType.js +0 -15
- package/dist/ContentType.js.map +0 -1
- package/dist/ErrorDetail.d.ts +0 -5
- package/dist/ErrorDetail.d.ts.map +0 -1
- package/dist/ErrorDetail.js +0 -5
- package/dist/ErrorDetail.js.map +0 -1
- package/dist/HostPolicy.d.ts +0 -11
- package/dist/HostPolicy.d.ts.map +0 -1
- package/dist/HostPolicy.js +0 -58
- package/dist/HostPolicy.js.map +0 -1
- package/dist/LiveAcquisitions.d.ts +0 -8
- package/dist/LiveAcquisitions.d.ts.map +0 -1
- package/dist/LiveAcquisitions.js +0 -30
- package/dist/LiveAcquisitions.js.map +0 -1
- package/dist/http-names.d.ts +0 -5
- package/dist/http-names.d.ts.map +0 -1
- package/dist/http-names.js +0 -6
- package/dist/http-names.js.map +0 -1
package/SPEC.md
CHANGED
|
@@ -1,640 +1,156 @@
|
|
|
1
|
-
# plurnk-schemes-http
|
|
1
|
+
# `@plurnk/plurnk-schemes-http` Specification
|
|
2
2
|
|
|
3
|
-
|
|
4
|
-
full-duplex scheme, plus automatic entry-acquisition, pluggable page
|
|
5
|
-
materializers, and the local HTML projection foundation. Both handlers implement
|
|
6
|
-
the DB-free `SchemeCtx` author contract.
|
|
3
|
+
Owns the `http(s)://` request/response scheme (`Http`), the `ws(s)://` full-duplex WebSocket scheme (`Ws`), automatic web acquisition (`WebFetcher`), URL/host policy (`Guard`, `HostPolicy`), and pluggable HTML materializers (`MaterializerRegistry`).
|
|
7
4
|
|
|
8
|
-
## §http-manifest
|
|
5
|
+
## §http-manifest HTTP manifest and operation surface
|
|
9
6
|
|
|
10
|
-
|
|
11
|
-
| --------------- | ----------------------------- |
|
|
12
|
-
| Registered name | `https` (plain `http` routes to it — supported, never advertised as a peer endpoint) |
|
|
13
|
-
| Category | `data` |
|
|
14
|
-
| Writers | `model`, `client` |
|
|
15
|
-
| Volatile | `true` |
|
|
16
|
-
| Model-visible | `true` |
|
|
17
|
-
| Requires web | `true` |
|
|
18
|
-
| Metadata modifier | `true` |
|
|
19
|
-
| Default channel | `body` |
|
|
7
|
+
`Http` registers as `https` (plain `http` routes to the same handler; `package.json#plurnk.schemes` registers `http` via default export and `wss` via `Ws`) with `category: "data"`, `writableBy: ["model", "client"]`, `volatile: true`, `modelVisible: true`, `metadataModifier: true`, `traits: ["web"]`, and channels:
|
|
20
8
|
|
|
21
|
-
| Channel
|
|
22
|
-
|
|
23
|
-
| `body`
|
|
24
|
-
| `header`
|
|
25
|
-
| `readable` | `text/markdown`
|
|
9
|
+
| Channel | Seed type | Meaning |
|
|
10
|
+
|---|---|---|
|
|
11
|
+
| `body` (default) | `application/octet-stream` | Source text or original binary bytes; SSE `data` lines for a live response |
|
|
12
|
+
| `header` | `text/plain` | Origin status/headers, acquisition stamps, cache variant, materializer, and projection evidence |
|
|
13
|
+
| `readable` | `text/markdown` | Optional derived text/facts from an HTML materializer or installed mimetype projection ({§channel-selection}) |
|
|
26
14
|
|
|
27
|
-
`
|
|
28
|
-
`wss` through `Ws`.
|
|
15
|
+
Identity and transport URLs normalize through `NetworkAddress` ({§network-address}): authority is `<host>[:<non-default-port>]`, pathname is `<path>[?<serialized-query>]` (`%28`/`%29` in path canonicalize to literal `(` / `)`, query order/duplicates/empty `?` preserved), `#fragment` selects the Plurnk channel, and URL userinfo is rejected (`400`).
|
|
29
16
|
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
17
|
+
| Operation | Remote action | Contract |
|
|
18
|
+
|---|---|---|
|
|
19
|
+
| Exact `READ (url) <scope?>` | `GET` (unless a fresh cached `GET` is usable) | Scope-blind `prepareRepresentation`, then Core selects channel and scope |
|
|
20
|
+
| Exact `FIND (url)` | `GET` only when acquisition is required | Same preparation, then universal query, matcher, weighting, and pagination |
|
|
21
|
+
| Pattern `FIND (pattern-url)` | None (`PathSyntax.hasGlob`, {§path-syntax}) | Query already-materialized web entries in the workspace without remote discovery |
|
|
22
|
+
| `SEND (url)` with body | `POST` | Proposes (`effect: "host"`, `202`); on apply, streams and persists response under `url` |
|
|
23
|
+
| `EDIT (url)` with body | `PUT` | Proposes (`effect: "host"`, `202`); replaces whole remote resource (line markers fail `400`: `Remove the line range and submit the complete replacement body.`) |
|
|
24
|
+
| `KILL (url)` | None | Cancels all live workspace acquisitions of `url` (owners settle `499`), else deletes local stored entry |
|
|
25
|
+
| `KILL (url) [{"remote": true}]` | `DELETE` | Proposes (`effect: "host"`, `202`); remaining options in the metadata block become request headers |
|
|
39
26
|
|
|
40
|
-
|
|
27
|
+
- **Headers and proposals (`http-outbound-proposes`):** Request headers are passed in one `[{"Key": "value"}]` metadata block ({§slot-order}). `POST`, `PUT`, remote `DELETE`, and WebSocket `EDIT`/`SEND` frames declare `effect: "host"` and return `202` ({§proposal}, {§exec-lifetime}), governed by `PLURNK_SERVICE_EFFECT_HOST`. `GET` and WebSocket `READ` connection opens are observational and ungated; a WebSocket frame checks for an open socket both before proposing and again at `applyResolution`, and the resolver may replace the proposed body before it is sent.
|
|
41
28
|
|
|
42
|
-
|
|
43
|
-
| ------------------------------------------ | ------------------------------------- | --------------------------------------------------------------------------------------- |
|
|
44
|
-
| Exact ```` ```READ (url) <scope?> ```` | GET unless a fresh GET copy is usable | Prepare one complete canonical representation, then let core select and project it |
|
|
45
|
-
| Exact ```` ```FIND (url) ```` | GET only when acquisition is required | Use the same preparation, then universal query, matcher, weighting, and pagination |
|
|
46
|
-
| ```` ```FIND (pattern-url) ```` | None | Query already-materialized web entries; a path pattern does not discover the remote web |
|
|
47
|
-
| ```` ```SEND (url) ```` with body | POST | Stream and persist the response under the addressed URL |
|
|
48
|
-
| ```` ```EDIT (url) ```` with body | PUT | Replace the whole remote resource; a line marker is invalid |
|
|
49
|
-
| ```` ```KILL (url) ```` | None | {§http-kill}: cancel the workspace's live acquisitions of the URL, else delete the local stored entry |
|
|
50
|
-
| ```` ```KILL (url) [{"remote": true}] ```` | DELETE | Delete the remote resource and stream its response; other options in the same metadata block are its headers |
|
|
51
|
-
|
|
52
|
-
Finite GET uses scope-blind representation preparation; POST, PUT, DELETE,
|
|
53
|
-
and genuinely live GET responses retain the subscription path. Request
|
|
54
|
-
headers are options in one `[{"Key": "value", ...}]` metadata block under
|
|
55
|
-
{§scheme-metadata-modifier}. HTTP interprets the merged options; the resource
|
|
56
|
-
target remains pure. Remote status and headers are persisted in `header`, not
|
|
57
|
-
interpreted as Plurnk operation or loop lifecycle signals.
|
|
58
|
-
Exact-versus-pattern FIND preparation uses the shared
|
|
59
|
-
`PathSyntax.hasGlob` classifier {§path-glob}; HTTP owns no reduced
|
|
60
|
-
path-pattern grammar.
|
|
61
|
-
|
|
62
|
-
§http-outbound-proposes **A request that changes a remote resource proposes.** POST, PUT, the remote DELETE, and a WebSocket frame (EDIT or SEND on an open `ws(s)://` socket) declare `effect: "host"` and return `202` under {§proposal}; the panel's `PLURNK_SERVICE_EFFECT_HOST` decides whether that settles by consent or runs unattended, exactly as it does for a subprocess ({§exec-host-proposes}). This is the same reason a local `EDIT` proposes: Plurnk does not claim containment, so consent — not a boundary — is what stands between the model and the world.
|
|
63
|
-
|
|
64
|
-
| Fact | Rule |
|
|
65
|
-
| --- | --- |
|
|
66
|
-
| GET, and the socket open a `ws(s)://` READ performs | observation, ungated: an acquisition leaks only what a URL can carry, and that is the same for both |
|
|
67
|
-
| A frame | needs an open socket, so that refusal arrives before any proposal — nobody is asked to approve a message to a connection that is not there — and the admission runs again at apply, because the socket is live and may have closed while the question was open |
|
|
68
|
-
| The proposal | carries the method, target, and body, and the resolver may replace the body before it is sent; nothing leaves the process until the settlement accepts |
|
|
69
|
-
|
|
70
|
-
## §http-lifecycle §3 Acquisition, materialization, and query lifecycle
|
|
71
|
-
|
|
72
|
-
<!-- map -->
|
|
73
|
-
| Tag | Promise |
|
|
74
|
-
| --- | --- |
|
|
75
|
-
| {§http-text-decoding} | Text response decoding |
|
|
76
|
-
| {§http-binary-source} | Finite binary responses retain bounded original bytes in `body`, with optional derived facts/text in… |
|
|
77
|
-
| {§http-json-presentation} | Finite `application/json` and `+json` responses use {§json-document-presentation} in their canonical body… |
|
|
78
|
-
| {§html-materialization} | Readable materialization |
|
|
79
|
-
| {§http-materializer-extensions} | Materializer extensions |
|
|
80
|
-
| {§http-channel-outcomes} | Channel outcomes |
|
|
81
|
-
| {§http-llms-txt} | Origin llms.txt companions. |
|
|
82
|
-
<!-- /map -->
|
|
29
|
+
## §http-lifecycle Acquisition, decoding, and materialization
|
|
83
30
|
|
|
84
31
|
```mermaid
|
|
85
32
|
flowchart TD
|
|
86
33
|
get["GET acquisition"] --> live{"text/event-stream?"}
|
|
87
|
-
live -->|yes|
|
|
34
|
+
live -->|yes| sse["Seed canonical channels, retain subscription, return 102"]
|
|
88
35
|
live -->|no| origin{"Finite origin outcome"}
|
|
89
|
-
origin -->|text
|
|
90
|
-
origin -->|binary|
|
|
91
|
-
origin -->|server HTML|
|
|
92
|
-
|
|
93
|
-
|
|
94
|
-
|
|
95
|
-
eligible -->|no| local["Installed HTML projection"]
|
|
96
|
-
eligible -->|yes| mat["Materializer Markdown"]
|
|
97
|
-
mat -->|success| providerText["Materializer readable"]
|
|
98
|
-
mat -->|recoverable failure<br/>and source exists| recovery["Local readable; status 203"]
|
|
99
|
-
mat -->|hard failure or<br/>no recovery source| readableError["Readable errored"]
|
|
100
|
-
local -->|present, including empty| localText["Local readable"]
|
|
101
|
-
local -->|absent or no source| readableError
|
|
102
|
-
text --> settle["Write complete canonical channels<br/>and durable producer results"]
|
|
103
|
-
binary --> settle
|
|
104
|
-
providerText --> settle
|
|
105
|
-
recovery --> settle
|
|
106
|
-
localText --> settle
|
|
107
|
-
readableError --> settle
|
|
108
|
-
settle --> core["Core selects channel and scope<br/>native media follows ordinary READ"]
|
|
36
|
+
origin -->|text / negotiated Markdown| text["Fetch UTF-8 text in body (JSON formatted via contract-representations)"]
|
|
37
|
+
origin -->|binary| bin["Bounded original bytes in body + optional projection in readable"]
|
|
38
|
+
origin -->|server HTML or public transport loss| html{"Generic public request with materializer?"}
|
|
39
|
+
html -->|yes| mat["Materializer extract -> readable (203 local fallback on recoverable error)"]
|
|
40
|
+
html -->|no| local["Installed HTML projection -> readable"]
|
|
41
|
+
text & bin & mat & local --> settle["Write complete canonical channels & durable producerResults atomically"]
|
|
109
42
|
```
|
|
110
43
|
|
|
111
|
-
|
|
112
|
-
|
|
113
|
-
|
|
114
|
-
|
|
115
|
-
|
|
116
|
-
|
|
117
|
-
|
|
118
|
-
|
|
|
119
|
-
|
|
|
120
|
-
|
|
|
121
|
-
|
|
122
|
-
|
|
123
|
-
|
|
124
|
-
|
|
|
125
|
-
|
|
126
|
-
|
|
|
127
|
-
|
|
|
128
|
-
|
|
|
129
|
-
|
|
130
|
-
|
|
131
|
-
|
|
132
|
-
|
|
133
|
-
|
|
134
|
-
|
|
135
|
-
|
|
136
|
-
|
|
137
|
-
|
|
138
|
-
|
|
139
|
-
|
|
140
|
-
|
|
141
|
-
|
|
|
142
|
-
|
|
143
|
-
|
|
|
144
|
-
|
|
|
145
|
-
|
|
|
146
|
-
|
|
|
147
|
-
|
|
148
|
-
|
|
149
|
-
|
|
150
|
-
|
|
151
|
-
|
|
152
|
-
|
|
153
|
-
|
|
154
|
-
|
|
155
|
-
|
|
156
|
-
|
|
157
|
-
|
|
158
|
-
|
|
159
|
-
|
|
160
|
-
|
|
161
|
-
|
|
162
|
-
{§
|
|
163
|
-
|
|
164
|
-
|
|
165
|
-
received partial text and the acquisition failure. Request bodies, authored
|
|
166
|
-
entries, non-JSON text, SSE, and JSONL retain their original formatting.
|
|
167
|
-
|
|
168
|
-
### §html-materialization Readable materialization
|
|
169
|
-
|
|
170
|
-
`WebFetcher.materialize` is the shared source/projection seam for exact
|
|
171
|
-
GET/FIND preparation and executor entry acquisition; `Http` publishes its
|
|
172
|
-
`WebFetcher` as the framework's {§web-materialization-contract}. It returns complete
|
|
173
|
-
source and derived channels for one atomic canonical entry write; SSE and
|
|
174
|
-
textual mutation responses retain incremental streaming.
|
|
175
|
-
|
|
176
|
-
| Input or event | Action | Result |
|
|
177
|
-
| ------------------------------------------- | ------------------------------------------- | ------------------------------------------------------------------ |
|
|
178
|
-
| Configured non-HTML text | Decode with Fetch UTF-8 | Original representation in `body` |
|
|
179
|
-
| Configured binary with reader | Retain bounded bytes and apply projection | Original `body`, derived `readable`, and projection identity |
|
|
180
|
-
| Configured binary without reader | Retain bounded bytes | Original `body` and projection identity; no `readable` |
|
|
181
|
-
| Binary input exceeds the common bound | Cancel and preserve the typed cause | `WebMaterializationError` caused by `ProjectionInputLimitError` |
|
|
182
|
-
| Negotiated origin `text/markdown` | Accept it without the materializer | Exact Markdown in `body`; no `readable`, no second request |
|
|
183
|
-
| Eligible HTML with a materializer selected | Use the materializer as the readable producer | Origin server source in `body`; materializer Markdown in `readable` |
|
|
184
|
-
| Ineligible HTML or no materializer selected | Use the installed HTML projection | Origin server source in `body`; local Markdown in `readable` |
|
|
185
|
-
| Recoverable materializer failure with HTML | Use the installed projection as recovery | Source in `body`; local Markdown in `readable` with terminal `203` |
|
|
186
|
-
| Hard materializer failure | Preserve evidence; do not run local recovery | `readable` errored; independently successful `body`/`header` survive |
|
|
187
|
-
| Materializer success after origin loss | Preserve provider text and origin failure | `readable` static; `body` errored with the origin's own failure |
|
|
188
|
-
| Projection implementation throws | Stop and preserve the cause | `WebMaterializationError` with stage `projection` |
|
|
189
|
-
|
|
190
|
-
A projection object is present even when its content is `""`; only `null`
|
|
191
|
-
denotes absence. A materialization exception retains its original `cause`; it
|
|
192
|
-
never enters the absence channel.
|
|
193
|
-
|
|
194
|
-
#### §http-materializer-extensions Materializer extensions
|
|
195
|
-
|
|
196
|
-
`PLURNK_SCHEMES_HTTP_MATERIALIZER` selects one discovered `http-materializer`
|
|
197
|
-
extension by id; unset means the installed HTML projection is the only readable
|
|
198
|
-
producer. A materializer package declares
|
|
199
|
-
`plurnk: { kind: "http-materializer", materializers: [{ id, module }] }` and
|
|
200
|
-
exports one `HttpMaterializer` per entry under the exec framework's
|
|
201
|
-
discovery, trust, and one-flat-id-namespace rules ({§extension-discovery}). The
|
|
202
|
-
selected materializer is consulted only for a credential-free generic request
|
|
203
|
-
whose target has been admitted as public. Authored request metadata—including
|
|
204
|
-
an authored `Accept` field—makes the request ineligible. The package-generated
|
|
205
|
-
Markdown negotiation, browser-compatible `User-Agent`, and conditional cache
|
|
206
|
-
fields are transport mechanics, not authored metadata. No authored request headers or
|
|
207
|
-
origin credentials cross the materializer boundary.
|
|
208
|
-
|
|
209
|
-
An npm-installed Agent Plugin puts the same declaration under
|
|
210
|
-
`plugin.json#extensions.ai.plurnk` ({§extension-manifest-read}); file entries live in
|
|
211
|
-
`ai.plurnk/`. Discovery and implementations are cached process-wide and loaded
|
|
212
|
-
lazily. A daemon instance neither registers nor removes materializers.
|
|
213
|
-
|
|
214
|
-
When no `Accept` is authored, origin acquisition offers
|
|
215
|
-
`text/markdown, text/html;q=0.9, */*;q=0.1`. An origin `text/markdown`
|
|
216
|
-
representation wins and the materializer is skipped; no second HTML request is
|
|
217
|
-
made. An authored `Accept` value is
|
|
218
|
-
sent unchanged and makes any returned HTML use the local projection route.
|
|
219
|
-
|
|
220
|
-
The materializer's `eligible(url)` returns its identity string when it will
|
|
221
|
-
serve the URL (its credentials and supported targets are its own); `extract`
|
|
222
|
-
returns `success` (its sanitized Markdown body plus its own evidence headers),
|
|
223
|
-
`recoverable` (local projection fallback with the materializer's evidence and
|
|
224
|
-
terminal `203`), or `hard` (an exact materializer-owned Problem; no local
|
|
225
|
-
recovery). A materializer `extract` throw is the materialization failure, never
|
|
226
|
-
a silent fallback. A selected id with no installed provider fails hard rather
|
|
227
|
-
than degrading silently.
|
|
228
|
-
|
|
229
|
-
Without server HTML, no local recovery input exists. A materializer success can
|
|
230
|
-
still close `readable`, but every materializer failure remains the readable
|
|
231
|
-
failure, and `body` closes errored with the origin's own failure.
|
|
232
|
-
|
|
233
|
-
#### §http-channel-outcomes Channel outcomes
|
|
234
|
-
|
|
235
|
-
For finite HTML-page production, `body`, `header`, and `readable` have independent
|
|
236
|
-
durable outcomes and are written atomically. The selected channel alone
|
|
237
|
-
determines the projected operation result: successful `#readable` or `#header` can
|
|
238
|
-
be read when `body` failed, and a successful default `body` is not invalidated
|
|
239
|
-
by an errored `readable`. Unselected failures remain `errored` channels carrying
|
|
240
|
-
their exact `producerResult`. A recoverable local projection remains a static
|
|
241
|
-
`readable` channel with producer status `203`, so cold and warm READ both return
|
|
242
|
-
its content with that same status. Every READ names the page's other channels
|
|
243
|
-
with their tokens ({§channel-selection-visibility}), so a READ of the source
|
|
244
|
-
shows that `#readable` exists before the model has ever listed the page.
|
|
245
|
-
|
|
246
|
-
Exact READ, exact FIND, and executor materialization preserve the same channel
|
|
247
|
-
representation. A missing source variant is an explicit empty `errored`
|
|
248
|
-
channel, never an absent fact that later cache use can reinterpret as successful
|
|
249
|
-
empty content. A finite origin `2xx` or `3xx` response completes ordinary local
|
|
250
|
-
production, independently of the origin's status meaning; `202` does not leave
|
|
251
|
-
an acquisition pending, and origin `203` is not a materializer recovery. Origin
|
|
252
|
-
`4xx`/`5xx` content remains available as evidence while origin-backed channels
|
|
253
|
-
carry the exact `http-response-status` producer Problem. A final status outside
|
|
254
|
-
`200`–`599` instead produces `502 invalid-response-status` with `originStatus`,
|
|
255
|
-
preserving the received content and header. A materializer-produced `readable`
|
|
256
|
-
and the acquisition `header` remain independent of an unavailable origin source.
|
|
257
|
-
Conditional `304` correspondence is checked before production under {§revalidation}.
|
|
258
|
-
|
|
259
|
-
§http-llms-txt **Origin llms.txt companions.** After a successful generic GET
|
|
260
|
-
materialization, the scheme opportunistically acquires `<origin>/llms.txt`
|
|
261
|
-
once per origin per TTL window and materializes it as its own origin entry
|
|
262
|
-
(`https://host/llms.txt`), surfaceable by FIND. A missing companion, any
|
|
263
|
-
non-2xx, binary content, or a transport failure is quiet: the companion is
|
|
264
|
-
never fabricated, the failed probe is remembered (not retried per READ), and
|
|
265
|
-
the piggyback never fails the READ that carried it. The companion itself
|
|
266
|
-
never recurses. This is lazy by construction — one extra request only when
|
|
267
|
-
an origin is first being read — and it is the operator's to allow:
|
|
268
|
-
`PLURNK_SCHEMES_HTTP_LLMS_TXT=0` makes only the requests a READ asks for.
|
|
269
|
-
|
|
270
|
-
## §http-status §4 HTTP status mapping
|
|
271
|
-
|
|
272
|
-
| Outcome | Operation status |
|
|
273
|
-
| ------------------------------------------------------------- | -------------------------------------------------------- |
|
|
274
|
-
| Finite origin `2xx`/`3xx` exact READ after preparation | Universal selected-channel result (`200` or `204`); origin status remains in `header` |
|
|
275
|
-
| Exact FIND after preparation | Exact universal query result |
|
|
276
|
-
| Exact acquisition returns no WebFetcher value | `404` (`not-materialized`) |
|
|
277
|
-
| Selected origin-backed channel received HTTP `4xx`/`5xx` | Exact durable `http-response-status` Problem |
|
|
278
|
-
| Selected origin-backed channel received an invalid final status | `502` (`invalid-response-status`) with `originStatus` |
|
|
279
|
-
| Selected HTML-page channel fails | That channel's exact durable producer Problem |
|
|
280
|
-
| Local HTML projection is absent | `422` (`no-readable-projection`) |
|
|
281
|
-
| Recoverable materializer failure uses local projection | `readable` with durable status `203` |
|
|
282
|
-
| Finite textual or binary response | Universal READ result |
|
|
283
|
-
| Finite empty text response | `204` |
|
|
284
|
-
| Binary projection input exceeds the configured byte bound | `413` (`projection-input-limit`) |
|
|
285
|
-
| `KILL` with no live acquisition | Exact entry-delete result |
|
|
286
|
-
| `KILL` of a live acquisition | `200`; the owner settles `499` through its aborted signal |
|
|
287
|
-
| Client-cancelled finite acquisition | `499` (`cancelled`) |
|
|
288
|
-
| SSE after acquisition | `102` initial; terminal selected-channel result |
|
|
289
|
-
| SSE cancellation after acquisition | `102` initial; terminal `499` |
|
|
290
|
-
| SSE parser/transfer failure after acquisition | `102` initial; terminal `502` |
|
|
291
|
-
| Multi-statement HTTP edit batch | `409` (`non-atomic-edit-batch`) |
|
|
292
|
-
| Invalid target, line edit, or URL userinfo | `400` with the corresponding stable Problem kind |
|
|
293
|
-
| Missing response channel | `404 channel-not-found` under {§channel-selection-missing}; no remote mutation |
|
|
294
|
-
| Non-corresponding 304 | `502` (`fetch-failed`) |
|
|
295
|
-
| Acquisition failure without successful materializer output | `502` (`fetch-failed`) |
|
|
296
|
-
| Projection exception | `500` (`projection-failed`) |
|
|
297
|
-
| Uninterpreted SEND status | `501` (`send-status-unsupported`) |
|
|
298
|
-
|
|
299
|
-
An HTTP error status is still a successfully acquired direct response: its
|
|
300
|
-
content and response evidence are preserved, and origin-backed channels carry
|
|
301
|
-
that exact status as durable producer evidence. HTML still enters page
|
|
302
|
-
production, so a successful materializer `readable` can remain readable while the
|
|
303
|
-
origin `body` records the HTTP error. Automatic WebFetcher acquisition instead treats
|
|
304
|
-
a non-2xx response as unavailable. Handler failures use RFC 9457 Problem Details.
|
|
305
|
-
Caught direct-acquisition diagnostics are bounded by
|
|
306
|
-
`PLURNK_SCHEMES_HTTP_ERROR_DETAIL_LIMIT` in model-facing detail while complete
|
|
307
|
-
errors remain in daemon diagnostics.
|
|
308
|
-
|
|
309
|
-
§http-replay A failed GET, PUT, or DELETE acquisition may recommend automatic
|
|
310
|
-
identical replay because those methods are idempotent. POST may already have
|
|
311
|
-
applied its effect when acquisition fails and is therefore non-retryable. Once
|
|
312
|
-
an SSE response has been acquired, a parser or transfer failure is likewise
|
|
313
|
-
non-retryable because replay could duplicate an already-consumed prefix.
|
|
314
|
-
|
|
315
|
-
Binary responses follow {§http-binary-source}, appending authoritative
|
|
316
|
-
`x-plurnk-projection-id` evidence after the origin and acquisition fields.
|
|
317
|
-
Exceeding the input ceiling returns `413` with configured and observed sizes;
|
|
318
|
-
a mutation response retains its lifecycle/header evidence and settles errored
|
|
319
|
-
without publishing partial bytes. This is a materialization failure, not the
|
|
320
|
-
remote HTTP outcome—a POST, PUT, or DELETE might already have changed the resource.
|
|
321
|
-
Missing or malformed `Content-Type` becomes `application/octet-stream`; the
|
|
322
|
-
handler does not sniff or guess unknown bytes.
|
|
323
|
-
|
|
324
|
-
## §5 Dependencies and configuration
|
|
325
|
-
|
|
326
|
-
<!-- map -->
|
|
327
|
-
| Tag | Promise |
|
|
328
|
-
| --- | --- |
|
|
329
|
-
| {§http-config} | Operator configuration |
|
|
330
|
-
| {§http-host-policy} | Operator web host policy |
|
|
331
|
-
| {§automatic-fetch-check} | Automatic acquisition URL check |
|
|
332
|
-
<!-- /map -->
|
|
333
|
-
|
|
334
|
-
| Surface | Runtime contract |
|
|
335
|
-
| ----------------- | --------------------------------------------------------------------------------------------------------- |
|
|
336
|
-
| Platform | Node ≥26 native fetch, streams, abort signals, decoding, DNS, and `WebSocket` |
|
|
337
|
-
| SSE | `eventsource-parser` for bounded WHATWG event-stream framing |
|
|
338
|
-
| Materializer extensions | Discovered `http-materializer` packages selected by `PLURNK_SCHEMES_HTTP_MATERIALIZER` {§http-materializer-extensions} |
|
|
339
|
-
|
|
340
|
-
### §http-config Operator configuration
|
|
341
|
-
|
|
342
|
-
| Concern | Contract |
|
|
343
|
-
| --------------------- | -------------------------------------------------------------------------------------------------------- |
|
|
344
|
-
| Canonical registry | Shipped `.env.defaults`; the daemon assembles it as a set-if-unset floor {§operator-config-env-defaults} |
|
|
345
|
-
| Required values | Missing or invalid required values fail at readiness/owning read; code carries no hidden fallback |
|
|
346
|
-
| Materializer selection | `PLURNK_SCHEMES_HTTP_MATERIALIZER=` names a discovered materializer id; unset = local projection only |
|
|
347
|
-
|
|
348
|
-
### §http-host-policy Operator web host policy
|
|
349
|
-
|
|
350
|
-
`PLURNK_SCHEMES_HTTP_HOSTS` confines every web acquisition to named hosts. Unset
|
|
351
|
-
or empty admits every host, as before. A JSON array admits exactly its members:
|
|
352
|
-
`"example.com"` names that host, `"*.example.com"` any subdomain of it, and `[]`
|
|
353
|
-
names none. A value that is not a JSON array of host names fails at first use.
|
|
354
|
-
Direct HTTP operations and WebSocket connections are refused with 403
|
|
355
|
-
`host-not-permitted` before any I/O, and a followed redirect that leaves the
|
|
356
|
-
admitted hosts is cancelled with the same refusal. Automatic acquisition checks
|
|
357
|
-
the target and every redirect hop, and a final URL outside the policy is the
|
|
358
|
-
ordinary unavailable `null`. The policy is operator configuration, alias-free and
|
|
359
|
-
not model teaching; an isolated benchmark sets `[]` as its network boundary and denies the
|
|
360
|
-
`web` trait ({§capability-admission}) as the model's.
|
|
361
|
-
|
|
362
|
-
### §automatic-fetch-check Automatic acquisition URL check
|
|
363
|
-
|
|
364
|
-
`WebFetcher` is the sole caller of `Guard.fetch`. Before automatic byte
|
|
365
|
-
acquisition, it accepts credential-free HTTP(S) targets only when every
|
|
366
|
-
resolved address is ordinary globally reachable unicast, and repeats the check
|
|
367
|
-
before every manually followed redirect. A refused or unresolvable target is
|
|
368
|
-
the ordinary unavailable `null` result. A transport failure after public
|
|
369
|
-
admission may still enter materializer page production.
|
|
370
|
-
|
|
371
|
-
Direct HTTP operations and WebSocket connections do not use this check. Loopback,
|
|
372
|
-
private, and link-local destinations are therefore valid explicit targets.
|
|
373
|
-
|
|
374
|
-
Redirect transitions follow WHATWG Fetch: 301/302 rewrite POST to GET; 303
|
|
375
|
-
rewrites methods other than GET/HEAD to GET; 307/308 preserve method and body.
|
|
376
|
-
A body rewrite removes body headers, cross-origin redirects remove
|
|
377
|
-
`Authorization`, and followed redirect bodies are cancelled. The configured
|
|
378
|
-
hop limit returns the last redirect rather than following beyond the limit.
|
|
379
|
-
Validation-time DNS answers are not pinned to connection-time resolution. This
|
|
380
|
-
check therefore makes no DNS-rebinding or total-egress claim.
|
|
381
|
-
|
|
382
|
-
## §materialization-lifecycle §6 Materialization lifecycle
|
|
383
|
-
|
|
384
|
-
<!-- map -->
|
|
385
|
-
| Tag | Promise |
|
|
386
|
-
| --- | --- |
|
|
387
|
-
| {§host-rewrite} | Acquisition target rewrite |
|
|
388
|
-
| {§revalidation} | GET representation freshness |
|
|
389
|
-
| {§http-kill} | KILL follows the entry rule; the remote DELETE is its own spelling. |
|
|
390
|
-
| {§sse} | Server-sent events |
|
|
391
|
-
<!-- /map -->
|
|
392
|
-
|
|
393
|
-
| Concern | Contract |
|
|
394
|
-
| -------------- | ---------------------------------------------------------------------------------------------------------- |
|
|
395
|
-
| Direct gate | Explicit targets use native origin transport; only a generic public request grants materializer authority |
|
|
396
|
-
| Automatic gate | Target and redirects require public admission; accepted generic HTML follows the same page producer |
|
|
397
|
-
| Readable owner | Origin Markdown needs no projection; HTML uses the eligible materializer or installed local reader, binary uses its mimetype handler |
|
|
398
|
-
| Source owner | `body` preserves origin text or bytes; it is never replaced by a derived projection |
|
|
399
|
-
| Projection | A present projection, including `""`, becomes `readable`; `null` alone means absence |
|
|
400
|
-
| Cancellation | One caller signal spans origin, auxiliary origin, projection, and materializer work |
|
|
401
|
-
|
|
402
|
-
### §host-rewrite Acquisition target rewrite
|
|
403
|
-
|
|
404
|
-
Only acquisition GETs are eligible for host rewriting. A GitHub
|
|
405
|
-
`…/blob/…` address uses the corresponding `raw.githubusercontent.com` source for
|
|
406
|
-
byte transport. Direct READ, exact FIND, and
|
|
407
|
-
WebFetcher prefetch share that rule. The originally addressed GitHub URL remains
|
|
408
|
-
entry identity. POST, PUT, and DELETE are never retargeted. Rewritten targets
|
|
409
|
-
are checked under {§automatic-fetch-check} only when `WebFetcher` acquires them.
|
|
410
|
-
|
|
411
|
-
### §revalidation GET representation freshness
|
|
412
|
-
|
|
413
|
-
Acquired responses append package-owned `x-plurnk-request-method`,
|
|
414
|
-
`x-plurnk-fetched-at`, and `x-plurnk-cache-variant` fields after origin headers.
|
|
415
|
-
Page bodies append `x-plurnk-materializer-id`; local projections append
|
|
416
|
-
`x-plurnk-projection-id`; The materializer appends its own evidence headers;
|
|
417
|
-
and bounded failure evidence. Only package metadata after the acquisition stamp
|
|
418
|
-
is authoritative.
|
|
419
|
-
Plurnk stores one representation per canonical URL rather than a variant set:
|
|
420
|
-
|
|
421
|
-
| Acquisition context | Package variant | Later cache use |
|
|
422
|
-
| -------------------------------------------- | --------------- | --------------- |
|
|
423
|
-
| No explicit request metadata; no `Vary` | `default` | Eligible |
|
|
424
|
-
| Any explicit request metadata | `bypass` | Ineligible |
|
|
425
|
-
| No explicit request metadata; any `Vary` | `bypass` | Ineligible |
|
|
426
|
-
|
|
427
|
-
This conservative selection avoids persisting request values or fabricating a
|
|
428
|
-
multi-variant store. Exact FIND passes request metadata through acquisition but
|
|
429
|
-
does not reuse that response later. A `304` that introduces `Vary` changes the
|
|
430
|
-
restored representation's package marker to `bypass`.
|
|
431
|
-
|
|
432
|
-
Only an eligible GET representation can supply a direct READ's body, TTL stamp,
|
|
433
|
-
conditional validators, or exact-FIND preparation. Completion comes from
|
|
434
|
-
channel lifecycle, not body length: `body` and `header` must be successful
|
|
435
|
-
(`static` or `closed`), and every channel must be terminal. An auxiliary channel
|
|
436
|
-
may be `errored`; `active` or unknown state is ineligible. Durable operation
|
|
437
|
-
evidence and HTTP reuse eligibility remain distinct: an acquired response stays
|
|
438
|
-
in the entry even when its origin policy prevents later cache use.
|
|
439
|
-
Workspace reuse follows shared-cache rules (RFC 9111); it does not change
|
|
440
|
-
retained evidence or workspace access. Reuse also requires a heuristically
|
|
441
|
-
cacheable origin status (RFC 9110 §15.1) or explicit cache permission (`public`,
|
|
442
|
-
`s-maxage`, `max-age`, or `Expires`).
|
|
443
|
-
Partial `206` responses remain evidence only: this cache does not combine or
|
|
444
|
-
select byte ranges. Ineligible responses supply neither cached content nor
|
|
445
|
-
validators to a later READ or exact FIND.
|
|
446
|
-
|
|
447
|
-
| Stored origin policy | Direct READ after acquisition | Exact FIND after acquisition |
|
|
448
|
-
| ------------------------------------- | ------------------------------------------------------ | ---------------------------------------- |
|
|
449
|
-
| `no-store` or `private` (qualified or unqualified) | Full acquisition without stored validators | Full acquisition |
|
|
450
|
-
| `no-cache` (qualified or unqualified) | Validate when a stored validator exists; else acquire | Full acquisition |
|
|
451
|
-
| Valid `s-maxage` | Serve only inside origin lifetime and operator ceiling | Reuse only while fresh under both limits |
|
|
452
|
-
| Valid `max-age`, without `s-maxage` | Same, using `max-age` | Same |
|
|
453
|
-
| Valid `Expires`, without either age directive | Same, using the origin expiration lifetime | Same |
|
|
454
|
-
| Invalid or ambiguous explicit expiry | Treat as stale; validate or acquire | Full acquisition |
|
|
455
|
-
| Eligible response without explicit lifetime | Use the operator TTL as Plurnk's heuristic | Reuse inside the operator TTL |
|
|
456
|
-
|
|
457
|
-
The operator ceiling is `PLURNK_SCHEMES_HTTP_TTL_MS`; `0` disables every
|
|
458
|
-
validation-free reuse. Origin age is the greater of the response's `Age` value
|
|
459
|
-
and apparent age from `Date`, plus residence since the authoritative package
|
|
460
|
-
stamp. A representation is fresh only while both origin lifetime and operator
|
|
461
|
-
ceiling permit it. `s-maxage` takes precedence even when invalid or duplicated:
|
|
462
|
-
such a value is stale, not a reason to fall back to `max-age` or `Expires`.
|
|
463
|
-
`private` and `no-store` prohibit reuse regardless of other permission directives.
|
|
464
|
-
Unknown cache extensions are inert. Stale content is never served, so
|
|
465
|
-
`must-revalidate` and `proxy-revalidate` require no separate path.
|
|
466
|
-
|
|
467
|
-
Outside the fresh window, a representation without a materializer identity may
|
|
468
|
-
send a stored ETag or Last-Modified only when that field is singular and
|
|
469
|
-
syntactically valid. A 304 can restore those stored channels only when its
|
|
470
|
-
validator corresponds to the nominated stored representation:
|
|
471
|
-
|
|
472
|
-
| 304 validator | Correspondence requirement |
|
|
473
|
-
| ----------------------------------------------------- | ------------------------------------------- |
|
|
474
|
-
| Strong ETag | Same stored strong ETag |
|
|
475
|
-
| Weak ETag | Same opaque tag under weak comparison |
|
|
476
|
-
| No ETag; Last-Modified | Same valid stored Last-Modified instant |
|
|
477
|
-
| Missing, malformed, or non-corresponding validator | Do not update the stored representation |
|
|
478
|
-
|
|
479
|
-
Response ETags use RFC 9111 §4.3.4 cache-update correspondence, not the origin's
|
|
480
|
-
`If-None-Match` precondition comparison. A strong response tag cannot promote a
|
|
481
|
-
stored weak tag into byte identity. A non-corresponding 304 triggers one GET
|
|
482
|
-
without conditional headers; a complete response is acquired normally. A
|
|
483
|
-
second 304, or a 304 without a reusable stored representation, returns `502`
|
|
484
|
-
(`fetch-failed`) without serving or replacing the stored body. No validators
|
|
485
|
-
are invented for an unsolicited 304.
|
|
486
|
-
|
|
487
|
-
A corresponding 304 restores the stored channels without rematerializing and
|
|
488
|
-
updates the header under the following ownership rule; any other response
|
|
489
|
-
replaces the channels:
|
|
490
|
-
|
|
491
|
-
| 304 metadata class | Stored-header action |
|
|
492
|
-
| --------------------------------------------------------------------- | ------------------------------------------------------- |
|
|
493
|
-
| Present end-to-end origin fields, including cache and validators | Replace prior fields of the same case-insensitive name |
|
|
494
|
-
| Origin fields absent from the 304 | Preserve |
|
|
495
|
-
| `Content-Type`, `Content-Encoding`, `Content-Range`, `Content-Length` | Preserve metadata describing the already-processed body |
|
|
496
|
-
| Package method, acquisition stamp, and variant | Rebuild authoritatively; refresh stamp and variant |
|
|
497
|
-
| Projection evidence | Preserve |
|
|
498
|
-
|
|
499
|
-
A 304-provided `Vary`, `private`, `no-store`, `no-cache`, expiry, or validator
|
|
500
|
-
therefore governs the next operation without relabeling derived Unicode as a
|
|
501
|
-
different source representation.
|
|
502
|
-
|
|
503
|
-
A stored derived representation requires the current projection identity. A
|
|
504
|
-
stored page body also requires its current route: negotiated origin Markdown,
|
|
505
|
-
local projection while no materializer is selected, metadata-ineligible local
|
|
506
|
-
projection, the configured materializer id, or the corresponding recoverable
|
|
507
|
-
local-fallback route. Selecting or changing the materializer invalidates the
|
|
508
|
-
affected route.
|
|
509
|
-
|
|
510
|
-
Once a page representation leaves its fresh window, Plurnk performs complete
|
|
511
|
-
reacquisition without old origin validators. Origin `304` can certify origin
|
|
512
|
-
bytes, but it cannot certify a composite that may also contain a fresh
|
|
513
|
-
materializer extraction or a newly negotiated Markdown representation.
|
|
514
|
-
Projection or materializer mismatch likewise invalidates
|
|
515
|
-
body, TTL, and validators together.
|
|
516
|
-
|
|
517
|
-
POST, PUT, and DELETE responses retain their method marker but cannot satisfy a
|
|
518
|
-
later GET or exact-FIND acquisition. An unmarked authored entry and an eligible
|
|
519
|
-
stored GET remain visible to universal FIND as durable evidence; exact HTTP
|
|
520
|
-
preparation applies the policy above. A metadata-less `KILL` deletes the stored entry ({§http-kill}).
|
|
44
|
+
| Response / surface | `body` | Auxiliary materialization (`readable` / `header`) | Completion |
|
|
45
|
+
|---|---|---|---|
|
|
46
|
+
| Negotiated origin `text/markdown` | Exact origin Markdown | None (single request; no `readable` needed) | Static representation |
|
|
47
|
+
| HTML page production | Exact server HTML when origin supplied it | Materializer Markdown, local Markdown (`200` or `203` fallback), or independent error in `readable` | Static representation; channels carry independent `producerResult`s |
|
|
48
|
+
| `text/event-stream` (SSE) | One joined `data` value + `\n` per `text/plain` chunk | Initial response metadata in `header` | `102`; origin close settles subscription |
|
|
49
|
+
| Empty response body | Present empty text `""` | Response and package metadata in `header` | Static representation (`READ` of `body` returns `204`) |
|
|
50
|
+
| Origin HTTP `4xx`/`5xx` | Available origin text or bytes preserved | Origin-backed channels carry durable `http-response-status` Problem | Selected channel's durable outcome |
|
|
51
|
+
| Configured textual / JSON type | Fetch `Response.text()` UTF-8 (streamed mutations use incremental `TextDecoder`; finite JSON formats via {§contract-representations}) | Response and package metadata in `header` (`charset` preserved as evidence) | Static representation |
|
|
52
|
+
| Configured binary type | Complete original bytes via {§mimetype-binary-input} (native attachment eligible on `READ` of `body`, `#bytes`, or `#readable`, {§log-wire-format}) | Derived Unicode in `readable` (when reader exists); `x-plurnk-projection-id` in `header` | Static representation, or `413 projection-input-limit` if over byte ceiling |
|
|
53
|
+
| Unreadable binary or unknown bytes | Complete original bytes (`application/octet-stream`) | No `readable` projection or native media claim | Static representation; ordinary hex `READ` |
|
|
54
|
+
|
|
55
|
+
`WebFetcher.materialize` outcomes across input and materializer states:
|
|
56
|
+
|
|
57
|
+
| Input or event | Action | Result |
|
|
58
|
+
|---|---|---|
|
|
59
|
+
| Configured non-HTML text | Decode with Fetch UTF-8 | Original representation in `body` |
|
|
60
|
+
| Configured binary with / without reader | Retain bounded bytes (and apply projection when present) | Original `body`, optional `readable`, and `projectionIdentity` (or `WebMaterializationError` from `ProjectionInputLimitError` when over limit) |
|
|
61
|
+
| Negotiated origin `text/markdown` | Accept without materializer | Exact Markdown in `body`; no `readable`, no second request |
|
|
62
|
+
| Eligible HTML with materializer selected | Run materializer `eligible(url)` + `extract` | Origin HTML in `body`; materializer Markdown in `readable` |
|
|
63
|
+
| Ineligible HTML or no materializer selected | Run installed HTML projection | Origin HTML in `body`; local Markdown in `readable` |
|
|
64
|
+
| Recoverable materializer failure with HTML (`502 materializer-recoverable`) | Fall back to installed HTML projection | Source in `body`; local Markdown in `readable` with durable status `203` |
|
|
65
|
+
| Hard materializer failure | Preserve evidence; no local recovery | `readable` errored; independently successful `body`/`header` survive |
|
|
66
|
+
| Materializer success after origin loss | Preserve provider text and origin failure | `readable` static; `body` errored with origin's own failure |
|
|
67
|
+
| Projection implementation throws | Stop and preserve cause | `WebMaterializationError` with stage `projection` |
|
|
68
|
+
|
|
69
|
+
- **Materializer extensions (`PLURNK_SCHEMES_HTTP_MATERIALIZER`):** `Http` publishes `WebFetcher` as `webMaterializer` ({§scheme-operation-dispatch}). When no `Accept` header is authored, `GET` sends `Accept: text/markdown, text/html;q=0.9, */*;q=0.1`. For a credential-free, metadata-free generic public request, `PLURNK_SCHEMES_HTTP_MATERIALIZER` selects a discovered `http-materializer` package (`plurnk: { kind: "http-materializer", materializers: [{ id, module }] }` or `plugin.json#extensions.ai.plurnk`, {§extension-discovery}). If `eligible(url)` returns an identity string, `extract` returns `success`, `recoverable` (`502 materializer-recoverable`: `The materializer could not extract <url> (<reason>); the failure is retryable.`), or `hard`.
|
|
70
|
+
- **Origin `llms.txt` companions (`PLURNK_SCHEMES_HTTP_LLMS_TXT`):** When `PLURNK_SCHEMES_HTTP_LLMS_TXT=1` (default), a successful generic `GET` opportunistically probes `<origin>/llms.txt` once per origin TTL window and materializes `https://<host>/llms.txt` as a discoverable workspace entry; missing/non-`2xx`/binary/failed probes are remembered quietly and never fail the initiating `READ`.
|
|
71
|
+
|
|
72
|
+
## §http-status Status mapping and replay
|
|
73
|
+
|
|
74
|
+
| Outcome | Operation status |
|
|
75
|
+
|---|---|
|
|
76
|
+
| Finite origin `2xx`/`3xx` exact `READ` / empty text response | `200` / `204` (origin status preserved in `header`) |
|
|
77
|
+
| Exact `FIND` after preparation / `KILL` with no live acquisition | Universal query result / exact entry-delete result |
|
|
78
|
+
| Automatic `WebFetcher` returns `null` / missing response channel | `404 not-materialized` / `404 channel-not-found` |
|
|
79
|
+
| Selected origin-backed channel received HTTP `4xx`/`5xx` | Origin status with durable `http-response-status` Problem |
|
|
80
|
+
| Final status outside `200..599` | `502 invalid-response-status` with `originStatus` |
|
|
81
|
+
| Recoverable materializer fallback / absent local HTML projection | `203` on `#readable` / `422 no-readable-projection` |
|
|
82
|
+
| Binary input exceeds byte bound | `413 projection-input-limit` |
|
|
83
|
+
| `KILL` of live acquisition / client-cancelled finite acquisition | `200` (owner settles `499`) / `499 cancelled` |
|
|
84
|
+
| `SSE` after acquisition: normal / cancelled / parser or transfer failure | `102` initial; settles channel result, `499`, or `502` |
|
|
85
|
+
| Multi-statement HTTP edit batch / invalid target, line edit, or userinfo | `409 non-atomic-edit-batch` / `400` |
|
|
86
|
+
| Non-corresponding `304` or transport failure without materializer output | `502 fetch-failed` (detail bounded by `PLURNK_SCHEMES_HTTP_ERROR_DETAIL_LIMIT`) |
|
|
87
|
+
| Projection exception / uninterpreted `SEND` status | `500 projection-failed` / `501 send-status-unsupported` |
|
|
88
|
+
|
|
89
|
+
- **Replay:** Failed `GET`, `PUT`, and `DELETE` acquisitions are retryable (idempotent); `POST` and post-open `SSE` stream failures are non-retryable.
|
|
90
|
+
|
|
91
|
+
## §http-host-policy Host policy and URL guard
|
|
92
|
+
|
|
93
|
+
| Surface | Contract |
|
|
94
|
+
|---|---|
|
|
95
|
+
| Configuration floor | `plurnk-schemes-http/.env.defaults` assembled via {§operator-config-precedence}; missing or invalid required values fail at readiness or owning read. |
|
|
96
|
+
| `PLURNK_SCHEMES_HTTP_HOSTS` | Unset or empty admits all hosts. A JSON array admits exact hostnames (`"example.com"`), wildcard subdomains (`"*.example.com"`), or none (`[]`). Disallowed direct `HTTP`/`WS` targets and redirect hops fail `403 host-not-permitted` before I/O; disallowed automatic acquisitions return `null` ({§capability-admission}). |
|
|
97
|
+
| `Guard.fetch` (automatic acquisition) | Solely used by `WebFetcher` for automatic byte acquisition: requires credential-free `http(s):` targets whose resolved DNS addresses are all globally reachable unicast, re-checked on every redirect hop (`301`/`302` rewrite `POST` $\to$ `GET`; `303` rewrites non-`GET`/`HEAD` $\to$ `GET`; `307`/`308` preserve method/body; cross-origin hops strip `Authorization`). Direct `HTTP` and `WebSocket` operations do not run `Guard.fetch` and permit loopback/private/link-local targets. |
|
|
521
98
|
|
|
522
|
-
§
|
|
99
|
+
## §revalidation Revalidation, target rewrite, and SSE
|
|
523
100
|
|
|
524
|
-
|
|
101
|
+
- **GitHub raw rewrite (`host-rewrite`):** Acquisition `GET`s (direct `READ`, exact `FIND`, `WebFetcher`) rewrite `https://github.com/<owner>/<repo>/blob/<ref>/<path>` to `raw.githubusercontent.com` for byte transport while preserving the addressed GitHub URL as entry identity. `POST`, `PUT`, and `DELETE` are never retargeted.
|
|
102
|
+
- **Cache variants and freshness (`PLURNK_SCHEMES_HTTP_TTL_MS`):** Acquired responses append authoritative `x-plurnk-request-method`, `x-plurnk-fetched-at`, `x-plurnk-cache-variant` (`default` when no explicit request metadata and no `Vary`, else `bypass`), `x-plurnk-materializer-id`, and `x-plurnk-projection-id`. Only terminal `default`-variant `GET` responses with successful `body` and `header` and cacheable status/directives (excluding `no-store`, `private`, and `206`) are eligible for reuse:
|
|
525
103
|
|
|
526
|
-
|
|
527
|
-
|
|
528
|
-
|
|
529
|
-
|
|
530
|
-
`
|
|
531
|
-
|
|
532
|
-
|
|
533
|
-
handler does not reconnect; events accumulate until the origin closes or the
|
|
534
|
-
operation is cancelled.
|
|
104
|
+
| Stored origin policy | Direct `READ` after acquisition | Exact `FIND` after acquisition |
|
|
105
|
+
|---|---|---|
|
|
106
|
+
| `no-store` or `private` (qualified or unqualified) | Full acquisition without stored validators | Full acquisition |
|
|
107
|
+
| `no-cache` (qualified or unqualified) | Validate when a stored validator exists; else acquire | Full acquisition |
|
|
108
|
+
| Valid `s-maxage` > `max-age` > `Expires` | Serve while fresh under both origin lifetime and `PLURNK_SCHEMES_HTTP_TTL_MS` (`s-maxage` overrides `max-age`/`Expires` even if invalid) | Reuse while fresh under both limits |
|
|
109
|
+
| Invalid or ambiguous explicit expiry | Treat as stale; validate or acquire | Full acquisition |
|
|
110
|
+
| Eligible response without explicit lifetime | Heuristic reuse within `PLURNK_SCHEMES_HTTP_TTL_MS` (`0` disables validation-free reuse) | Reuse within `PLURNK_SCHEMES_HTTP_TTL_MS` |
|
|
535
111
|
|
|
536
|
-
|
|
112
|
+
- **Conditional `304` revalidation:** Outside the fresh window, a non-materializer representation with a singular valid `ETag` or `Last-Modified` sends conditional headers. A `304` restores stored channels only when its validator corresponds per RFC 9111 §4.3.4:
|
|
537
113
|
|
|
538
|
-
|
|
|
539
|
-
|
|
540
|
-
|
|
|
541
|
-
|
|
|
542
|
-
|
|
|
543
|
-
| Admitted origin transport failure | Bounded origin failure plus eligibility for provider-only page production |
|
|
544
|
-
| Materialized configured text | Complete Fetch-decoded UTF-8 body |
|
|
545
|
-
| Materialized binary | Original bounded `body` bytes, optional `readable`, and projection identity in `header` |
|
|
546
|
-
| Materialized HTML page | Independent source/readable outcomes and complete route/provider evidence |
|
|
547
|
-
| Automatic top-level `null` | Refused target, non-2xx response, or unavailable response with no provider route |
|
|
548
|
-
| Caller-cancelled acquisition | Rejects with the caller signal's exact reason |
|
|
114
|
+
| `304` validator / metadata class | Correspondence & stored-header action |
|
|
115
|
+
|---|---|
|
|
116
|
+
| Strong `ETag` / Weak `ETag` / `Last-Modified` (no `ETag`) | Must match stored strong `ETag`, weak `ETag` (weak comparison; strong `304` tag cannot promote stored weak tag), or exact `Last-Modified` instant; non-corresponding `304` triggers one unconditional `GET`, and a second or unsolicited `304` fails `502 fetch-failed`. |
|
|
117
|
+
| Present end-to-end origin fields vs absent origin fields | Present fields replace prior case-insensitive matches; absent origin fields are preserved (`Vary` on `304` downgrades variant to `bypass`). |
|
|
118
|
+
| `Content-Type`, `Content-Encoding`, `Content-Range`, `Content-Length`, projection evidence | Preserved from the stored representation; package method, acquisition stamp, and cache variant are rebuilt authoritatively. |
|
|
549
119
|
|
|
550
|
-
|
|
551
|
-
failure. Caller cancellation is not liveness: a pre-aborted caller fails before
|
|
552
|
-
acquisition and a caller abort wins with its exact reason. Package-owned origin
|
|
553
|
-
timeouts are ordinary classified acquisition outcomes; materializer timeouts
|
|
554
|
-
are the materializer's own classified outcomes. WebFetcher
|
|
555
|
-
owns no entry identity, registry selection, selected-channel policy, or query
|
|
556
|
-
policy; its consumers supply those boundaries and the projection capability.
|
|
120
|
+
- **Server-sent events (`sse`):** Direct `GET` with `text/event-stream` persists `header`, returns `102`, and streams each event's joined `data` + `\n` into `body` (`text/plain`) via `eventsource-parser` and the retained `StreamSubscription` (no reconnect; buffer/transport failure settles `502`, cancellation settles `499`).
|
|
557
121
|
|
|
558
|
-
## §
|
|
122
|
+
## §prefetch WebFetcher
|
|
559
123
|
|
|
560
|
-
|
|
561
|
-
| Tag | Promise |
|
|
562
|
-
| --- | --- |
|
|
563
|
-
| {§ws-lifecycle} | Socket ownership and settlement |
|
|
564
|
-
| {§problems-http} | Web materialization Problems. |
|
|
565
|
-
| {§pinned-wording-http} | Pinned wording. |
|
|
566
|
-
<!-- /map -->
|
|
124
|
+
`WebFetcher` implements `webMaterializer` ({§scheme-operation-dispatch}) over `Guard.fetch`, `HostPolicy`, and `MaterializerRegistry`:
|
|
567
125
|
|
|
568
|
-
|
|
569
|
-
|
|
570
|
-
|
|
571
|
-
|
|
572
|
-
|
|
573
|
-
|
|
126
|
+
| Result | Meaning |
|
|
127
|
+
|---|---|
|
|
128
|
+
| Origin HTML / Negotiated origin Markdown | Exact source text (negotiated Markdown skips a second HTML request), MIME type, package-stamped headers, and page-producer eligibility |
|
|
129
|
+
| Other accepted body | Unconsumed byte stream, MIME type, package-stamped headers, and cancellation owner |
|
|
130
|
+
| Admitted origin transport failure | Bounded origin failure plus eligibility for materializer-only page production |
|
|
131
|
+
| `materialize`: text / binary / HTML page | Complete UTF-8 `body`; bounded `body` bytes + optional `readable` + `projectionIdentity`; or independent HTML `body`/`readable` outcomes |
|
|
132
|
+
| Automatic top-level `null` vs caller cancellation | Refused target, non-`2xx` automatic response, or unavailable target with no materializer returns `null`; caller `AbortSignal` aborts reject with the caller's exact reason |
|
|
574
133
|
|
|
575
|
-
|
|
134
|
+
## §ws-lifecycle WebSocket scheme and lifecycle
|
|
576
135
|
|
|
577
136
|
```mermaid
|
|
578
137
|
stateDiagram-v2
|
|
579
138
|
[*] --> Idle
|
|
580
|
-
Idle --> Claimed:
|
|
581
|
-
Claimed --> Connecting: seed entry, open
|
|
139
|
+
Idle --> Claimed: READ claims workspace + canonical URL
|
|
140
|
+
Claimed --> Connecting: seed entry, open StreamSubscription, construct WebSocket
|
|
582
141
|
Claimed --> Idle: setup or construction failure
|
|
583
|
-
Connecting --> Open: native open
|
|
584
|
-
Connecting --> Settling: pre-open close/error, KILL, cancel,
|
|
585
|
-
Open --> Open: ordered inbound frame, EDIT, or SEND 200
|
|
586
|
-
Open --> Settling:
|
|
587
|
-
Settling --> Idle:
|
|
142
|
+
Connecting --> Open: native open -> messages active -> READ returns 102
|
|
143
|
+
Connecting --> Settling: pre-open close/error (502 connection-failed), KILL, cancel, or shutdown
|
|
144
|
+
Open --> Open: ordered inbound text frame, single EDIT, or SEND 200
|
|
145
|
+
Open --> Settling: close/error, KILL, cancel, binary frame (415 + close 4003), persistence failure (500), or shutdown
|
|
146
|
+
Settling --> Idle: drain accepted frame prefix, close transport & subscription, release claim
|
|
588
147
|
```
|
|
589
148
|
|
|
590
|
-
|
|
591
|
-
| --------------------------------- | ---------------------------------------------------------------------------------------------------------------- |
|
|
592
|
-
| ```` ```READ (ws(s)://…) ```` | Claim, seed/subscribe, construct `CONNECTING`, then return `102` after native `open` plus durable activation |
|
|
593
|
-
| Concurrent duplicate READ | Join the pending acquisition; once acquired, observe the shared connection with `200` and its connection state. Never create a second subscription or socket. |
|
|
594
|
-
| ```` ```EDIT (ws(s)://…) ```` | Send one whole text frame only for an open owner; line ranges and multi-statement batches are rejected |
|
|
595
|
-
| ```` ```SEND (ws(s)://…) ```` with body | Send only for owner `open` plus native `readyState=OPEN`; absent or non-open owner is `409`; send throw is `502` |
|
|
596
|
-
| ```` ```KILL (ws(s)://…) ```` | Close/cancel the claimed owner; no owner is `404`; an attempted close throw is `502` |
|
|
597
|
-
| Inbound frame after native `open` | Join the owner's persistence chain; the next write begins only after the preceding write succeeds |
|
|
598
|
-
| Binary frame after native `open` | Retain the text prefix, prune the binary and later frames, settle `415 binary-frame-unsupported`, close with private-use code `4003` |
|
|
599
|
-
| First inbound persistence failure | Retain the successful prefix, prune queued and later frames, and settle with `500 message-persistence-failed` |
|
|
600
|
-
| Socket closes before `open` | Close subscription with `502 connection-failed`; the pending READ returns that exact failure |
|
|
601
|
-
| Socket closes after `open` | Drain the accepted frame prefix; initial READ remains `102`; settle with `200` unless a drained write fails |
|
|
602
|
-
| Failure after `open` | Initial READ remains `102`; persist the exact terminal failure and wake through subscription settlement |
|
|
603
|
-
|
|
604
|
-
Markerless single-statement EDIT and SEND with signal `200` converge on one
|
|
605
|
-
outbound text-frame path and have the same connection and transport outcomes.
|
|
606
|
-
Authored dispatch order ({§op-execution-order}) lets either write follow the
|
|
607
|
-
opening READ in the same turn, after acquisition establishes the live owner.
|
|
608
|
-
|
|
609
|
-
The in-instance registry is keyed by workspace and complete canonical network
|
|
610
|
-
URL, matching the commons entry. Workers share the connection and may SEND or
|
|
611
|
-
KILL it; acquisition attribution and settlement remain with its initiating
|
|
612
|
-
operation ({§runtime-resource-binding}). The claim remains registered through terminal
|
|
613
|
-
cleanup, so a new READ cannot overlap an owner's subscription settlement. Every
|
|
614
|
-
terminal path drains retained owner work, closes the transport when necessary,
|
|
615
|
-
closes the durable subscription, then releases the claim. A persistence failure
|
|
616
|
-
found while draining supersedes a graceful terminal result; cancellation and
|
|
617
|
-
transport failures retain their exact result. Handler shutdown requests
|
|
618
|
-
settlement for every remainder, awaits every owner, and aggregates
|
|
619
|
-
transport-close failures under {§handler-lifecycle}.
|
|
620
|
-
|
|
621
|
-
| Transport limit | Current contract |
|
|
622
|
-
| ------------------ | ------------------------------------------------------------------------------------------------------- |
|
|
623
|
-
| Payload projection | String event data only into `text/plain`; binary data terminates with Plurnk `415` and private-use WebSocket code `4003` |
|
|
624
|
-
| Reconnection | None; READ again after terminal cleanup |
|
|
625
|
-
| Handshake metadata | `[metadata]` is unsupported; the default global-WebSocket identity is used |
|
|
626
|
-
| Runtime | Node ≥26 |
|
|
627
|
-
|
|
628
|
-
§problems-http **Web materialization Problems.** Every code minted here, its status, and the sentence that is its contract (placeholders in *italics* are filled at emission; a fixed recovery follows its detail).
|
|
629
|
-
|
|
630
|
-
| code | status | contract |
|
|
631
|
-
|---|---:|---|
|
|
632
|
-
| `materializer-recoverable` | 502 | The materializer could not extract *url* (*reason*); the failure is retryable. |
|
|
633
|
-
|
|
634
|
-
§pinned-wording-http **Pinned wording.** Verbatim sentences tests pin: each is contract, and a change here is a change of contract.
|
|
149
|
+
`Ws` registers as `wss` (`ws` routes to it) with default `text/plain` channel `messages` and canonical `NetworkAddress` identity (no `[metadata]` support; Node $\ge 26$):
|
|
635
150
|
|
|
636
|
-
|
|
|
151
|
+
| Operation / event | Contract |
|
|
637
152
|
|---|---|
|
|
638
|
-
|
|
|
639
|
-
| READ the WebSocket URL before sending a message
|
|
640
|
-
|
|
|
153
|
+
| `READ (ws(s)://...)` | Claims workspace/URL coordinate, seeds entry and `StreamSubscription`, opens native `WebSocket`, and returns `102` on `open`. Concurrent duplicate `READ`s join the pending open and return `200` with connection state without opening a second socket. |
|
|
154
|
+
| `EDIT (ws(s)://...)` / `SEND (ws(s)://...)` | Single-statement markerless `EDIT` and `SEND` (`200`) propose (`202`, `effect: "host"`) and send one text frame when the owner is `open` (`readyState === OPEN`). Absent/non-open owner fails `409` (`READ the WebSocket URL before sending a message.`); send throw fails `502` (`The WebSocket message could not be sent.`); line ranges and multi-statement batches fail `400`/`409`. |
|
|
155
|
+
| `KILL (ws(s)://...)` | Closes/cancels the claimed owner (`404` when no live owner exists; close throw fails `502`: `The WebSocket connection could not be closed.`). |
|
|
156
|
+
| Inbound frames & settlement | Text frames chain sequentially onto `messages` after `open`. A binary frame retains the text prefix, prunes later frames, closes with WebSocket code `4003`, and settles `415 binary-frame-unsupported`. First persistence failure prunes later frames and settles `500 message-persistence-failed`. Pre-open close settles `502 connection-failed`; post-open close drains accepted frames and settles `200` ({§execution-output-identity}, {§scheme-operation-dispatch}). |
|