@twinsearth/w2m-dsh-plugin 1.0.5 → 1.0.7

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -2,6 +2,121 @@
2
2
 
3
3
  All notable changes to this project are documented here.
4
4
 
5
+ ## [1.0.7] — 2026-10-17
6
+
7
+ **The fleet sync channel, both directions: a machine can publish its collective memory to the relay and another
8
+ machine in the same fleet can fetch it back by hash.**
9
+
10
+ ### #1 is now a channel, not a backup
11
+
12
+ v1.0.6 shipped the storage layer and said plainly that it had no route and no caller. This release supplies both:
13
+
14
+ | Piece | Where |
15
+ |---|---|
16
+ | content-addressed store | `src/relay/persistence.mjs` — `putBlob`/`getBlob`, 8 MiB per blob, 256 per fleet, oldest-first eviction that never takes the last one |
17
+ | hash → fleet ownership | `<state>/blobs/index.json`, so a content-addressed store is *not* a shared one |
18
+ | two routes | `PUT`/`GET /v1/fleet/blobs/<sha256>` in `src/relay/server.mjs`, behind the same device-token gate as every other route |
19
+ | scoping | `state.scopeBlobFleet` reuses `scopeFleetId` — **the same rule task dispatch obeys**, not a second copy of it |
20
+ | caller | `w2m_memory` `action: "publish"` and `"pull"` |
21
+
22
+ **Why a content-addressed store does not contradict "the relay keeps only hashes".** It persists `stdout_sha256`
23
+ and never the text, and `src/agent/storage.mjs` records why: each machine's `memory.jsonl` is the only durable copy
24
+ of what a command printed. **That invariant is about the relay not interpreting content.** A blob keyed by its own
25
+ hash preserves it exactly — the relay still interprets nothing, and **the hash is what says the bytes are the
26
+ bytes**, as `stdout_sha256` does for a result. What is new is only that a hash the relay already holds can now have
27
+ its bytes fetched from the same place.
28
+
29
+ ### The decisions worth recording
30
+
31
+ **The write verifies the key, and refuses rather than correcting.** A content-addressed store that accepts bytes
32
+ under a key they do not hash to is not content-addressed: the key stops being a claim, and every later reader's
33
+ verification becomes decorative. A mismatch is `422 BLOB_HASH_MISMATCH` and nothing is stored.
34
+
35
+ **Ownership is first-writer-wins.** A second fleet PUT-ing an identical blob does **not** take ownership of it.
36
+ Otherwise any fleet could claim another's blob by replaying the same bytes, which is the same failure as having no
37
+ index at all.
38
+
39
+ **A cross-fleet read is `404`, not `403`.** A distinguishable "not yours" tells a prober that this hash exists on
40
+ this relay. The reader cannot tell "never stored" from "another fleet's", deliberately.
41
+
42
+ **The pull side re-hashes what it receives.** A verification performed only by the party that could be wrong is not
43
+ a verification, so a reader that gets bytes hashing to something else raises `W2M_BAD_BLOB` rather than passing
44
+ them on.
45
+
46
+ ### What is verified, and what is not
47
+
48
+ **Verified — executed:** four new tests in `test/relay.test.mjs` drive the real routes against a real relay:
49
+ store-and-return-unchanged; a mismatched key is refused (422) and leaves the key absent (404); one fleet cannot read
50
+ another's blob (404); an unauthenticated request is refused (401). Full suite **2356 tests, 2342 pass, 0 fail, 14
51
+ skip**; pre-flight **GREEN, 15 of 15 checks**.
52
+
53
+ **NOT verified: the `publish`/`pull` tool actions end to end.** They are type-correct and lint-clean, and their
54
+ failure paths are written, but **no test executes them** — they need a profile with a `fleet-memory.md` and a live
55
+ relay. They are marked unverified here rather than counted as working, which is the distinction this project keeps
56
+ having to restate.
57
+
58
+ **A known limit of the reader:** `pull` returns the blob as text and re-hashes it as UTF-8. That is exact for a
59
+ `fleet-memory.md`; it would not be for arbitrary binary content, and the channel does not claim to carry that.
60
+
61
+ ### Not in this release
62
+
63
+ The trigger (#3) and merge semantics (#2) are deliberately absent. Publishing is a decision a machine makes, not
64
+ something that happens on join, and merge needs a conflict rule that has not been decided: two machines writing
65
+ different facts produce **two hashes**, and nothing yet says who wins. **Nothing is overwritten in the meantime —
66
+ that is the property #1 was built to provide.**
67
+
68
+ ## [1.0.6] — 2026-10-17
69
+
70
+ **DSH-GPW now ships *findable*: installed with the package, present in the plugin list, and off until an operator
71
+ turns it on.**
72
+
73
+ ### What v1.0.5 got wrong
74
+
75
+ It shipped the subpath export `@twinsearth/w2m-dsh-plugin/gpw` and no config row. **A plugin the composed tree does
76
+ not mention is a plugin the plugin manager cannot list** -- so "enable it manually" had **nothing to click**, and
77
+ the only way in was hand-editing a profile patch.
78
+
79
+ **An optional component nobody can find is not optional, it is absent.** That is the same distinction as the seven
80
+ written-but-unreachable modules in this repository, seen from the user's side instead of the code's.
81
+
82
+ ### The shape that fixes it
83
+
84
+ The package's own bundle patch now carries a second entry:
85
+
86
+ - id: dsh-gpw
87
+ name: '@twinsearth/w2m-dsh-plugin/gpw'
88
+ disabled: true
89
+ config:
90
+ maxPerWindow: 10
91
+ windowMs: 60000
92
+ minGapMs: 3000
93
+
94
+ **`disabled: true` is the load-bearing field.** The row exists, so the component appears; the toggle is off, so
95
+ nothing answers `approval/request`; and **flipping it in the UI is the whole enablement step** -- no file edit, no
96
+ restart of anything but the client.
97
+
98
+ **It stays in the same `insert:` list as the w2m row**, and that is still the correct form: this is an insert
99
+ because no earlier layer creates a `dsh-gpw` row. (A *bare* `- id:` patch aimed at a nonexistent row is skipped
100
+ silently -- the failure recorded in this file and in TROUBLESHOOTING 3.2.)
101
+
102
+ ### The distinction the two releases now make
103
+
104
+ | | v1.0.5 | v1.0.6 |
105
+ |---|---|---|
106
+ | Installs with the package | yes | yes |
107
+ | Appears in the plugin manager | **no** | **yes** |
108
+ | Enabled by default | no | no |
109
+ | How you turn it on | hand-edit a profile patch | **click its toggle** |
110
+
111
+ **Not verified: that the toggle actually flips it on and that it then intercepts.** That needs a live host. The
112
+ observable evidence is its own startup line followed by its counted line --
113
+
114
+ approval-ratelimit: active -- at most 10 auto-approvals per 60 s, at least 3000 ms apart, process-wide
115
+ approval-ratelimit: N auto-approved, M deferred to the human
116
+
117
+ -- **because without the second line, "mounted and working" and "mounted and doing nothing" are the same picture**,
118
+ which is the failure this watchdog's own counters exist to prevent.
119
+
5
120
  ## [1.0.5] — 2026-10-17
6
121
 
7
122
  **DSH-GPW ships as an optional sub-plugin, and the reason it never activated is fixed.**
package/cordis.patch.yml CHANGED
@@ -115,3 +115,23 @@
115
115
  allowedCommands: []
116
116
  p2pMode: auto
117
117
  stunServers: []
118
+ # DSH-GPW (Global Permission Watchdog) ships with this package and is **off until an operator turns it on**.
119
+ #
120
+ # It is in this insert list rather than left out of the tree entirely, because a plugin the composed tree does
121
+ # not mention is a plugin the plugin manager cannot list -- and then "enable it manually" has no control to
122
+ # click. **A row with `disabled: true` is the shape that makes it installable with the package and switchable
123
+ # from the UI**: the component shows up, its toggle is off, and flipping it needs no file edit.
124
+ #
125
+ # Found this way, not assumed: v1.0.5 shipped the subpath export but no row, so the watchdog was reachable only
126
+ # by hand-editing a profile patch. **An optional component nobody can find is not optional, it is absent.**
127
+ #
128
+ # What it does once enabled: answers `approval/request` with at most 10 auto-approvals per rolling 60 s, at
129
+ # least 3000 ms apart, and defers everything past that to the human prompt, where it waits.
130
+ - id: dsh-gpw
131
+ name: '@twinsearth/w2m-dsh-plugin/gpw'
132
+ disabled: true
133
+ config:
134
+ maxPerWindow: 10
135
+ windowMs: 60000
136
+ minGapMs: 3000
137
+
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@twinsearth/w2m-dsh-plugin",
3
- "version": "1.0.5",
3
+ "version": "1.0.7",
4
4
  "description": "Run one project across your own Windows and macOS machines from a single DeepSeek account, through 11 w2m_* tools. Four coordination modes (replicate, split, broadcast, pipeline) and six aggregate verdicts (consistent, divergent, divergent-platform, failed, partial, unverifiable). P2P hole punching by default, with a bundled relay, STUN responder and ledger fallback, so the fleet needs no third-party service. Joining needs a pairing code plus autoStartAgent: true; installing alone does not join.",
5
5
  "type": "module",
6
6
  "main": "lib/index.js",
package/plugin.json CHANGED
@@ -1,7 +1,7 @@
1
1
  {
2
2
  "$schema": "https://agent-plugins.org/schemas/1.0.0/plugin.schema.json",
3
3
  "name": "twinsearth-w2m-dsh",
4
- "version": "1.0.5",
4
+ "version": "1.0.7",
5
5
  "description": "Run one project across your own Windows and macOS machines from a single DeepSeek account, through 11 w2m_* tools. Four coordination modes (replicate, split, broadcast, pipeline) and six aggregate verdicts (consistent, divergent, divergent-platform, failed, partial, unverifiable). P2P hole punching by default, with a bundled relay, STUN responder and ledger fallback, so the fleet needs no third-party service. Joining needs a pairing code plus autoStartAgent: true; installing alone does not join.",
6
6
  "license": "MIT",
7
7
  "keywords": [
@@ -208,7 +208,7 @@ const MAX_STAGES = 16;
208
208
  * `package.json` during `scripts/pack.mjs` rather than hand-maintained, and `W2M_PLUGIN_VERSION`
209
209
  * overrides it at runtime for tests and packagers.
210
210
  */
211
- const PLUGIN_VERSION = '1.0.5';
211
+ const PLUGIN_VERSION = '1.0.7';
212
212
 
213
213
  /** How long any single HTTP call may take before it is aborted, in milliseconds. */
214
214
  const DEFAULT_REQUEST_TIMEOUT_MS = 15_000;
@@ -2590,10 +2590,27 @@ async function request(options) {
2590
2590
  pathname,
2591
2591
  query,
2592
2592
  body,
2593
+ /**
2594
+ * v1.0.7: raw bytes to send instead of a JSON body.
2595
+ *
2596
+ * Added for the blob channel (#1), which PUTs a memory document's exact octets. **Added here rather than as a
2597
+ * second `fetch` in the caller**, because `tools.mjs:1283` already states the rule: the direct path must not grow
2598
+ * a second way to talk to the relay -- bearer-token resolution, base-path joining, timeouts and abort handling
2599
+ * all live in this one function, and a duplicate would be a duplicate of all four.
2600
+ *
2601
+ * `body` and `rawBody` are mutually exclusive: passing both is a caller bug, not a merge to guess at.
2602
+ */
2603
+ rawBody,
2593
2604
  maxBytes = MAX_RESPONSE_BYTES,
2594
2605
  accept = 'application/json',
2595
2606
  } = options;
2596
2607
 
2608
+ if (rawBody !== undefined && body !== undefined) {
2609
+ throw new W2MError('W2M_CONFIG', 'W2M_CONFIG: `body` and `rawBody` are mutually exclusive', {
2610
+ hint: 'pass `rawBody` for octets, `body` for JSON -- not both',
2611
+ });
2612
+ }
2613
+
2597
2614
  throwIfAborted(signal);
2598
2615
 
2599
2616
  // Named here rather than left to `new URL`, whose failure would say only "Invalid URL" and send
@@ -2613,7 +2630,10 @@ async function request(options) {
2613
2630
 
2614
2631
  const headers = { accept };
2615
2632
  if (token) headers.authorization = `Bearer ${token}`;
2616
- if (body !== undefined) headers['content-type'] = 'application/json';
2633
+ // v1.0.7: octets keep their own content type. JSON-stringifying them would corrupt the bytes and, because the
2634
+ // relay verifies `sha256(body) === key`, corrupting them here would surface as a 422 blamed on the caller.
2635
+ if (rawBody !== undefined) headers['content-type'] = 'application/octet-stream';
2636
+ else if (body !== undefined) headers['content-type'] = 'application/json';
2617
2637
 
2618
2638
  const timeoutController = new AbortController();
2619
2639
  const timer = setTimeout(() => timeoutController.abort(new Error('timeout')), timeoutMs);
@@ -2624,7 +2644,7 @@ async function request(options) {
2624
2644
  response = await fetch(target, {
2625
2645
  method,
2626
2646
  headers,
2627
- body: body === undefined ? undefined : JSON.stringify(body),
2647
+ body: rawBody !== undefined ? rawBody : (body === undefined ? undefined : JSON.stringify(body)),
2628
2648
  signal: combined.signal,
2629
2649
  });
2630
2650
  } catch (error) {
@@ -5112,6 +5132,30 @@ export async function apply(ctx, config = {}) {
5112
5132
  'relay keeps only hashes, so these files are the only place a command\'s output survives. Use it ' +
5113
5133
  'before re-running work -- it is what turns a dozen rediscovery dispatches into one lookup.',
5114
5134
  parameters: {
5135
+ /**
5136
+ * v1.0.7: `read` (the default, and the only behaviour before this release), `publish`, or `pull`.
5137
+ *
5138
+ * Defaulting to `read` is deliberate: every existing caller says nothing about `action`, and a tool that
5139
+ * changed what "call `w2m_memory`" means would be a silent behaviour change on the one path that is supposed
5140
+ * to be safe to call blindly.
5141
+ */
5142
+ action: {
5143
+ type: 'string',
5144
+ enum: ['read', 'publish', 'pull'],
5145
+ description:
5146
+ 'read (default) = answer from memory. publish = upload the curated collective memory '
5147
+ + '(`fleet-memory.md`) to the relay as a content-addressed blob and return its SHA-256. pull = fetch one '
5148
+ + 'blob by its SHA-256 and return its text, re-hashing what came back against the name it was asked for. '
5149
+ + 'Neither publish nor pull modifies the local file.',
5150
+ default: 'read',
5151
+ },
5152
+ hash: {
5153
+ type: 'string',
5154
+ description:
5155
+ 'The SHA-256 naming the blob, for action="pull": 64 lowercase hex characters. A hash that exists on the '
5156
+ + 'relay but belongs to another fleet is reported as not found, exactly like a hash that was never '
5157
+ + 'stored -- a distinguishable refusal would confirm to a prober which hashes this relay holds.',
5158
+ },
5115
5159
  scope: {
5116
5160
  type: 'string',
5117
5161
  enum: ['local', 'fleet'],
@@ -5199,6 +5243,154 @@ export async function apply(ctx, config = {}) {
5199
5243
  output: jsonOutput,
5200
5244
  async execute(args, exec) {
5201
5245
  throwIfAborted(exec?.signal);
5246
+
5247
+ /**
5248
+ * v1.0.7 (#1c): publish the collective memory as a content-addressed blob.
5249
+ *
5250
+ * ## Why this exists at all
5251
+ *
5252
+ * The blob channel (1a/1b) is a route and a store; **without a caller it is the same defect as the seven
5253
+ * written-but-unreachable modules this repository keeps an accepted-orphans list for.** This is the caller:
5254
+ * a machine that wants its curated memory to be fetchable by its own fleet says so, and gets back the hash
5255
+ * that names it.
5256
+ *
5257
+ * ## Why it reads the file and does not rewrite it
5258
+ *
5259
+ * Publishing is a **copy, not a move**. If it deleted or normalised the file, the only durable copy of what
5260
+ * this machine knows would depend on the relay having accepted it -- and the relay is explicitly not the
5261
+ * durable store (`storage.mjs`: each machine's `memory.jsonl` is the only copy of what a command printed).
5262
+ *
5263
+ * ## Why a missing file is a named refusal
5264
+ *
5265
+ * `putBlob` rejects an empty body with `BLOB_EMPTY`, which is correct but arrives too late to say anything
5266
+ * useful. A machine with no `fleet-memory.md` has not published anything yet -- that is a state, not a
5267
+ * protocol violation, and the message should say which file is missing and where it was looked for.
5268
+ */
5269
+ if (typeof args?.action === 'string' && args.action === 'publish') {
5270
+ const file = collectivePath();
5271
+ if (file === null) {
5272
+ throw new W2MError('W2M_CONFIG', 'W2M_CONFIG: there is no profile to publish collective memory from', {
5273
+ hint: 'the collective layer lives beside the profile that mounted this plugin; set `stateDir` or run '
5274
+ + 'with a profile',
5275
+ });
5276
+ }
5277
+ let bytes;
5278
+ try {
5279
+ // `fs` is `promises as fs` in this file (line 93), so this is the promise API -- a bare `readFile` is not
5280
+ // in scope and would throw ReferenceError at the exact moment the caller wanted to publish.
5281
+ bytes = await fs.readFile(file);
5282
+ } catch {
5283
+ // The error object is deliberately not bound: every refusal of a read here is the same refusal (there is
5284
+ // no document at that path), and naming the errno would tell the caller to debug a filesystem this plugin
5285
+ // does not own.
5286
+ throw new W2MError(
5287
+ 'W2M_NOT_FOUND',
5288
+ `W2M_NOT_FOUND: no collective memory to publish at ${file}`,
5289
+ { hint: 'write measured facts to it first; publishing copies the file, it does not create one' },
5290
+ );
5291
+ }
5292
+ const hash = createHash('sha256').update(bytes).digest('hex');
5293
+ const base = requireRabbit(cfg);
5294
+ const token = await resolveToken(cfg);
5295
+ const result = await request({
5296
+ rabbitUrl: base,
5297
+ token,
5298
+ method: 'PUT',
5299
+ pathname: `/v1/fleet/blobs/${hash}`,
5300
+ rawBody: bytes,
5301
+ signal: exec?.signal,
5302
+ });
5303
+ return JSON.stringify({
5304
+ ok: true,
5305
+ published: true,
5306
+ hash,
5307
+ bytes: bytes.length,
5308
+ file,
5309
+ relay: {
5310
+ http_status: result.status,
5311
+ // `request()` returns `{status, ok, json, text, error}` -- the parsed body is `json`, not `body`.
5312
+ evicted: result.json?.evicted ?? [],
5313
+ },
5314
+ next: 'another machine in the same fleet can GET ' + `/v1/fleet/blobs/${hash}` + ' to fetch it',
5315
+ });
5316
+ }
5317
+
5318
+ /**
5319
+ * v1.0.7 (#1d, the other direction): fetch one blob by the hash that names it.
5320
+ *
5321
+ * ## Why this had to exist before the channel was called finished
5322
+ *
5323
+ * A `publish` with no `pull` is a backup, not a channel: the bytes leave the machine and nothing can bring
5324
+ * them back through the tool surface. The relay route existed and was tested, and the plugin could still only
5325
+ * write -- which is the same "half a mechanism" shape the whole 1a-1d sequence was organised to avoid.
5326
+ *
5327
+ * ## Why the reader re-hashes instead of trusting the relay
5328
+ *
5329
+ * The relay verifies `sha256(body) === key` on write, so in principle what comes back is what was stored. **A
5330
+ * reader must not depend on that.** The hash is the only thing that makes the bytes a claim rather than an
5331
+ * assertion, and a verification performed only by the party that could be wrong is not a verification. So the
5332
+ * content is re-hashed here, and a mismatch is a failure rather than a warning -- accepting unverified bytes
5333
+ * under a name they do not match would make every later reader's check decorative too.
5334
+ */
5335
+ if (typeof args?.action === 'string' && args.action === 'pull') {
5336
+ const wanted = typeof args?.hash === 'string' ? args.hash.trim().toLowerCase() : '';
5337
+ if (!/^[0-9a-f]{64}$/.test(wanted)) {
5338
+ throw new W2MError(
5339
+ 'W2M_CONFIG',
5340
+ 'W2M_CONFIG: `action: "pull"` needs `hash`, a SHA-256 of 64 lowercase hex characters',
5341
+ { hint: 'the hash is what `action: "publish"` returned for the document you want' },
5342
+ );
5343
+ }
5344
+ const base = requireRabbit(cfg);
5345
+ const token = await resolveToken(cfg);
5346
+ const result = await request({
5347
+ rabbitUrl: base,
5348
+ token,
5349
+ pathname: `/v1/fleet/blobs/${wanted}`,
5350
+ accept: 'application/octet-stream',
5351
+ signal: exec?.signal,
5352
+ });
5353
+ if (result.status === 404) {
5354
+ // Deliberately the same answer for "never stored" and "another fleet's": see the `hash` parameter note.
5355
+ throw new W2MError(
5356
+ 'W2M_NOT_FOUND',
5357
+ `W2M_NOT_FOUND: no blob named ${wanted} is readable by this machine`,
5358
+ {
5359
+ hint: 'the hash may belong to another fleet, or the relay may have evicted it -- blobs are evicted '
5360
+ + 'oldest-first once a fleet holds 256',
5361
+ },
5362
+ );
5363
+ }
5364
+ if (result.status !== 200) {
5365
+ throw new W2MError(
5366
+ 'W2M_RABBIT_REFUSED',
5367
+ `W2M_RABBIT_REFUSED: GET /v1/fleet/blobs/${wanted} answered HTTP ${result.status}`,
5368
+ { http_status: result.status, detail: result.error ?? null },
5369
+ );
5370
+ }
5371
+ const body = result.text ?? '';
5372
+ const actual = createHash('sha256').update(body, 'utf8').digest('hex');
5373
+ if (actual !== wanted) {
5374
+ throw new W2MError(
5375
+ 'W2M_BAD_BLOB',
5376
+ `W2M_BAD_BLOB: the bytes returned for ${wanted} hash to ${actual} instead`,
5377
+ {
5378
+ hint: 'the reader refuses unverified content rather than passing it on; nothing was written locally',
5379
+ expected: wanted,
5380
+ actual,
5381
+ },
5382
+ );
5383
+ }
5384
+ return JSON.stringify({
5385
+ ok: true,
5386
+ pulled: true,
5387
+ hash: wanted,
5388
+ bytes: Buffer.byteLength(body, 'utf8'),
5389
+ verified: true,
5390
+ content: body,
5391
+ });
5392
+ }
5393
+
5202
5394
  const limitRaw = Number(args?.limit);
5203
5395
  const limit = Number.isFinite(limitRaw) && limitRaw > 0 ? Math.min(Math.floor(limitRaw), 500) : 20;
5204
5396
 
@@ -4,6 +4,7 @@
4
4
  * `<state>/devices.json` atomic snapshot (tmp file + rename)
5
5
  * `<state>/ledger.jsonl` append-only, one JSON object per line
6
6
  * `<state>/operator-token.txt` single line, 0600
7
+ * `<state>/blobs/<aa>/<sha256>` content-addressed bytes, v1.0.7 (see `putBlob`)
7
8
  *
8
9
  * This is the only relay module allowed to touch the filesystem.
9
10
  * Corruption is never silent: every skipped line / unreadable snapshot is
@@ -12,12 +13,31 @@
12
13
 
13
14
  import fs from 'node:fs';
14
15
  import path from 'node:path';
16
+ import { createHash } from 'node:crypto';
15
17
 
16
18
  import { rfc3339 } from './state.mjs';
17
19
 
18
20
  export const DEVICES_FILE = 'devices.json';
19
21
  export const LEDGER_FILE = 'ledger.jsonl';
20
22
  export const OPERATOR_TOKEN_FILE = 'operator-token.txt';
23
+ /** v1.0.7: the content-addressed blob directory, and the two caps that keep it bounded. */
24
+ export const BLOB_DIR = 'blobs';
25
+ export const MAX_BLOB_BYTES = 8 * 1024 * 1024;
26
+ export const MAX_BLOBS = 256;
27
+ /**
28
+ * v1.0.7: **which fleet wrote each blob.**
29
+ *
30
+ * Content addressing keys a blob by its own hash, and a hash knows nothing about fleets. So two fleets that store
31
+ * the same bytes land at the same path -- and without this index, the second one could read the first one's content.
32
+ * **A content-addressed store with no ownership record is not isolated, it is shared.**
33
+ *
34
+ * The index is a separate document rather than part of the path (`blobs/<fleet>/<aa>/<hash>`) for one concrete
35
+ * reason: **deduplication is only sound when the content is identical**, and identical bytes stored twice would
36
+ * otherwise occupy two entries. The path stays a pure function of the content; the fleet is metadata.
37
+ *
38
+ * Shape: `{ "<sha256>": { fleet_id: string|null, added_at_ms: number } }`.
39
+ */
40
+ export const BLOB_INDEX_FILE = 'index.json';
21
41
 
22
42
  /** Ledger entry types understood by `RabbitState#restore`. */
23
43
  export const LEDGER_TYPES = Object.freeze([
@@ -250,8 +270,232 @@ export class Persistence {
250
270
  return true;
251
271
  }
252
272
 
273
+ /* ------------------------------------------------------------------------------------------- */
274
+ /* v1.0.7: content-addressed blobs — the fleet sync channel (#1) */
275
+ /* ------------------------------------------------------------------------------------------- */
276
+
277
+ /**
278
+ * Store bytes under their own SHA-256, and read them back by that hash.
279
+ *
280
+ * ## Why the relay may carry bytes at all, when it deliberately carries none today
281
+ *
282
+ * The relay persists `stdout_sha256` and never the text, and `src/agent/storage.mjs` records why that matters:
283
+ * `memory.jsonl` on each machine is the only durable copy of what a command printed. **That invariant is about
284
+ * the relay not becoming a document store** — it never interprets content and never asserts that content is
285
+ * authentic.
286
+ *
287
+ * A blob keyed by **its own hash** preserves both halves. The relay still interprets nothing; **the hash is what
288
+ * says the bytes are the bytes**, exactly as `stdout_sha256` does for a result. The only thing that is new is
289
+ * that a hash the relay already holds can now have its bytes fetched from the same place.
290
+ *
291
+ * ## The two limits, because an uncapped writer is what caused the original incident
292
+ *
293
+ * `storage.mjs`'s header records the incident that produced its whole design: a single uncapped writer reached
294
+ * 100% filesystem at 14.7 GB/day. A blob channel with no ceiling reproduces that on the relay, so:
295
+ *
296
+ * * **8 MiB per blob** — a `fleet-memory.md` is tens of KB; this is three orders of magnitude of headroom.
297
+ * Larger content is an archive, not a memory.
298
+ * * **256 blobs** — bounded *count* matters as much as bounded size: 256 × 8 MiB caps a fleet's share at
299
+ * 2 GiB regardless of what it writes.
300
+ *
301
+ * Eviction is oldest-first and **never takes the last blob**: deleting the only copy of a fact is the failure
302
+ * `UNDELETABLE_TIERS` exists to prevent locally.
303
+ *
304
+ * @param {Buffer} bytes
305
+ * @param {{fleetId?: string|null}} [options] the owning fleet, recorded so another fleet cannot read it back
306
+ * @returns {{ok:true, hash:string, bytes:number, evicted:string[]}|{ok:false, reason:string, detail:string}}
307
+ */
308
+ putBlob(bytes, options = {}) {
309
+ if (!this.enabled) return { ok: false, reason: 'PERSISTENCE_DISABLED', detail: 'the relay has no state directory' };
310
+ if (!Buffer.isBuffer(bytes) || bytes.length === 0) {
311
+ return { ok: false, reason: 'BLOB_EMPTY', detail: 'an empty blob is a caller bug, not content' };
312
+ }
313
+ if (bytes.length > MAX_BLOB_BYTES) {
314
+ return {
315
+ ok: false,
316
+ reason: 'BLOB_TOO_LARGE',
317
+ detail: `${bytes.length} bytes exceeds the ${MAX_BLOB_BYTES}-byte limit`,
318
+ };
319
+ }
320
+ const hash = createHash('sha256').update(bytes).digest('hex');
321
+ const file = this.blobFile(hash);
322
+ if (file === null) return { ok: false, reason: 'PERSISTENCE_DISABLED', detail: 'no blob directory' };
323
+
324
+ const index = this.readBlobIndex();
325
+ const fleetId = typeof options.fleetId === 'string' && options.fleetId !== '' ? options.fleetId : null;
326
+
327
+ // Content addressing makes the write idempotent: the same bytes always land at the same path.
328
+ if (fs.existsSync(file)) {
329
+ /**
330
+ * Already stored. **The first writer's fleet keeps ownership** -- re-assigning it on a second write would let
331
+ * one fleet claim a blob another fleet already owns by simply PUT-ting the same bytes, which is the same
332
+ * failure as having no index at all.
333
+ */
334
+ const entry = index[hash];
335
+ if (entry === undefined) {
336
+ index[hash] = { fleet_id: fleetId, added_at_ms: this._now() };
337
+ this.writeBlobIndex(index);
338
+ }
339
+ return { ok: true, hash, bytes: bytes.length, evicted: [] };
340
+ }
341
+
342
+ const evicted = this.evictBlobsFor(1);
343
+ for (const gone of evicted) delete index[gone];
344
+ try {
345
+ fs.mkdirSync(path.dirname(file), { recursive: true });
346
+ const tmp = `${file}.tmp-${process.pid}-${this._now()}`;
347
+ fs.writeFileSync(tmp, bytes);
348
+ fs.renameSync(tmp, file);
349
+ } catch (error) {
350
+ this.warn(BLOB_DIR, `blob write failed: ${error?.message ?? String(error)}`);
351
+ return { ok: false, reason: 'BLOB_WRITE_FAILED', detail: error?.message ?? String(error) };
352
+ }
353
+ index[hash] = { fleet_id: fleetId, added_at_ms: this._now() };
354
+ this.writeBlobIndex(index);
355
+ return { ok: true, hash, bytes: bytes.length, evicted };
356
+ }
357
+
358
+ /**
359
+ * The bytes for a hash, or `null`.
360
+ *
361
+ * Never throws -- an unreadable blob is a miss, not a crash -- and **when `fleetId` is supplied, a blob owned by a
362
+ * different fleet is a miss too.** The caller cannot distinguish "no such blob" from "not yours", which is
363
+ * deliberate: a distinguishable refusal tells a prober that a hash exists on this relay.
364
+ *
365
+ * @param {string} hash
366
+ * @param {{fleetId?: string|null|undefined}} [options] omit `fleetId` entirely for an unscoped read
367
+ */
368
+ getBlob(hash, options = {}) {
369
+ const file = this.blobFile(hash);
370
+ if (file === null) return null;
371
+ if (options.fleetId !== undefined) {
372
+ const entry = this.readBlobIndex()[hash];
373
+ // A blob with no recorded owner predates the index or was written unscoped; it is not this fleet's.
374
+ if (entry === undefined || entry.fleet_id !== options.fleetId) return null;
375
+ }
376
+ try {
377
+ return fs.readFileSync(file);
378
+ } catch {
379
+ return null;
380
+ }
381
+ }
382
+
383
+ hasBlob(hash) {
384
+ const file = this.blobFile(hash);
385
+ if (file === null) return false;
386
+ try {
387
+ return fs.statSync(file).isFile();
388
+ } catch {
389
+ return false;
390
+ }
391
+ }
392
+
393
+ /**
394
+ * Every stored blob, newest last, with the size and mtime needed to enforce the count cap.
395
+ *
396
+ * @returns {{hash:string, bytes:number, mtimeMs:number}[]}
397
+ */
398
+ listBlobs() {
399
+ if (!this.enabled) return [];
400
+ const root = path.join(this.dir, BLOB_DIR);
401
+ let shards;
402
+ try {
403
+ shards = fs.readdirSync(root, { withFileTypes: true });
404
+ } catch {
405
+ return [];
406
+ }
407
+ const out = [];
408
+ for (const shard of shards) {
409
+ if (!shard.isDirectory()) continue;
410
+ const shardPath = path.join(root, shard.name);
411
+ let names;
412
+ try {
413
+ names = fs.readdirSync(shardPath, { withFileTypes: true });
414
+ } catch {
415
+ continue;
416
+ }
417
+ for (const entry of names) {
418
+ if (!entry.isFile() || entry.name.includes('.tmp-')) continue;
419
+ try {
420
+ const stat = fs.statSync(path.join(shardPath, entry.name));
421
+ out.push({ hash: entry.name, bytes: stat.size, mtimeMs: stat.mtimeMs });
422
+ } catch {
423
+ /* a blob that vanished mid-scan is not a failure */
424
+ }
425
+ }
426
+ }
427
+ out.sort((a, b) => a.mtimeMs - b.mtimeMs);
428
+ return out;
429
+ }
430
+
431
+ /**
432
+ * Make room for `incoming` more blobs, oldest first, and never take the last one.
433
+ *
434
+ * **The caller must prune the index for the returned hashes** -- this method deletes files, and the index is a
435
+ * separate document. `putBlob` does that pruning; a future caller that evicts for another reason owes the same.
436
+ *
437
+ * @param {number} incoming
438
+ * @returns {string[]} the hashes evicted
439
+ */
440
+ evictBlobsFor(incoming) {
441
+ const existing = this.listBlobs();
442
+ const over = existing.length + incoming - MAX_BLOBS;
443
+ if (over <= 0) return [];
444
+ const evicted = [];
445
+ let count = existing.length;
446
+ for (const blob of existing) {
447
+ if (count <= 1) break; // never delete the only copy of anything
448
+ try {
449
+ fs.rmSync(path.join(this.dir, BLOB_DIR, blob.hash.slice(0, 2), blob.hash), { force: true });
450
+ evicted.push(blob.hash);
451
+ count -= 1;
452
+ } catch {
453
+ /* leave it; the caller will see the count did not drop */
454
+ }
455
+ if (evicted.length >= over) break;
456
+ }
457
+ if (evicted.length > 0) this.warn(BLOB_DIR, `evicted ${evicted.length} blob(s) to stay within ${MAX_BLOBS}`);
458
+ return evicted;
459
+ }
460
+
461
+ /** Read the hash -> owner map. A missing or corrupt index reads as empty rather than taking the relay down. */
462
+ readBlobIndex() {
463
+ if (!this.enabled) return {};
464
+ try {
465
+ const raw = fs.readFileSync(path.join(this.dir, BLOB_DIR, BLOB_INDEX_FILE), 'utf8');
466
+ const parsed = JSON.parse(raw);
467
+ return parsed !== null && typeof parsed === 'object' && !Array.isArray(parsed) ? parsed : {};
468
+ } catch {
469
+ return {};
470
+ }
471
+ }
472
+
473
+ /** Atomically replace the index. Best-effort: a failure is warned about, never thrown at the caller. */
474
+ writeBlobIndex(index) {
475
+ if (!this.enabled) return false;
476
+ const file = path.join(this.dir, BLOB_DIR, BLOB_INDEX_FILE);
477
+ try {
478
+ fs.mkdirSync(path.dirname(file), { recursive: true });
479
+ const tmp = `${file}.tmp-${process.pid}-${this._now()}`;
480
+ fs.writeFileSync(tmp, `${JSON.stringify(index)}\n`, 'utf8');
481
+ fs.renameSync(tmp, file);
482
+ return true;
483
+ } catch (error) {
484
+ this.warn(BLOB_DIR, `blob index write failed: ${error?.message ?? String(error)}`);
485
+ return false;
486
+ }
487
+ }
488
+
489
+ /** `<state>/blobs/<aa>/<sha256>`, or `null` when persistence is off or the hash is not a sha256. */
490
+ blobFile(hash) {
491
+ if (!this.enabled) return null;
492
+ if (typeof hash !== 'string' || !/^[0-9a-f]{64}$/.test(hash)) return null;
493
+ return path.join(this.dir, BLOB_DIR, hash.slice(0, 2), hash);
494
+ }
495
+
253
496
  /** Redacted view for `/healthz` — never contains tokens. */
254
497
  describe() {
498
+ const blobs = this.listBlobs();
255
499
  return {
256
500
  enabled: this.enabled,
257
501
  dir: this.dir,
@@ -261,6 +505,11 @@ export class Persistence {
261
505
  devices_missing: this.devicesMissing,
262
506
  ledger_lines_read: this.ledgerLinesRead,
263
507
  ledger_lines_skipped: this.ledgerLinesSkipped,
508
+ /** v1.0.7: the fleet sync channel's occupancy, reported as counts and bytes — never contents. */
509
+ blobs: blobs.length,
510
+ blob_bytes: blobs.reduce((sum, b) => sum + b.bytes, 0),
511
+ blobs_limit: MAX_BLOBS,
512
+ blob_bytes_limit: MAX_BLOB_BYTES,
264
513
  /**
265
514
  * v0.6.2. `degraded: true` means the relay is answering requests while failing to record them,
266
515
  * which is the one relay failure that looks exactly like health from the outside.
@@ -13,7 +13,7 @@ import https from 'node:https';
13
13
  import fs from 'node:fs';
14
14
  import os from 'node:os';
15
15
  import path from 'node:path';
16
- import { randomBytes } from 'node:crypto';
16
+ import { createHash, randomBytes } from 'node:crypto';
17
17
 
18
18
  import {
19
19
  MAX_FRAME_BYTES,
@@ -761,6 +761,77 @@ export class RelayServer {
761
761
  const { _instanceKey, ...clean } = result;
762
762
  return sendJson(res, 200, clean);
763
763
  }
764
+ /**
765
+ * v1.0.7: the content-addressed blob channel (#1).
766
+ *
767
+ * ## Why the relay may carry bytes at all
768
+ *
769
+ * It persists `stdout_sha256` and never the text, and `src/agent/storage.mjs` records why: each machine's
770
+ * `memory.jsonl` is the only durable copy of what a command printed. **That invariant is about the relay not
771
+ * interpreting content** -- it never decides what bytes mean and never asserts they are authentic.
772
+ *
773
+ * A blob keyed by its own hash preserves exactly that. The relay still interprets nothing; **the hash is what
774
+ * says the bytes are the bytes**, exactly as `stdout_sha256` does for a result. What is new is only that a hash
775
+ * the relay already holds can have its bytes fetched from the same place.
776
+ *
777
+ * ## Why the hash is verified before storing
778
+ *
779
+ * A content-addressed store that accepts bytes under a key they do not hash to is not content-addressed -- the
780
+ * key stops being a claim about the content, and every later reader's verification becomes decorative. So a
781
+ * mismatch is **refused (422)**, never stored and never corrected: correcting it would silently re-key a blob
782
+ * the caller believes it stored under a different name.
783
+ *
784
+ * ## Fleet scoping
785
+ *
786
+ * `putFleetBlob`/`getFleetBlob` reuse `scopeFleetId`, the same rule task dispatch obeys. A blob written by one
787
+ * fleet is **not readable by another**, and the read refusal is a 404 rather than a 403: a distinguishable
788
+ * "not yours" tells a prober that the hash exists here.
789
+ */
790
+ const blobMatch = /^\/v1\/fleet\/blobs\/([0-9a-f]{64})$/.exec(path);
791
+ if (blobMatch !== null) {
792
+ const hash = blobMatch[1];
793
+ if (method === 'PUT') {
794
+ const received = rawBody ?? Buffer.alloc(0);
795
+ const actual = createHash('sha256').update(received).digest('hex');
796
+ if (actual !== hash) {
797
+ throw new ProtocolError('BLOB_HASH_MISMATCH', 'the body does not hash to the key it was sent under', {
798
+ expected: hash,
799
+ actual,
800
+ bytes: received.length,
801
+ });
802
+ }
803
+ const stored = this.state.putFleetBlob(device, received);
804
+ if (stored.ok !== true) {
805
+ throw new ProtocolError(stored.reason ?? 'BLOB_WRITE_FAILED', stored.detail ?? 'the blob was refused', {
806
+ hash,
807
+ });
808
+ }
809
+ return sendJson(res, 201, {
810
+ protocol_version: PROTOCOL_VERSION,
811
+ ok: true,
812
+ hash: stored.hash,
813
+ bytes: stored.bytes,
814
+ evicted: stored.evicted,
815
+ });
816
+ }
817
+ if (method === 'GET') {
818
+ const bytes = this.state.getFleetBlob(device, hash);
819
+ if (bytes === null) {
820
+ // Same code and shape for "no such blob" and "not your fleet's" -- deliberately indistinguishable.
821
+ throw new ProtocolError('NOT_FOUND', 'no such blob is readable by this device', { hash });
822
+ }
823
+ res.writeHead(200, {
824
+ 'content-type': 'application/octet-stream',
825
+ 'content-length': bytes.length,
826
+ // Same reason as `/metrics`: a memory document must not be cached by an intermediary.
827
+ 'cache-control': 'no-store',
828
+ });
829
+ res.end(bytes);
830
+ return true;
831
+ }
832
+ throw new ProtocolError('BAD_REQUEST', 'a blob is addressed with PUT or GET', { method });
833
+ }
834
+
764
835
  if (method === 'GET' && path === '/v1/devices') {
765
836
  return sendJson(res, 200, {
766
837
  protocol_version: PROTOCOL_VERSION,
@@ -17,6 +17,9 @@ import { createHash, randomBytes } from 'node:crypto';
17
17
  // exhaustive tests — three lines inside this object would be untestable without a live relay.
18
18
  import { targetsFor } from '../chatops/fleet-wiring.mjs';
19
19
  import { deviceFleetFields } from '../chatops/fleet-wiring.mjs';
20
+ // v1.0.7: the one home for "which fleet does this requester act as", reused by the blob channel so it cannot
21
+ // drift from the rule the task routes already obey.
22
+ import { scopeFleetId } from '../chatops/fleet-wiring.mjs';
20
23
  // v0.8.1: the account-derived fleet key, so a machine can join with no pairing code.
21
24
  import { planJoin } from '../chatops/fleet-join.mjs';
22
25
 
@@ -56,6 +59,21 @@ export const ERROR_STATUS = Object.freeze({
56
59
  SIGNATURE_EXPIRED: 401,
57
60
  SIGNATURE_REPLAY: 401,
58
61
  SIGNING_NOT_CONFIGURED: 500,
62
+
63
+ /* v1.0.7: the content-addressed blob channel.
64
+ *
65
+ * `BLOB_HASH_MISMATCH` is **422**, not 400: the request is well-formed and the bytes are fine, but they do not
66
+ * hash to the key they were sent under, so storing them would break content addressing for every later reader.
67
+ * That is a semantically invalid request, not a malformed one -- 422 is the status that says so.
68
+ *
69
+ * `PERSISTENCE_DISABLED` is **503**: the relay was started without a state directory, so the channel cannot
70
+ * exist. Retrying will not fix a missing directory, but a caller is still right to treat it as "this relay cannot
71
+ * serve this right now" rather than as its own mistake. */
72
+ BLOB_HASH_MISMATCH: 422,
73
+ BLOB_TOO_LARGE: 413,
74
+ BLOB_EMPTY: 400,
75
+ BLOB_WRITE_FAILED: 500,
76
+ PERSISTENCE_DISABLED: 503,
59
77
  });
60
78
 
61
79
  /** §5.1 required envelope fields — missing any one means `unverifiable`. */
@@ -510,6 +528,57 @@ export class RabbitState {
510
528
  /** Inject/replace the persistence backend (also used by tests). */
511
529
  setPersistence(persistence) { this._persist = persistence ?? null; return this._persist; }
512
530
 
531
+ /* ---------------- v1.0.7: the content-addressed blob channel (#1) ---------------- */
532
+
533
+ /**
534
+ * Which fleet a device acts as when it writes or reads a blob.
535
+ *
536
+ * **Reuses `scopeFleetId` rather than re-deriving it.** The rule "the requester's fleet comes from the device
537
+ * record the relay wrote at pairing time, never from the request" is already stated, tested and enforced for task
538
+ * dispatch; a second copy here would be a second place for it to drift, and the blob channel is exactly where a
539
+ * drift would leak one fleet's memory into another's.
540
+ *
541
+ * @param {{machine_id?: string, fleet_id?: string|null}|null} device
542
+ * @returns {{ok:true, fleetId:string|null, legacy:boolean}|{ok:false, reason:string, text:string}}
543
+ */
544
+ scopeBlobFleet(device) {
545
+ return scopeFleetId({ device, relayFleetId: this.fleetId ?? null });
546
+ }
547
+
548
+ /**
549
+ * Store bytes under their SHA-256, owned by the writer's fleet.
550
+ *
551
+ * Delegates to `Persistence` -- this class never touches the filesystem (v0.1.2 §7), and that constraint is what
552
+ * keeps the durable path in one place.
553
+ *
554
+ * @param {object} device the authenticated device
555
+ * @param {Buffer} bytes
556
+ */
557
+ putFleetBlob(device, bytes) {
558
+ if (!this._persist) {
559
+ return { ok: false, reason: 'PERSISTENCE_DISABLED', detail: 'the relay has no persistence backend' };
560
+ }
561
+ const scope = this.scopeBlobFleet(device);
562
+ if (scope.ok !== true) return { ok: false, reason: scope.reason, detail: scope.text };
563
+ return this._persist.putBlob(bytes, { fleetId: scope.fleetId });
564
+ }
565
+
566
+ /**
567
+ * The bytes for a hash, or `null`.
568
+ *
569
+ * `null` covers three cases on purpose -- no such blob, a blob owned by another fleet, and no persistence -- so a
570
+ * caller cannot use the relay to enumerate which hashes exist elsewhere on it.
571
+ *
572
+ * @param {object} device the authenticated device
573
+ * @param {string} hash
574
+ */
575
+ getFleetBlob(device, hash) {
576
+ if (!this._persist) return null;
577
+ const scope = this.scopeBlobFleet(device);
578
+ if (scope.ok !== true) return null;
579
+ return this._persist.getBlob(hash, { fleetId: scope.fleetId });
580
+ }
581
+
513
582
  /** Snapshot the device table; failures must never break the request path. */
514
583
  _saveDevices() {
515
584
  if (this._restoring || !this._persist) return false;