@twinsearth/w2m-dsh-plugin 1.0.6 → 1.0.7

package/CHANGELOG.md

@@ -2,6 +2,69 @@
 
 All notable changes to this project are documented here.
 
+## [1.0.7] — 2026-10-17
+
+**The fleet sync channel, both directions: a machine can publish its collective memory to the relay and another
+machine in the same fleet can fetch it back by hash.**
+
+### #1 is now a channel, not a backup
+
+v1.0.6 shipped the storage layer and said plainly that it had no route and no caller. This release supplies both:
+
+| Piece | Where |
+|---|---|
+| content-addressed store | `src/relay/persistence.mjs` — `putBlob`/`getBlob`, 8 MiB per blob, 256 per fleet, oldest-first eviction that never takes the last one |
+| hash → fleet ownership | `<state>/blobs/index.json`, so a content-addressed store is *not* a shared one |
+| two routes | `PUT`/`GET /v1/fleet/blobs/<sha256>` in `src/relay/server.mjs`, behind the same device-token gate as every other route |
+| scoping | `state.scopeBlobFleet` reuses `scopeFleetId` — **the same rule task dispatch obeys**, not a second copy of it |
+| caller | `w2m_memory` `action: "publish"` and `"pull"` |
+
+**Why a content-addressed store does not contradict "the relay keeps only hashes".** It persists `stdout_sha256`
+and never the text, and `src/agent/storage.mjs` records why: each machine's `memory.jsonl` is the only durable copy
+of what a command printed. **That invariant is about the relay not interpreting content.** A blob keyed by its own
+hash preserves it exactly — the relay still interprets nothing, and **the hash is what says the bytes are the
+bytes**, as `stdout_sha256` does for a result. What is new is only that a hash the relay already holds can now have
+its bytes fetched from the same place.
+
+### The decisions worth recording
+
+**The write verifies the key, and refuses rather than correcting.** A content-addressed store that accepts bytes
+under a key they do not hash to is not content-addressed: the key stops being a claim, and every later reader's
+verification becomes decorative. A mismatch is `422 BLOB_HASH_MISMATCH` and nothing is stored.
+
+**Ownership is first-writer-wins.** A second fleet PUT-ing an identical blob does **not** take ownership of it.
+Otherwise any fleet could claim another's blob by replaying the same bytes, which is the same failure as having no
+index at all.
+
+**A cross-fleet read is `404`, not `403`.** A distinguishable "not yours" tells a prober that this hash exists on
+this relay. The reader cannot tell "never stored" from "another fleet's", deliberately.
+
+**The pull side re-hashes what it receives.** A verification performed only by the party that could be wrong is not
+a verification, so a reader that gets bytes hashing to something else raises `W2M_BAD_BLOB` rather than passing
+them on.
+
+### What is verified, and what is not
+
+**Verified — executed:** four new tests in `test/relay.test.mjs` drive the real routes against a real relay:
+store-and-return-unchanged; a mismatched key is refused (422) and leaves the key absent (404); one fleet cannot read
+another's blob (404); an unauthenticated request is refused (401). Full suite **2356 tests, 2342 pass, 0 fail, 14
+skip**; pre-flight **GREEN, 15 of 15 checks**.
+
+**NOT verified: the `publish`/`pull` tool actions end to end.** They are type-correct and lint-clean, and their
+failure paths are written, but **no test executes them** — they need a profile with a `fleet-memory.md` and a live
+relay. They are marked unverified here rather than counted as working, which is the distinction this project keeps
+having to restate.
+
+**A known limit of the reader:** `pull` returns the blob as text and re-hashes it as UTF-8. That is exact for a
+`fleet-memory.md`; it would not be for arbitrary binary content, and the channel does not claim to carry that.
+
+### Not in this release
+
+The trigger (#3) and merge semantics (#2) are deliberately absent. Publishing is a decision a machine makes, not
+something that happens on join, and merge needs a conflict rule that has not been decided: two machines writing
+different facts produce **two hashes**, and nothing yet says who wins. **Nothing is overwritten in the meantime —
+that is the property #1 was built to provide.**
+
 ## [1.0.6] — 2026-10-17
 
 **DSH-GPW now ships *findable*: installed with the package, present in the plugin list, and off until an operator

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@twinsearth/w2m-dsh-plugin",
-  "version": "1.0.6",
+  "version": "1.0.7",
   "description": "Run one project across your own Windows and macOS machines from a single DeepSeek account, through 11 w2m_* tools. Four coordination modes (replicate, split, broadcast, pipeline) and six aggregate verdicts (consistent, divergent, divergent-platform, failed, partial, unverifiable). P2P hole punching by default, with a bundled relay, STUN responder and ledger fallback, so the fleet needs no third-party service. Joining needs a pairing code plus autoStartAgent: true; installing alone does not join.",
   "type": "module",
   "main": "lib/index.js",

package/plugin.json

@@ -1,7 +1,7 @@
 {
   "$schema": "https://agent-plugins.org/schemas/1.0.0/plugin.schema.json",
   "name": "twinsearth-w2m-dsh",
-  "version": "1.0.6",
+  "version": "1.0.7",
   "description": "Run one project across your own Windows and macOS machines from a single DeepSeek account, through 11 w2m_* tools. Four coordination modes (replicate, split, broadcast, pipeline) and six aggregate verdicts (consistent, divergent, divergent-platform, failed, partial, unverifiable). P2P hole punching by default, with a bundled relay, STUN responder and ledger fallback, so the fleet needs no third-party service. Joining needs a pairing code plus autoStartAgent: true; installing alone does not join.",
   "license": "MIT",
   "keywords": [

package/src/plugin/tools.mjs

@@ -208,7 +208,7 @@ const MAX_STAGES = 16;
  * `package.json` during `scripts/pack.mjs` rather than hand-maintained, and `W2M_PLUGIN_VERSION`
  * overrides it at runtime for tests and packagers.
  */
-const PLUGIN_VERSION = '1.0.6';
+const PLUGIN_VERSION = '1.0.7';
 
 /** How long any single HTTP call may take before it is aborted, in milliseconds. */
 const DEFAULT_REQUEST_TIMEOUT_MS = 15_000;
@@ -2590,10 +2590,27 @@ async function request(options) {
     pathname,
     query,
     body,
+    /**
+     * v1.0.7: raw bytes to send instead of a JSON body.
+     *
+     * Added for the blob channel (#1), which PUTs a memory document's exact octets. **Added here rather than as a
+     * second `fetch` in the caller**, because `tools.mjs:1283` already states the rule: the direct path must not grow
+     * a second way to talk to the relay -- bearer-token resolution, base-path joining, timeouts and abort handling
+     * all live in this one function, and a duplicate would be a duplicate of all four.
+     *
+     * `body` and `rawBody` are mutually exclusive: passing both is a caller bug, not a merge to guess at.
+     */
+    rawBody,
     maxBytes = MAX_RESPONSE_BYTES,
     accept = 'application/json',
   } = options;
 
+  if (rawBody !== undefined && body !== undefined) {
+    throw new W2MError('W2M_CONFIG', 'W2M_CONFIG: `body` and `rawBody` are mutually exclusive', {
+      hint: 'pass `rawBody` for octets, `body` for JSON -- not both',
+    });
+  }
+
   throwIfAborted(signal);
 
   // Named here rather than left to `new URL`, whose failure would say only "Invalid URL" and send
@@ -2613,7 +2630,10 @@ async function request(options) {
 
   const headers = { accept };
   if (token) headers.authorization = `Bearer ${token}`;
-  if (body !== undefined) headers['content-type'] = 'application/json';
+  // v1.0.7: octets keep their own content type. JSON-stringifying them would corrupt the bytes and, because the
+  // relay verifies `sha256(body) === key`, corrupting them here would surface as a 422 blamed on the caller.
+  if (rawBody !== undefined) headers['content-type'] = 'application/octet-stream';
+  else if (body !== undefined) headers['content-type'] = 'application/json';
 
   const timeoutController = new AbortController();
   const timer = setTimeout(() => timeoutController.abort(new Error('timeout')), timeoutMs);
@@ -2624,7 +2644,7 @@ async function request(options) {
     response = await fetch(target, {
       method,
       headers,
-      body: body === undefined ? undefined : JSON.stringify(body),
+      body: rawBody !== undefined ? rawBody : (body === undefined ? undefined : JSON.stringify(body)),
       signal: combined.signal,
     });
   } catch (error) {
@@ -5112,6 +5132,30 @@ export async function apply(ctx, config = {}) {
       'relay keeps only hashes, so these files are the only place a command\'s output survives. Use it ' +
       'before re-running work -- it is what turns a dozen rediscovery dispatches into one lookup.',
     parameters: {
+      /**
+       * v1.0.7: `read` (the default, and the only behaviour before this release), `publish`, or `pull`.
+       *
+       * Defaulting to `read` is deliberate: every existing caller says nothing about `action`, and a tool that
+       * changed what "call `w2m_memory`" means would be a silent behaviour change on the one path that is supposed
+       * to be safe to call blindly.
+       */
+      action: {
+        type: 'string',
+        enum: ['read', 'publish', 'pull'],
+        description:
+          'read (default) = answer from memory. publish = upload the curated collective memory '
+          + '(`fleet-memory.md`) to the relay as a content-addressed blob and return its SHA-256. pull = fetch one '
+          + 'blob by its SHA-256 and return its text, re-hashing what came back against the name it was asked for. '
+          + 'Neither publish nor pull modifies the local file.',
+        default: 'read',
+      },
+      hash: {
+        type: 'string',
+        description:
+          'The SHA-256 naming the blob, for action="pull": 64 lowercase hex characters. A hash that exists on the '
+          + 'relay but belongs to another fleet is reported as not found, exactly like a hash that was never '
+          + 'stored -- a distinguishable refusal would confirm to a prober which hashes this relay holds.',
+      },
       scope: {
         type: 'string',
         enum: ['local', 'fleet'],
@@ -5199,6 +5243,154 @@ export async function apply(ctx, config = {}) {
     output: jsonOutput,
     async execute(args, exec) {
       throwIfAborted(exec?.signal);
+
+      /**
+       * v1.0.7 (#1c): publish the collective memory as a content-addressed blob.
+       *
+       * ## Why this exists at all
+       *
+       * The blob channel (1a/1b) is a route and a store; **without a caller it is the same defect as the seven
+       * written-but-unreachable modules this repository keeps an accepted-orphans list for.** This is the caller:
+       * a machine that wants its curated memory to be fetchable by its own fleet says so, and gets back the hash
+       * that names it.
+       *
+       * ## Why it reads the file and does not rewrite it
+       *
+       * Publishing is a **copy, not a move**. If it deleted or normalised the file, the only durable copy of what
+       * this machine knows would depend on the relay having accepted it -- and the relay is explicitly not the
+       * durable store (`storage.mjs`: each machine's `memory.jsonl` is the only copy of what a command printed).
+       *
+       * ## Why a missing file is a named refusal
+       *
+       * `putBlob` rejects an empty body with `BLOB_EMPTY`, which is correct but arrives too late to say anything
+       * useful. A machine with no `fleet-memory.md` has not published anything yet -- that is a state, not a
+       * protocol violation, and the message should say which file is missing and where it was looked for.
+       */
+      if (typeof args?.action === 'string' && args.action === 'publish') {
+        const file = collectivePath();
+        if (file === null) {
+          throw new W2MError('W2M_CONFIG', 'W2M_CONFIG: there is no profile to publish collective memory from', {
+            hint: 'the collective layer lives beside the profile that mounted this plugin; set `stateDir` or run '
+              + 'with a profile',
+          });
+        }
+        let bytes;
+        try {
+          // `fs` is `promises as fs` in this file (line 93), so this is the promise API -- a bare `readFile` is not
+          // in scope and would throw ReferenceError at the exact moment the caller wanted to publish.
+          bytes = await fs.readFile(file);
+        } catch {
+          // The error object is deliberately not bound: every refusal of a read here is the same refusal (there is
+          // no document at that path), and naming the errno would tell the caller to debug a filesystem this plugin
+          // does not own.
+          throw new W2MError(
+            'W2M_NOT_FOUND',
+            `W2M_NOT_FOUND: no collective memory to publish at ${file}`,
+            { hint: 'write measured facts to it first; publishing copies the file, it does not create one' },
+          );
+        }
+        const hash = createHash('sha256').update(bytes).digest('hex');
+        const base = requireRabbit(cfg);
+        const token = await resolveToken(cfg);
+        const result = await request({
+          rabbitUrl: base,
+          token,
+          method: 'PUT',
+          pathname: `/v1/fleet/blobs/${hash}`,
+          rawBody: bytes,
+          signal: exec?.signal,
+        });
+        return JSON.stringify({
+          ok: true,
+          published: true,
+          hash,
+          bytes: bytes.length,
+          file,
+          relay: {
+            http_status: result.status,
+            // `request()` returns `{status, ok, json, text, error}` -- the parsed body is `json`, not `body`.
+            evicted: result.json?.evicted ?? [],
+          },
+          next: 'another machine in the same fleet can GET ' + `/v1/fleet/blobs/${hash}` + ' to fetch it',
+        });
+      }
+
+      /**
+       * v1.0.7 (#1d, the other direction): fetch one blob by the hash that names it.
+       *
+       * ## Why this had to exist before the channel was called finished
+       *
+       * A `publish` with no `pull` is a backup, not a channel: the bytes leave the machine and nothing can bring
+       * them back through the tool surface. The relay route existed and was tested, and the plugin could still only
+       * write -- which is the same "half a mechanism" shape the whole 1a-1d sequence was organised to avoid.
+       *
+       * ## Why the reader re-hashes instead of trusting the relay
+       *
+       * The relay verifies `sha256(body) === key` on write, so in principle what comes back is what was stored. **A
+       * reader must not depend on that.** The hash is the only thing that makes the bytes a claim rather than an
+       * assertion, and a verification performed only by the party that could be wrong is not a verification. So the
+       * content is re-hashed here, and a mismatch is a failure rather than a warning -- accepting unverified bytes
+       * under a name they do not match would make every later reader's check decorative too.
+       */
+      if (typeof args?.action === 'string' && args.action === 'pull') {
+        const wanted = typeof args?.hash === 'string' ? args.hash.trim().toLowerCase() : '';
+        if (!/^[0-9a-f]{64}$/.test(wanted)) {
+          throw new W2MError(
+            'W2M_CONFIG',
+            'W2M_CONFIG: `action: "pull"` needs `hash`, a SHA-256 of 64 lowercase hex characters',
+            { hint: 'the hash is what `action: "publish"` returned for the document you want' },
+          );
+        }
+        const base = requireRabbit(cfg);
+        const token = await resolveToken(cfg);
+        const result = await request({
+          rabbitUrl: base,
+          token,
+          pathname: `/v1/fleet/blobs/${wanted}`,
+          accept: 'application/octet-stream',
+          signal: exec?.signal,
+        });
+        if (result.status === 404) {
+          // Deliberately the same answer for "never stored" and "another fleet's": see the `hash` parameter note.
+          throw new W2MError(
+            'W2M_NOT_FOUND',
+            `W2M_NOT_FOUND: no blob named ${wanted} is readable by this machine`,
+            {
+              hint: 'the hash may belong to another fleet, or the relay may have evicted it -- blobs are evicted '
+                + 'oldest-first once a fleet holds 256',
+            },
+          );
+        }
+        if (result.status !== 200) {
+          throw new W2MError(
+            'W2M_RABBIT_REFUSED',
+            `W2M_RABBIT_REFUSED: GET /v1/fleet/blobs/${wanted} answered HTTP ${result.status}`,
+            { http_status: result.status, detail: result.error ?? null },
+          );
+        }
+        const body = result.text ?? '';
+        const actual = createHash('sha256').update(body, 'utf8').digest('hex');
+        if (actual !== wanted) {
+          throw new W2MError(
+            'W2M_BAD_BLOB',
+            `W2M_BAD_BLOB: the bytes returned for ${wanted} hash to ${actual} instead`,
+            {
+              hint: 'the reader refuses unverified content rather than passing it on; nothing was written locally',
+              expected: wanted,
+              actual,
+            },
+          );
+        }
+        return JSON.stringify({
+          ok: true,
+          pulled: true,
+          hash: wanted,
+          bytes: Buffer.byteLength(body, 'utf8'),
+          verified: true,
+          content: body,
+        });
+      }
+
       const limitRaw = Number(args?.limit);
       const limit = Number.isFinite(limitRaw) && limitRaw > 0 ? Math.min(Math.floor(limitRaw), 500) : 20;
 

package/src/relay/server.mjs

@@ -13,7 +13,7 @@ import https from 'node:https';
 import fs from 'node:fs';
 import os from 'node:os';
 import path from 'node:path';
-import { randomBytes } from 'node:crypto';
+import { createHash, randomBytes } from 'node:crypto';
 
 import {
   MAX_FRAME_BYTES,
@@ -761,6 +761,77 @@ export class RelayServer {
       const { _instanceKey, ...clean } = result;
       return sendJson(res, 200, clean);
     }
+    /**
+     * v1.0.7: the content-addressed blob channel (#1).
+     *
+     * ## Why the relay may carry bytes at all
+     *
+     * It persists `stdout_sha256` and never the text, and `src/agent/storage.mjs` records why: each machine's
+     * `memory.jsonl` is the only durable copy of what a command printed. **That invariant is about the relay not
+     * interpreting content** -- it never decides what bytes mean and never asserts they are authentic.
+     *
+     * A blob keyed by its own hash preserves exactly that. The relay still interprets nothing; **the hash is what
+     * says the bytes are the bytes**, exactly as `stdout_sha256` does for a result. What is new is only that a hash
+     * the relay already holds can have its bytes fetched from the same place.
+     *
+     * ## Why the hash is verified before storing
+     *
+     * A content-addressed store that accepts bytes under a key they do not hash to is not content-addressed -- the
+     * key stops being a claim about the content, and every later reader's verification becomes decorative. So a
+     * mismatch is **refused (422)**, never stored and never corrected: correcting it would silently re-key a blob
+     * the caller believes it stored under a different name.
+     *
+     * ## Fleet scoping
+     *
+     * `putFleetBlob`/`getFleetBlob` reuse `scopeFleetId`, the same rule task dispatch obeys. A blob written by one
+     * fleet is **not readable by another**, and the read refusal is a 404 rather than a 403: a distinguishable
+     * "not yours" tells a prober that the hash exists here.
+     */
+    const blobMatch = /^\/v1\/fleet\/blobs\/([0-9a-f]{64})$/.exec(path);
+    if (blobMatch !== null) {
+      const hash = blobMatch[1];
+      if (method === 'PUT') {
+        const received = rawBody ?? Buffer.alloc(0);
+        const actual = createHash('sha256').update(received).digest('hex');
+        if (actual !== hash) {
+          throw new ProtocolError('BLOB_HASH_MISMATCH', 'the body does not hash to the key it was sent under', {
+            expected: hash,
+            actual,
+            bytes: received.length,
+          });
+        }
+        const stored = this.state.putFleetBlob(device, received);
+        if (stored.ok !== true) {
+          throw new ProtocolError(stored.reason ?? 'BLOB_WRITE_FAILED', stored.detail ?? 'the blob was refused', {
+            hash,
+          });
+        }
+        return sendJson(res, 201, {
+          protocol_version: PROTOCOL_VERSION,
+          ok: true,
+          hash: stored.hash,
+          bytes: stored.bytes,
+          evicted: stored.evicted,
+        });
+      }
+      if (method === 'GET') {
+        const bytes = this.state.getFleetBlob(device, hash);
+        if (bytes === null) {
+          // Same code and shape for "no such blob" and "not your fleet's" -- deliberately indistinguishable.
+          throw new ProtocolError('NOT_FOUND', 'no such blob is readable by this device', { hash });
+        }
+        res.writeHead(200, {
+          'content-type': 'application/octet-stream',
+          'content-length': bytes.length,
+          // Same reason as `/metrics`: a memory document must not be cached by an intermediary.
+          'cache-control': 'no-store',
+        });
+        res.end(bytes);
+        return true;
+      }
+      throw new ProtocolError('BAD_REQUEST', 'a blob is addressed with PUT or GET', { method });
+    }
+
     if (method === 'GET' && path === '/v1/devices') {
       return sendJson(res, 200, {
         protocol_version: PROTOCOL_VERSION,

package/src/relay/state.mjs

@@ -17,6 +17,9 @@ import { createHash, randomBytes } from 'node:crypto';
 // exhaustive tests — three lines inside this object would be untestable without a live relay.
 import { targetsFor } from '../chatops/fleet-wiring.mjs';
 import { deviceFleetFields } from '../chatops/fleet-wiring.mjs';
+// v1.0.7: the one home for "which fleet does this requester act as", reused by the blob channel so it cannot
+// drift from the rule the task routes already obey.
+import { scopeFleetId } from '../chatops/fleet-wiring.mjs';
 // v0.8.1: the account-derived fleet key, so a machine can join with no pairing code.
 import { planJoin } from '../chatops/fleet-join.mjs';
 
@@ -56,6 +59,21 @@ export const ERROR_STATUS = Object.freeze({
   SIGNATURE_EXPIRED: 401,
   SIGNATURE_REPLAY: 401,
   SIGNING_NOT_CONFIGURED: 500,
+
+  /* v1.0.7: the content-addressed blob channel.
+   *
+   * `BLOB_HASH_MISMATCH` is **422**, not 400: the request is well-formed and the bytes are fine, but they do not
+   * hash to the key they were sent under, so storing them would break content addressing for every later reader.
+   * That is a semantically invalid request, not a malformed one -- 422 is the status that says so.
+   *
+   * `PERSISTENCE_DISABLED` is **503**: the relay was started without a state directory, so the channel cannot
+   * exist. Retrying will not fix a missing directory, but a caller is still right to treat it as "this relay cannot
+   * serve this right now" rather than as its own mistake. */
+  BLOB_HASH_MISMATCH: 422,
+  BLOB_TOO_LARGE: 413,
+  BLOB_EMPTY: 400,
+  BLOB_WRITE_FAILED: 500,
+  PERSISTENCE_DISABLED: 503,
 });
 
 /** §5.1 required envelope fields — missing any one means `unverifiable`. */
@@ -510,6 +528,57 @@ export class RabbitState {
   /** Inject/replace the persistence backend (also used by tests). */
   setPersistence(persistence) { this._persist = persistence ?? null; return this._persist; }
 
+  /* ---------------- v1.0.7: the content-addressed blob channel (#1) ---------------- */
+
+  /**
+   * Which fleet a device acts as when it writes or reads a blob.
+   *
+   * **Reuses `scopeFleetId` rather than re-deriving it.** The rule "the requester's fleet comes from the device
+   * record the relay wrote at pairing time, never from the request" is already stated, tested and enforced for task
+   * dispatch; a second copy here would be a second place for it to drift, and the blob channel is exactly where a
+   * drift would leak one fleet's memory into another's.
+   *
+   * @param {{machine_id?: string, fleet_id?: string|null}|null} device
+   * @returns {{ok:true, fleetId:string|null, legacy:boolean}|{ok:false, reason:string, text:string}}
+   */
+  scopeBlobFleet(device) {
+    return scopeFleetId({ device, relayFleetId: this.fleetId ?? null });
+  }
+
+  /**
+   * Store bytes under their SHA-256, owned by the writer's fleet.
+   *
+   * Delegates to `Persistence` -- this class never touches the filesystem (v0.1.2 §7), and that constraint is what
+   * keeps the durable path in one place.
+   *
+   * @param {object} device the authenticated device
+   * @param {Buffer} bytes
+   */
+  putFleetBlob(device, bytes) {
+    if (!this._persist) {
+      return { ok: false, reason: 'PERSISTENCE_DISABLED', detail: 'the relay has no persistence backend' };
+    }
+    const scope = this.scopeBlobFleet(device);
+    if (scope.ok !== true) return { ok: false, reason: scope.reason, detail: scope.text };
+    return this._persist.putBlob(bytes, { fleetId: scope.fleetId });
+  }
+
+  /**
+   * The bytes for a hash, or `null`.
+   *
+   * `null` covers three cases on purpose -- no such blob, a blob owned by another fleet, and no persistence -- so a
+   * caller cannot use the relay to enumerate which hashes exist elsewhere on it.
+   *
+   * @param {object} device the authenticated device
+   * @param {string} hash
+   */
+  getFleetBlob(device, hash) {
+    if (!this._persist) return null;
+    const scope = this.scopeBlobFleet(device);
+    if (scope.ok !== true) return null;
+    return this._persist.getBlob(hash, { fleetId: scope.fleetId });
+  }
+
   /** Snapshot the device table; failures must never break the request path. */
   _saveDevices() {
     if (this._restoring || !this._persist) return false;