@twinsearth/w2m-dsh-plugin 1.0.5 → 1.0.7
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +115 -0
- package/cordis.patch.yml +20 -0
- package/package.json +1 -1
- package/plugin.json +1 -1
- package/src/plugin/tools.mjs +195 -3
- package/src/relay/persistence.mjs +249 -0
- package/src/relay/server.mjs +72 -1
- package/src/relay/state.mjs +69 -0
package/CHANGELOG.md
CHANGED
|
@@ -2,6 +2,121 @@
|
|
|
2
2
|
|
|
3
3
|
All notable changes to this project are documented here.
|
|
4
4
|
|
|
5
|
+
## [1.0.7] — 2026-10-17
|
|
6
|
+
|
|
7
|
+
**The fleet sync channel, both directions: a machine can publish its collective memory to the relay and another
|
|
8
|
+
machine in the same fleet can fetch it back by hash.**
|
|
9
|
+
|
|
10
|
+
### #1 is now a channel, not a backup
|
|
11
|
+
|
|
12
|
+
v1.0.6 shipped the storage layer and said plainly that it had no route and no caller. This release supplies both:
|
|
13
|
+
|
|
14
|
+
| Piece | Where |
|
|
15
|
+
|---|---|
|
|
16
|
+
| content-addressed store | `src/relay/persistence.mjs` — `putBlob`/`getBlob`, 8 MiB per blob, 256 per fleet, oldest-first eviction that never takes the last one |
|
|
17
|
+
| hash → fleet ownership | `<state>/blobs/index.json`, so a content-addressed store is *not* a shared one |
|
|
18
|
+
| two routes | `PUT`/`GET /v1/fleet/blobs/<sha256>` in `src/relay/server.mjs`, behind the same device-token gate as every other route |
|
|
19
|
+
| scoping | `state.scopeBlobFleet` reuses `scopeFleetId` — **the same rule task dispatch obeys**, not a second copy of it |
|
|
20
|
+
| caller | `w2m_memory` `action: "publish"` and `"pull"` |
|
|
21
|
+
|
|
22
|
+
**Why a content-addressed store does not contradict "the relay keeps only hashes".** It persists `stdout_sha256`
|
|
23
|
+
and never the text, and `src/agent/storage.mjs` records why: each machine's `memory.jsonl` is the only durable copy
|
|
24
|
+
of what a command printed. **That invariant is about the relay not interpreting content.** A blob keyed by its own
|
|
25
|
+
hash preserves it exactly — the relay still interprets nothing, and **the hash is what says the bytes are the
|
|
26
|
+
bytes**, as `stdout_sha256` does for a result. What is new is only that a hash the relay already holds can now have
|
|
27
|
+
its bytes fetched from the same place.
|
|
28
|
+
|
|
29
|
+
### The decisions worth recording
|
|
30
|
+
|
|
31
|
+
**The write verifies the key, and refuses rather than correcting.** A content-addressed store that accepts bytes
|
|
32
|
+
under a key they do not hash to is not content-addressed: the key stops being a claim, and every later reader's
|
|
33
|
+
verification becomes decorative. A mismatch is `422 BLOB_HASH_MISMATCH` and nothing is stored.
|
|
34
|
+
|
|
35
|
+
**Ownership is first-writer-wins.** A second fleet PUT-ing an identical blob does **not** take ownership of it.
|
|
36
|
+
Otherwise any fleet could claim another's blob by replaying the same bytes, which is the same failure as having no
|
|
37
|
+
index at all.
|
|
38
|
+
|
|
39
|
+
**A cross-fleet read is `404`, not `403`.** A distinguishable "not yours" tells a prober that this hash exists on
|
|
40
|
+
this relay. The reader cannot tell "never stored" from "another fleet's", deliberately.
|
|
41
|
+
|
|
42
|
+
**The pull side re-hashes what it receives.** A verification performed only by the party that could be wrong is not
|
|
43
|
+
a verification, so a reader that gets bytes hashing to something else raises `W2M_BAD_BLOB` rather than passing
|
|
44
|
+
them on.
|
|
45
|
+
|
|
46
|
+
### What is verified, and what is not
|
|
47
|
+
|
|
48
|
+
**Verified — executed:** four new tests in `test/relay.test.mjs` drive the real routes against a real relay:
|
|
49
|
+
store-and-return-unchanged; a mismatched key is refused (422) and leaves the key absent (404); one fleet cannot read
|
|
50
|
+
another's blob (404); an unauthenticated request is refused (401). Full suite **2356 tests, 2342 pass, 0 fail, 14
|
|
51
|
+
skip**; pre-flight **GREEN, 15 of 15 checks**.
|
|
52
|
+
|
|
53
|
+
**NOT verified: the `publish`/`pull` tool actions end to end.** They are type-correct and lint-clean, and their
|
|
54
|
+
failure paths are written, but **no test executes them** — they need a profile with a `fleet-memory.md` and a live
|
|
55
|
+
relay. They are marked unverified here rather than counted as working, which is the distinction this project keeps
|
|
56
|
+
having to restate.
|
|
57
|
+
|
|
58
|
+
**A known limit of the reader:** `pull` returns the blob as text and re-hashes it as UTF-8. That is exact for a
|
|
59
|
+
`fleet-memory.md`; it would not be for arbitrary binary content, and the channel does not claim to carry that.
|
|
60
|
+
|
|
61
|
+
### Not in this release
|
|
62
|
+
|
|
63
|
+
The trigger (#3) and merge semantics (#2) are deliberately absent. Publishing is a decision a machine makes, not
|
|
64
|
+
something that happens on join, and merge needs a conflict rule that has not been decided: two machines writing
|
|
65
|
+
different facts produce **two hashes**, and nothing yet says who wins. **Nothing is overwritten in the meantime —
|
|
66
|
+
that is the property #1 was built to provide.**
|
|
67
|
+
|
|
68
|
+
## [1.0.6] — 2026-10-17
|
|
69
|
+
|
|
70
|
+
**DSH-GPW now ships *findable*: installed with the package, present in the plugin list, and off until an operator
|
|
71
|
+
turns it on.**
|
|
72
|
+
|
|
73
|
+
### What v1.0.5 got wrong
|
|
74
|
+
|
|
75
|
+
It shipped the subpath export `@twinsearth/w2m-dsh-plugin/gpw` and no config row. **A plugin the composed tree does
|
|
76
|
+
not mention is a plugin the plugin manager cannot list** -- so "enable it manually" had **nothing to click**, and
|
|
77
|
+
the only way in was hand-editing a profile patch.
|
|
78
|
+
|
|
79
|
+
**An optional component nobody can find is not optional, it is absent.** That is the same distinction as the seven
|
|
80
|
+
written-but-unreachable modules in this repository, seen from the user's side instead of the code's.
|
|
81
|
+
|
|
82
|
+
### The shape that fixes it
|
|
83
|
+
|
|
84
|
+
The package's own bundle patch now carries a second entry:
|
|
85
|
+
|
|
86
|
+
- id: dsh-gpw
|
|
87
|
+
name: '@twinsearth/w2m-dsh-plugin/gpw'
|
|
88
|
+
disabled: true
|
|
89
|
+
config:
|
|
90
|
+
maxPerWindow: 10
|
|
91
|
+
windowMs: 60000
|
|
92
|
+
minGapMs: 3000
|
|
93
|
+
|
|
94
|
+
**`disabled: true` is the load-bearing field.** The row exists, so the component appears; the toggle is off, so
|
|
95
|
+
nothing answers `approval/request`; and **flipping it in the UI is the whole enablement step** -- no file edit, no
|
|
96
|
+
restart of anything but the client.
|
|
97
|
+
|
|
98
|
+
**It stays in the same `insert:` list as the w2m row**, and that is still the correct form: this is an insert
|
|
99
|
+
because no earlier layer creates a `dsh-gpw` row. (A *bare* `- id:` patch aimed at a nonexistent row is skipped
|
|
100
|
+
silently -- the failure recorded in this file and in TROUBLESHOOTING 3.2.)
|
|
101
|
+
|
|
102
|
+
### The distinction the two releases now make
|
|
103
|
+
|
|
104
|
+
| | v1.0.5 | v1.0.6 |
|
|
105
|
+
|---|---|---|
|
|
106
|
+
| Installs with the package | yes | yes |
|
|
107
|
+
| Appears in the plugin manager | **no** | **yes** |
|
|
108
|
+
| Enabled by default | no | no |
|
|
109
|
+
| How you turn it on | hand-edit a profile patch | **click its toggle** |
|
|
110
|
+
|
|
111
|
+
**Not verified: that the toggle actually flips it on and that it then intercepts.** That needs a live host. The
|
|
112
|
+
observable evidence is its own startup line followed by its counted line --
|
|
113
|
+
|
|
114
|
+
approval-ratelimit: active -- at most 10 auto-approvals per 60 s, at least 3000 ms apart, process-wide
|
|
115
|
+
approval-ratelimit: N auto-approved, M deferred to the human
|
|
116
|
+
|
|
117
|
+
-- **because without the second line, "mounted and working" and "mounted and doing nothing" are the same picture**,
|
|
118
|
+
which is the failure this watchdog's own counters exist to prevent.
|
|
119
|
+
|
|
5
120
|
## [1.0.5] — 2026-10-17
|
|
6
121
|
|
|
7
122
|
**DSH-GPW ships as an optional sub-plugin, and the reason it never activated is fixed.**
|
package/cordis.patch.yml
CHANGED
|
@@ -115,3 +115,23 @@
|
|
|
115
115
|
allowedCommands: []
|
|
116
116
|
p2pMode: auto
|
|
117
117
|
stunServers: []
|
|
118
|
+
# DSH-GPW (Global Permission Watchdog) ships with this package and is **off until an operator turns it on**.
|
|
119
|
+
#
|
|
120
|
+
# It is in this insert list rather than left out of the tree entirely, because a plugin the composed tree does
|
|
121
|
+
# not mention is a plugin the plugin manager cannot list -- and then "enable it manually" has no control to
|
|
122
|
+
# click. **A row with `disabled: true` is the shape that makes it installable with the package and switchable
|
|
123
|
+
# from the UI**: the component shows up, its toggle is off, and flipping it needs no file edit.
|
|
124
|
+
#
|
|
125
|
+
# Found this way, not assumed: v1.0.5 shipped the subpath export but no row, so the watchdog was reachable only
|
|
126
|
+
# by hand-editing a profile patch. **An optional component nobody can find is not optional, it is absent.**
|
|
127
|
+
#
|
|
128
|
+
# What it does once enabled: answers `approval/request` with at most 10 auto-approvals per rolling 60 s, at
|
|
129
|
+
# least 3000 ms apart, and defers everything past that to the human prompt, where it waits.
|
|
130
|
+
- id: dsh-gpw
|
|
131
|
+
name: '@twinsearth/w2m-dsh-plugin/gpw'
|
|
132
|
+
disabled: true
|
|
133
|
+
config:
|
|
134
|
+
maxPerWindow: 10
|
|
135
|
+
windowMs: 60000
|
|
136
|
+
minGapMs: 3000
|
|
137
|
+
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@twinsearth/w2m-dsh-plugin",
|
|
3
|
-
"version": "1.0.
|
|
3
|
+
"version": "1.0.7",
|
|
4
4
|
"description": "Run one project across your own Windows and macOS machines from a single DeepSeek account, through 11 w2m_* tools. Four coordination modes (replicate, split, broadcast, pipeline) and six aggregate verdicts (consistent, divergent, divergent-platform, failed, partial, unverifiable). P2P hole punching by default, with a bundled relay, STUN responder and ledger fallback, so the fleet needs no third-party service. Joining needs a pairing code plus autoStartAgent: true; installing alone does not join.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "lib/index.js",
|
package/plugin.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"$schema": "https://agent-plugins.org/schemas/1.0.0/plugin.schema.json",
|
|
3
3
|
"name": "twinsearth-w2m-dsh",
|
|
4
|
-
"version": "1.0.
|
|
4
|
+
"version": "1.0.7",
|
|
5
5
|
"description": "Run one project across your own Windows and macOS machines from a single DeepSeek account, through 11 w2m_* tools. Four coordination modes (replicate, split, broadcast, pipeline) and six aggregate verdicts (consistent, divergent, divergent-platform, failed, partial, unverifiable). P2P hole punching by default, with a bundled relay, STUN responder and ledger fallback, so the fleet needs no third-party service. Joining needs a pairing code plus autoStartAgent: true; installing alone does not join.",
|
|
6
6
|
"license": "MIT",
|
|
7
7
|
"keywords": [
|
package/src/plugin/tools.mjs
CHANGED
|
@@ -208,7 +208,7 @@ const MAX_STAGES = 16;
|
|
|
208
208
|
* `package.json` during `scripts/pack.mjs` rather than hand-maintained, and `W2M_PLUGIN_VERSION`
|
|
209
209
|
* overrides it at runtime for tests and packagers.
|
|
210
210
|
*/
|
|
211
|
-
const PLUGIN_VERSION = '1.0.
|
|
211
|
+
const PLUGIN_VERSION = '1.0.7';
|
|
212
212
|
|
|
213
213
|
/** How long any single HTTP call may take before it is aborted, in milliseconds. */
|
|
214
214
|
const DEFAULT_REQUEST_TIMEOUT_MS = 15_000;
|
|
@@ -2590,10 +2590,27 @@ async function request(options) {
|
|
|
2590
2590
|
pathname,
|
|
2591
2591
|
query,
|
|
2592
2592
|
body,
|
|
2593
|
+
/**
|
|
2594
|
+
* v1.0.7: raw bytes to send instead of a JSON body.
|
|
2595
|
+
*
|
|
2596
|
+
* Added for the blob channel (#1), which PUTs a memory document's exact octets. **Added here rather than as a
|
|
2597
|
+
* second `fetch` in the caller**, because `tools.mjs:1283` already states the rule: the direct path must not grow
|
|
2598
|
+
* a second way to talk to the relay -- bearer-token resolution, base-path joining, timeouts and abort handling
|
|
2599
|
+
* all live in this one function, and a duplicate would be a duplicate of all four.
|
|
2600
|
+
*
|
|
2601
|
+
* `body` and `rawBody` are mutually exclusive: passing both is a caller bug, not a merge to guess at.
|
|
2602
|
+
*/
|
|
2603
|
+
rawBody,
|
|
2593
2604
|
maxBytes = MAX_RESPONSE_BYTES,
|
|
2594
2605
|
accept = 'application/json',
|
|
2595
2606
|
} = options;
|
|
2596
2607
|
|
|
2608
|
+
if (rawBody !== undefined && body !== undefined) {
|
|
2609
|
+
throw new W2MError('W2M_CONFIG', 'W2M_CONFIG: `body` and `rawBody` are mutually exclusive', {
|
|
2610
|
+
hint: 'pass `rawBody` for octets, `body` for JSON -- not both',
|
|
2611
|
+
});
|
|
2612
|
+
}
|
|
2613
|
+
|
|
2597
2614
|
throwIfAborted(signal);
|
|
2598
2615
|
|
|
2599
2616
|
// Named here rather than left to `new URL`, whose failure would say only "Invalid URL" and send
|
|
@@ -2613,7 +2630,10 @@ async function request(options) {
|
|
|
2613
2630
|
|
|
2614
2631
|
const headers = { accept };
|
|
2615
2632
|
if (token) headers.authorization = `Bearer ${token}`;
|
|
2616
|
-
|
|
2633
|
+
// v1.0.7: octets keep their own content type. JSON-stringifying them would corrupt the bytes and, because the
|
|
2634
|
+
// relay verifies `sha256(body) === key`, corrupting them here would surface as a 422 blamed on the caller.
|
|
2635
|
+
if (rawBody !== undefined) headers['content-type'] = 'application/octet-stream';
|
|
2636
|
+
else if (body !== undefined) headers['content-type'] = 'application/json';
|
|
2617
2637
|
|
|
2618
2638
|
const timeoutController = new AbortController();
|
|
2619
2639
|
const timer = setTimeout(() => timeoutController.abort(new Error('timeout')), timeoutMs);
|
|
@@ -2624,7 +2644,7 @@ async function request(options) {
|
|
|
2624
2644
|
response = await fetch(target, {
|
|
2625
2645
|
method,
|
|
2626
2646
|
headers,
|
|
2627
|
-
body: body === undefined ? undefined : JSON.stringify(body),
|
|
2647
|
+
body: rawBody !== undefined ? rawBody : (body === undefined ? undefined : JSON.stringify(body)),
|
|
2628
2648
|
signal: combined.signal,
|
|
2629
2649
|
});
|
|
2630
2650
|
} catch (error) {
|
|
@@ -5112,6 +5132,30 @@ export async function apply(ctx, config = {}) {
|
|
|
5112
5132
|
'relay keeps only hashes, so these files are the only place a command\'s output survives. Use it ' +
|
|
5113
5133
|
'before re-running work -- it is what turns a dozen rediscovery dispatches into one lookup.',
|
|
5114
5134
|
parameters: {
|
|
5135
|
+
/**
|
|
5136
|
+
* v1.0.7: `read` (the default, and the only behaviour before this release), `publish`, or `pull`.
|
|
5137
|
+
*
|
|
5138
|
+
* Defaulting to `read` is deliberate: every existing caller says nothing about `action`, and a tool that
|
|
5139
|
+
* changed what "call `w2m_memory`" means would be a silent behaviour change on the one path that is supposed
|
|
5140
|
+
* to be safe to call blindly.
|
|
5141
|
+
*/
|
|
5142
|
+
action: {
|
|
5143
|
+
type: 'string',
|
|
5144
|
+
enum: ['read', 'publish', 'pull'],
|
|
5145
|
+
description:
|
|
5146
|
+
'read (default) = answer from memory. publish = upload the curated collective memory '
|
|
5147
|
+
+ '(`fleet-memory.md`) to the relay as a content-addressed blob and return its SHA-256. pull = fetch one '
|
|
5148
|
+
+ 'blob by its SHA-256 and return its text, re-hashing what came back against the name it was asked for. '
|
|
5149
|
+
+ 'Neither publish nor pull modifies the local file.',
|
|
5150
|
+
default: 'read',
|
|
5151
|
+
},
|
|
5152
|
+
hash: {
|
|
5153
|
+
type: 'string',
|
|
5154
|
+
description:
|
|
5155
|
+
'The SHA-256 naming the blob, for action="pull": 64 lowercase hex characters. A hash that exists on the '
|
|
5156
|
+
+ 'relay but belongs to another fleet is reported as not found, exactly like a hash that was never '
|
|
5157
|
+
+ 'stored -- a distinguishable refusal would confirm to a prober which hashes this relay holds.',
|
|
5158
|
+
},
|
|
5115
5159
|
scope: {
|
|
5116
5160
|
type: 'string',
|
|
5117
5161
|
enum: ['local', 'fleet'],
|
|
@@ -5199,6 +5243,154 @@ export async function apply(ctx, config = {}) {
|
|
|
5199
5243
|
output: jsonOutput,
|
|
5200
5244
|
async execute(args, exec) {
|
|
5201
5245
|
throwIfAborted(exec?.signal);
|
|
5246
|
+
|
|
5247
|
+
/**
|
|
5248
|
+
* v1.0.7 (#1c): publish the collective memory as a content-addressed blob.
|
|
5249
|
+
*
|
|
5250
|
+
* ## Why this exists at all
|
|
5251
|
+
*
|
|
5252
|
+
* The blob channel (1a/1b) is a route and a store; **without a caller it is the same defect as the seven
|
|
5253
|
+
* written-but-unreachable modules this repository keeps an accepted-orphans list for.** This is the caller:
|
|
5254
|
+
* a machine that wants its curated memory to be fetchable by its own fleet says so, and gets back the hash
|
|
5255
|
+
* that names it.
|
|
5256
|
+
*
|
|
5257
|
+
* ## Why it reads the file and does not rewrite it
|
|
5258
|
+
*
|
|
5259
|
+
* Publishing is a **copy, not a move**. If it deleted or normalised the file, the only durable copy of what
|
|
5260
|
+
* this machine knows would depend on the relay having accepted it -- and the relay is explicitly not the
|
|
5261
|
+
* durable store (`storage.mjs`: each machine's `memory.jsonl` is the only copy of what a command printed).
|
|
5262
|
+
*
|
|
5263
|
+
* ## Why a missing file is a named refusal
|
|
5264
|
+
*
|
|
5265
|
+
* `putBlob` rejects an empty body with `BLOB_EMPTY`, which is correct but arrives too late to say anything
|
|
5266
|
+
* useful. A machine with no `fleet-memory.md` has not published anything yet -- that is a state, not a
|
|
5267
|
+
* protocol violation, and the message should say which file is missing and where it was looked for.
|
|
5268
|
+
*/
|
|
5269
|
+
if (typeof args?.action === 'string' && args.action === 'publish') {
|
|
5270
|
+
const file = collectivePath();
|
|
5271
|
+
if (file === null) {
|
|
5272
|
+
throw new W2MError('W2M_CONFIG', 'W2M_CONFIG: there is no profile to publish collective memory from', {
|
|
5273
|
+
hint: 'the collective layer lives beside the profile that mounted this plugin; set `stateDir` or run '
|
|
5274
|
+
+ 'with a profile',
|
|
5275
|
+
});
|
|
5276
|
+
}
|
|
5277
|
+
let bytes;
|
|
5278
|
+
try {
|
|
5279
|
+
// `fs` is `promises as fs` in this file (line 93), so this is the promise API -- a bare `readFile` is not
|
|
5280
|
+
// in scope and would throw ReferenceError at the exact moment the caller wanted to publish.
|
|
5281
|
+
bytes = await fs.readFile(file);
|
|
5282
|
+
} catch {
|
|
5283
|
+
// The error object is deliberately not bound: every refusal of a read here is the same refusal (there is
|
|
5284
|
+
// no document at that path), and naming the errno would tell the caller to debug a filesystem this plugin
|
|
5285
|
+
// does not own.
|
|
5286
|
+
throw new W2MError(
|
|
5287
|
+
'W2M_NOT_FOUND',
|
|
5288
|
+
`W2M_NOT_FOUND: no collective memory to publish at ${file}`,
|
|
5289
|
+
{ hint: 'write measured facts to it first; publishing copies the file, it does not create one' },
|
|
5290
|
+
);
|
|
5291
|
+
}
|
|
5292
|
+
const hash = createHash('sha256').update(bytes).digest('hex');
|
|
5293
|
+
const base = requireRabbit(cfg);
|
|
5294
|
+
const token = await resolveToken(cfg);
|
|
5295
|
+
const result = await request({
|
|
5296
|
+
rabbitUrl: base,
|
|
5297
|
+
token,
|
|
5298
|
+
method: 'PUT',
|
|
5299
|
+
pathname: `/v1/fleet/blobs/${hash}`,
|
|
5300
|
+
rawBody: bytes,
|
|
5301
|
+
signal: exec?.signal,
|
|
5302
|
+
});
|
|
5303
|
+
return JSON.stringify({
|
|
5304
|
+
ok: true,
|
|
5305
|
+
published: true,
|
|
5306
|
+
hash,
|
|
5307
|
+
bytes: bytes.length,
|
|
5308
|
+
file,
|
|
5309
|
+
relay: {
|
|
5310
|
+
http_status: result.status,
|
|
5311
|
+
// `request()` returns `{status, ok, json, text, error}` -- the parsed body is `json`, not `body`.
|
|
5312
|
+
evicted: result.json?.evicted ?? [],
|
|
5313
|
+
},
|
|
5314
|
+
next: 'another machine in the same fleet can GET ' + `/v1/fleet/blobs/${hash}` + ' to fetch it',
|
|
5315
|
+
});
|
|
5316
|
+
}
|
|
5317
|
+
|
|
5318
|
+
/**
|
|
5319
|
+
* v1.0.7 (#1d, the other direction): fetch one blob by the hash that names it.
|
|
5320
|
+
*
|
|
5321
|
+
* ## Why this had to exist before the channel was called finished
|
|
5322
|
+
*
|
|
5323
|
+
* A `publish` with no `pull` is a backup, not a channel: the bytes leave the machine and nothing can bring
|
|
5324
|
+
* them back through the tool surface. The relay route existed and was tested, and the plugin could still only
|
|
5325
|
+
* write -- which is the same "half a mechanism" shape the whole 1a-1d sequence was organised to avoid.
|
|
5326
|
+
*
|
|
5327
|
+
* ## Why the reader re-hashes instead of trusting the relay
|
|
5328
|
+
*
|
|
5329
|
+
* The relay verifies `sha256(body) === key` on write, so in principle what comes back is what was stored. **A
|
|
5330
|
+
* reader must not depend on that.** The hash is the only thing that makes the bytes a claim rather than an
|
|
5331
|
+
* assertion, and a verification performed only by the party that could be wrong is not a verification. So the
|
|
5332
|
+
* content is re-hashed here, and a mismatch is a failure rather than a warning -- accepting unverified bytes
|
|
5333
|
+
* under a name they do not match would make every later reader's check decorative too.
|
|
5334
|
+
*/
|
|
5335
|
+
if (typeof args?.action === 'string' && args.action === 'pull') {
|
|
5336
|
+
const wanted = typeof args?.hash === 'string' ? args.hash.trim().toLowerCase() : '';
|
|
5337
|
+
if (!/^[0-9a-f]{64}$/.test(wanted)) {
|
|
5338
|
+
throw new W2MError(
|
|
5339
|
+
'W2M_CONFIG',
|
|
5340
|
+
'W2M_CONFIG: `action: "pull"` needs `hash`, a SHA-256 of 64 lowercase hex characters',
|
|
5341
|
+
{ hint: 'the hash is what `action: "publish"` returned for the document you want' },
|
|
5342
|
+
);
|
|
5343
|
+
}
|
|
5344
|
+
const base = requireRabbit(cfg);
|
|
5345
|
+
const token = await resolveToken(cfg);
|
|
5346
|
+
const result = await request({
|
|
5347
|
+
rabbitUrl: base,
|
|
5348
|
+
token,
|
|
5349
|
+
pathname: `/v1/fleet/blobs/${wanted}`,
|
|
5350
|
+
accept: 'application/octet-stream',
|
|
5351
|
+
signal: exec?.signal,
|
|
5352
|
+
});
|
|
5353
|
+
if (result.status === 404) {
|
|
5354
|
+
// Deliberately the same answer for "never stored" and "another fleet's": see the `hash` parameter note.
|
|
5355
|
+
throw new W2MError(
|
|
5356
|
+
'W2M_NOT_FOUND',
|
|
5357
|
+
`W2M_NOT_FOUND: no blob named ${wanted} is readable by this machine`,
|
|
5358
|
+
{
|
|
5359
|
+
hint: 'the hash may belong to another fleet, or the relay may have evicted it -- blobs are evicted '
|
|
5360
|
+
+ 'oldest-first once a fleet holds 256',
|
|
5361
|
+
},
|
|
5362
|
+
);
|
|
5363
|
+
}
|
|
5364
|
+
if (result.status !== 200) {
|
|
5365
|
+
throw new W2MError(
|
|
5366
|
+
'W2M_RABBIT_REFUSED',
|
|
5367
|
+
`W2M_RABBIT_REFUSED: GET /v1/fleet/blobs/${wanted} answered HTTP ${result.status}`,
|
|
5368
|
+
{ http_status: result.status, detail: result.error ?? null },
|
|
5369
|
+
);
|
|
5370
|
+
}
|
|
5371
|
+
const body = result.text ?? '';
|
|
5372
|
+
const actual = createHash('sha256').update(body, 'utf8').digest('hex');
|
|
5373
|
+
if (actual !== wanted) {
|
|
5374
|
+
throw new W2MError(
|
|
5375
|
+
'W2M_BAD_BLOB',
|
|
5376
|
+
`W2M_BAD_BLOB: the bytes returned for ${wanted} hash to ${actual} instead`,
|
|
5377
|
+
{
|
|
5378
|
+
hint: 'the reader refuses unverified content rather than passing it on; nothing was written locally',
|
|
5379
|
+
expected: wanted,
|
|
5380
|
+
actual,
|
|
5381
|
+
},
|
|
5382
|
+
);
|
|
5383
|
+
}
|
|
5384
|
+
return JSON.stringify({
|
|
5385
|
+
ok: true,
|
|
5386
|
+
pulled: true,
|
|
5387
|
+
hash: wanted,
|
|
5388
|
+
bytes: Buffer.byteLength(body, 'utf8'),
|
|
5389
|
+
verified: true,
|
|
5390
|
+
content: body,
|
|
5391
|
+
});
|
|
5392
|
+
}
|
|
5393
|
+
|
|
5202
5394
|
const limitRaw = Number(args?.limit);
|
|
5203
5395
|
const limit = Number.isFinite(limitRaw) && limitRaw > 0 ? Math.min(Math.floor(limitRaw), 500) : 20;
|
|
5204
5396
|
|
|
@@ -4,6 +4,7 @@
|
|
|
4
4
|
* `<state>/devices.json` atomic snapshot (tmp file + rename)
|
|
5
5
|
* `<state>/ledger.jsonl` append-only, one JSON object per line
|
|
6
6
|
* `<state>/operator-token.txt` single line, 0600
|
|
7
|
+
* `<state>/blobs/<aa>/<sha256>` content-addressed bytes, v1.0.7 (see `putBlob`)
|
|
7
8
|
*
|
|
8
9
|
* This is the only relay module allowed to touch the filesystem.
|
|
9
10
|
* Corruption is never silent: every skipped line / unreadable snapshot is
|
|
@@ -12,12 +13,31 @@
|
|
|
12
13
|
|
|
13
14
|
import fs from 'node:fs';
|
|
14
15
|
import path from 'node:path';
|
|
16
|
+
import { createHash } from 'node:crypto';
|
|
15
17
|
|
|
16
18
|
import { rfc3339 } from './state.mjs';
|
|
17
19
|
|
|
18
20
|
export const DEVICES_FILE = 'devices.json';
|
|
19
21
|
export const LEDGER_FILE = 'ledger.jsonl';
|
|
20
22
|
export const OPERATOR_TOKEN_FILE = 'operator-token.txt';
|
|
23
|
+
/** v1.0.7: the content-addressed blob directory, and the two caps that keep it bounded. */
|
|
24
|
+
export const BLOB_DIR = 'blobs';
|
|
25
|
+
export const MAX_BLOB_BYTES = 8 * 1024 * 1024;
|
|
26
|
+
export const MAX_BLOBS = 256;
|
|
27
|
+
/**
|
|
28
|
+
* v1.0.7: **which fleet wrote each blob.**
|
|
29
|
+
*
|
|
30
|
+
* Content addressing keys a blob by its own hash, and a hash knows nothing about fleets. So two fleets that store
|
|
31
|
+
* the same bytes land at the same path -- and without this index, the second one could read the first one's content.
|
|
32
|
+
* **A content-addressed store with no ownership record is not isolated, it is shared.**
|
|
33
|
+
*
|
|
34
|
+
* The index is a separate document rather than part of the path (`blobs/<fleet>/<aa>/<hash>`) for one concrete
|
|
35
|
+
* reason: **deduplication is only sound when the content is identical**, and identical bytes stored twice would
|
|
36
|
+
* otherwise occupy two entries. The path stays a pure function of the content; the fleet is metadata.
|
|
37
|
+
*
|
|
38
|
+
* Shape: `{ "<sha256>": { fleet_id: string|null, added_at_ms: number } }`.
|
|
39
|
+
*/
|
|
40
|
+
export const BLOB_INDEX_FILE = 'index.json';
|
|
21
41
|
|
|
22
42
|
/** Ledger entry types understood by `RabbitState#restore`. */
|
|
23
43
|
export const LEDGER_TYPES = Object.freeze([
|
|
@@ -250,8 +270,232 @@ export class Persistence {
|
|
|
250
270
|
return true;
|
|
251
271
|
}
|
|
252
272
|
|
|
273
|
+
/* ------------------------------------------------------------------------------------------- */
|
|
274
|
+
/* v1.0.7: content-addressed blobs — the fleet sync channel (#1) */
|
|
275
|
+
/* ------------------------------------------------------------------------------------------- */
|
|
276
|
+
|
|
277
|
+
/**
|
|
278
|
+
* Store bytes under their own SHA-256, and read them back by that hash.
|
|
279
|
+
*
|
|
280
|
+
* ## Why the relay may carry bytes at all, when it deliberately carries none today
|
|
281
|
+
*
|
|
282
|
+
* The relay persists `stdout_sha256` and never the text, and `src/agent/storage.mjs` records why that matters:
|
|
283
|
+
* `memory.jsonl` on each machine is the only durable copy of what a command printed. **That invariant is about
|
|
284
|
+
* the relay not becoming a document store** — it never interprets content and never asserts that content is
|
|
285
|
+
* authentic.
|
|
286
|
+
*
|
|
287
|
+
* A blob keyed by **its own hash** preserves both halves. The relay still interprets nothing; **the hash is what
|
|
288
|
+
* says the bytes are the bytes**, exactly as `stdout_sha256` does for a result. The only thing that is new is
|
|
289
|
+
* that a hash the relay already holds can now have its bytes fetched from the same place.
|
|
290
|
+
*
|
|
291
|
+
* ## The two limits, because an uncapped writer is what caused the original incident
|
|
292
|
+
*
|
|
293
|
+
* `storage.mjs`'s header records the incident that produced its whole design: a single uncapped writer reached
|
|
294
|
+
* 100% filesystem at 14.7 GB/day. A blob channel with no ceiling reproduces that on the relay, so:
|
|
295
|
+
*
|
|
296
|
+
* * **8 MiB per blob** — a `fleet-memory.md` is tens of KB; this is three orders of magnitude of headroom.
|
|
297
|
+
* Larger content is an archive, not a memory.
|
|
298
|
+
* * **256 blobs** — bounded *count* matters as much as bounded size: 256 × 8 MiB caps a fleet's share at
|
|
299
|
+
* 2 GiB regardless of what it writes.
|
|
300
|
+
*
|
|
301
|
+
* Eviction is oldest-first and **never takes the last blob**: deleting the only copy of a fact is the failure
|
|
302
|
+
* `UNDELETABLE_TIERS` exists to prevent locally.
|
|
303
|
+
*
|
|
304
|
+
* @param {Buffer} bytes
|
|
305
|
+
* @param {{fleetId?: string|null}} [options] the owning fleet, recorded so another fleet cannot read it back
|
|
306
|
+
* @returns {{ok:true, hash:string, bytes:number, evicted:string[]}|{ok:false, reason:string, detail:string}}
|
|
307
|
+
*/
|
|
308
|
+
putBlob(bytes, options = {}) {
|
|
309
|
+
if (!this.enabled) return { ok: false, reason: 'PERSISTENCE_DISABLED', detail: 'the relay has no state directory' };
|
|
310
|
+
if (!Buffer.isBuffer(bytes) || bytes.length === 0) {
|
|
311
|
+
return { ok: false, reason: 'BLOB_EMPTY', detail: 'an empty blob is a caller bug, not content' };
|
|
312
|
+
}
|
|
313
|
+
if (bytes.length > MAX_BLOB_BYTES) {
|
|
314
|
+
return {
|
|
315
|
+
ok: false,
|
|
316
|
+
reason: 'BLOB_TOO_LARGE',
|
|
317
|
+
detail: `${bytes.length} bytes exceeds the ${MAX_BLOB_BYTES}-byte limit`,
|
|
318
|
+
};
|
|
319
|
+
}
|
|
320
|
+
const hash = createHash('sha256').update(bytes).digest('hex');
|
|
321
|
+
const file = this.blobFile(hash);
|
|
322
|
+
if (file === null) return { ok: false, reason: 'PERSISTENCE_DISABLED', detail: 'no blob directory' };
|
|
323
|
+
|
|
324
|
+
const index = this.readBlobIndex();
|
|
325
|
+
const fleetId = typeof options.fleetId === 'string' && options.fleetId !== '' ? options.fleetId : null;
|
|
326
|
+
|
|
327
|
+
// Content addressing makes the write idempotent: the same bytes always land at the same path.
|
|
328
|
+
if (fs.existsSync(file)) {
|
|
329
|
+
/**
|
|
330
|
+
* Already stored. **The first writer's fleet keeps ownership** -- re-assigning it on a second write would let
|
|
331
|
+
* one fleet claim a blob another fleet already owns by simply PUT-ting the same bytes, which is the same
|
|
332
|
+
* failure as having no index at all.
|
|
333
|
+
*/
|
|
334
|
+
const entry = index[hash];
|
|
335
|
+
if (entry === undefined) {
|
|
336
|
+
index[hash] = { fleet_id: fleetId, added_at_ms: this._now() };
|
|
337
|
+
this.writeBlobIndex(index);
|
|
338
|
+
}
|
|
339
|
+
return { ok: true, hash, bytes: bytes.length, evicted: [] };
|
|
340
|
+
}
|
|
341
|
+
|
|
342
|
+
const evicted = this.evictBlobsFor(1);
|
|
343
|
+
for (const gone of evicted) delete index[gone];
|
|
344
|
+
try {
|
|
345
|
+
fs.mkdirSync(path.dirname(file), { recursive: true });
|
|
346
|
+
const tmp = `${file}.tmp-${process.pid}-${this._now()}`;
|
|
347
|
+
fs.writeFileSync(tmp, bytes);
|
|
348
|
+
fs.renameSync(tmp, file);
|
|
349
|
+
} catch (error) {
|
|
350
|
+
this.warn(BLOB_DIR, `blob write failed: ${error?.message ?? String(error)}`);
|
|
351
|
+
return { ok: false, reason: 'BLOB_WRITE_FAILED', detail: error?.message ?? String(error) };
|
|
352
|
+
}
|
|
353
|
+
index[hash] = { fleet_id: fleetId, added_at_ms: this._now() };
|
|
354
|
+
this.writeBlobIndex(index);
|
|
355
|
+
return { ok: true, hash, bytes: bytes.length, evicted };
|
|
356
|
+
}
|
|
357
|
+
|
|
358
|
+
/**
|
|
359
|
+
* The bytes for a hash, or `null`.
|
|
360
|
+
*
|
|
361
|
+
* Never throws -- an unreadable blob is a miss, not a crash -- and **when `fleetId` is supplied, a blob owned by a
|
|
362
|
+
* different fleet is a miss too.** The caller cannot distinguish "no such blob" from "not yours", which is
|
|
363
|
+
* deliberate: a distinguishable refusal tells a prober that a hash exists on this relay.
|
|
364
|
+
*
|
|
365
|
+
* @param {string} hash
|
|
366
|
+
* @param {{fleetId?: string|null|undefined}} [options] omit `fleetId` entirely for an unscoped read
|
|
367
|
+
*/
|
|
368
|
+
getBlob(hash, options = {}) {
|
|
369
|
+
const file = this.blobFile(hash);
|
|
370
|
+
if (file === null) return null;
|
|
371
|
+
if (options.fleetId !== undefined) {
|
|
372
|
+
const entry = this.readBlobIndex()[hash];
|
|
373
|
+
// A blob with no recorded owner predates the index or was written unscoped; it is not this fleet's.
|
|
374
|
+
if (entry === undefined || entry.fleet_id !== options.fleetId) return null;
|
|
375
|
+
}
|
|
376
|
+
try {
|
|
377
|
+
return fs.readFileSync(file);
|
|
378
|
+
} catch {
|
|
379
|
+
return null;
|
|
380
|
+
}
|
|
381
|
+
}
|
|
382
|
+
|
|
383
|
+
hasBlob(hash) {
|
|
384
|
+
const file = this.blobFile(hash);
|
|
385
|
+
if (file === null) return false;
|
|
386
|
+
try {
|
|
387
|
+
return fs.statSync(file).isFile();
|
|
388
|
+
} catch {
|
|
389
|
+
return false;
|
|
390
|
+
}
|
|
391
|
+
}
|
|
392
|
+
|
|
393
|
+
/**
|
|
394
|
+
* Every stored blob, newest last, with the size and mtime needed to enforce the count cap.
|
|
395
|
+
*
|
|
396
|
+
* @returns {{hash:string, bytes:number, mtimeMs:number}[]}
|
|
397
|
+
*/
|
|
398
|
+
listBlobs() {
|
|
399
|
+
if (!this.enabled) return [];
|
|
400
|
+
const root = path.join(this.dir, BLOB_DIR);
|
|
401
|
+
let shards;
|
|
402
|
+
try {
|
|
403
|
+
shards = fs.readdirSync(root, { withFileTypes: true });
|
|
404
|
+
} catch {
|
|
405
|
+
return [];
|
|
406
|
+
}
|
|
407
|
+
const out = [];
|
|
408
|
+
for (const shard of shards) {
|
|
409
|
+
if (!shard.isDirectory()) continue;
|
|
410
|
+
const shardPath = path.join(root, shard.name);
|
|
411
|
+
let names;
|
|
412
|
+
try {
|
|
413
|
+
names = fs.readdirSync(shardPath, { withFileTypes: true });
|
|
414
|
+
} catch {
|
|
415
|
+
continue;
|
|
416
|
+
}
|
|
417
|
+
for (const entry of names) {
|
|
418
|
+
if (!entry.isFile() || entry.name.includes('.tmp-')) continue;
|
|
419
|
+
try {
|
|
420
|
+
const stat = fs.statSync(path.join(shardPath, entry.name));
|
|
421
|
+
out.push({ hash: entry.name, bytes: stat.size, mtimeMs: stat.mtimeMs });
|
|
422
|
+
} catch {
|
|
423
|
+
/* a blob that vanished mid-scan is not a failure */
|
|
424
|
+
}
|
|
425
|
+
}
|
|
426
|
+
}
|
|
427
|
+
out.sort((a, b) => a.mtimeMs - b.mtimeMs);
|
|
428
|
+
return out;
|
|
429
|
+
}
|
|
430
|
+
|
|
431
|
+
/**
|
|
432
|
+
* Make room for `incoming` more blobs, oldest first, and never take the last one.
|
|
433
|
+
*
|
|
434
|
+
* **The caller must prune the index for the returned hashes** -- this method deletes files, and the index is a
|
|
435
|
+
* separate document. `putBlob` does that pruning; a future caller that evicts for another reason owes the same.
|
|
436
|
+
*
|
|
437
|
+
* @param {number} incoming
|
|
438
|
+
* @returns {string[]} the hashes evicted
|
|
439
|
+
*/
|
|
440
|
+
evictBlobsFor(incoming) {
|
|
441
|
+
const existing = this.listBlobs();
|
|
442
|
+
const over = existing.length + incoming - MAX_BLOBS;
|
|
443
|
+
if (over <= 0) return [];
|
|
444
|
+
const evicted = [];
|
|
445
|
+
let count = existing.length;
|
|
446
|
+
for (const blob of existing) {
|
|
447
|
+
if (count <= 1) break; // never delete the only copy of anything
|
|
448
|
+
try {
|
|
449
|
+
fs.rmSync(path.join(this.dir, BLOB_DIR, blob.hash.slice(0, 2), blob.hash), { force: true });
|
|
450
|
+
evicted.push(blob.hash);
|
|
451
|
+
count -= 1;
|
|
452
|
+
} catch {
|
|
453
|
+
/* leave it; the caller will see the count did not drop */
|
|
454
|
+
}
|
|
455
|
+
if (evicted.length >= over) break;
|
|
456
|
+
}
|
|
457
|
+
if (evicted.length > 0) this.warn(BLOB_DIR, `evicted ${evicted.length} blob(s) to stay within ${MAX_BLOBS}`);
|
|
458
|
+
return evicted;
|
|
459
|
+
}
|
|
460
|
+
|
|
461
|
+
/** Read the hash -> owner map. A missing or corrupt index reads as empty rather than taking the relay down. */
|
|
462
|
+
readBlobIndex() {
|
|
463
|
+
if (!this.enabled) return {};
|
|
464
|
+
try {
|
|
465
|
+
const raw = fs.readFileSync(path.join(this.dir, BLOB_DIR, BLOB_INDEX_FILE), 'utf8');
|
|
466
|
+
const parsed = JSON.parse(raw);
|
|
467
|
+
return parsed !== null && typeof parsed === 'object' && !Array.isArray(parsed) ? parsed : {};
|
|
468
|
+
} catch {
|
|
469
|
+
return {};
|
|
470
|
+
}
|
|
471
|
+
}
|
|
472
|
+
|
|
473
|
+
/** Atomically replace the index. Best-effort: a failure is warned about, never thrown at the caller. */
|
|
474
|
+
writeBlobIndex(index) {
|
|
475
|
+
if (!this.enabled) return false;
|
|
476
|
+
const file = path.join(this.dir, BLOB_DIR, BLOB_INDEX_FILE);
|
|
477
|
+
try {
|
|
478
|
+
fs.mkdirSync(path.dirname(file), { recursive: true });
|
|
479
|
+
const tmp = `${file}.tmp-${process.pid}-${this._now()}`;
|
|
480
|
+
fs.writeFileSync(tmp, `${JSON.stringify(index)}\n`, 'utf8');
|
|
481
|
+
fs.renameSync(tmp, file);
|
|
482
|
+
return true;
|
|
483
|
+
} catch (error) {
|
|
484
|
+
this.warn(BLOB_DIR, `blob index write failed: ${error?.message ?? String(error)}`);
|
|
485
|
+
return false;
|
|
486
|
+
}
|
|
487
|
+
}
|
|
488
|
+
|
|
489
|
+
/** `<state>/blobs/<aa>/<sha256>`, or `null` when persistence is off or the hash is not a sha256. */
|
|
490
|
+
blobFile(hash) {
|
|
491
|
+
if (!this.enabled) return null;
|
|
492
|
+
if (typeof hash !== 'string' || !/^[0-9a-f]{64}$/.test(hash)) return null;
|
|
493
|
+
return path.join(this.dir, BLOB_DIR, hash.slice(0, 2), hash);
|
|
494
|
+
}
|
|
495
|
+
|
|
253
496
|
/** Redacted view for `/healthz` — never contains tokens. */
|
|
254
497
|
describe() {
|
|
498
|
+
const blobs = this.listBlobs();
|
|
255
499
|
return {
|
|
256
500
|
enabled: this.enabled,
|
|
257
501
|
dir: this.dir,
|
|
@@ -261,6 +505,11 @@ export class Persistence {
|
|
|
261
505
|
devices_missing: this.devicesMissing,
|
|
262
506
|
ledger_lines_read: this.ledgerLinesRead,
|
|
263
507
|
ledger_lines_skipped: this.ledgerLinesSkipped,
|
|
508
|
+
/** v1.0.7: the fleet sync channel's occupancy, reported as counts and bytes — never contents. */
|
|
509
|
+
blobs: blobs.length,
|
|
510
|
+
blob_bytes: blobs.reduce((sum, b) => sum + b.bytes, 0),
|
|
511
|
+
blobs_limit: MAX_BLOBS,
|
|
512
|
+
blob_bytes_limit: MAX_BLOB_BYTES,
|
|
264
513
|
/**
|
|
265
514
|
* v0.6.2. `degraded: true` means the relay is answering requests while failing to record them,
|
|
266
515
|
* which is the one relay failure that looks exactly like health from the outside.
|
package/src/relay/server.mjs
CHANGED
|
@@ -13,7 +13,7 @@ import https from 'node:https';
|
|
|
13
13
|
import fs from 'node:fs';
|
|
14
14
|
import os from 'node:os';
|
|
15
15
|
import path from 'node:path';
|
|
16
|
-
import { randomBytes } from 'node:crypto';
|
|
16
|
+
import { createHash, randomBytes } from 'node:crypto';
|
|
17
17
|
|
|
18
18
|
import {
|
|
19
19
|
MAX_FRAME_BYTES,
|
|
@@ -761,6 +761,77 @@ export class RelayServer {
|
|
|
761
761
|
const { _instanceKey, ...clean } = result;
|
|
762
762
|
return sendJson(res, 200, clean);
|
|
763
763
|
}
|
|
764
|
+
/**
|
|
765
|
+
* v1.0.7: the content-addressed blob channel (#1).
|
|
766
|
+
*
|
|
767
|
+
* ## Why the relay may carry bytes at all
|
|
768
|
+
*
|
|
769
|
+
* It persists `stdout_sha256` and never the text, and `src/agent/storage.mjs` records why: each machine's
|
|
770
|
+
* `memory.jsonl` is the only durable copy of what a command printed. **That invariant is about the relay not
|
|
771
|
+
* interpreting content** -- it never decides what bytes mean and never asserts they are authentic.
|
|
772
|
+
*
|
|
773
|
+
* A blob keyed by its own hash preserves exactly that. The relay still interprets nothing; **the hash is what
|
|
774
|
+
* says the bytes are the bytes**, exactly as `stdout_sha256` does for a result. What is new is only that a hash
|
|
775
|
+
* the relay already holds can have its bytes fetched from the same place.
|
|
776
|
+
*
|
|
777
|
+
* ## Why the hash is verified before storing
|
|
778
|
+
*
|
|
779
|
+
* A content-addressed store that accepts bytes under a key they do not hash to is not content-addressed -- the
|
|
780
|
+
* key stops being a claim about the content, and every later reader's verification becomes decorative. So a
|
|
781
|
+
* mismatch is **refused (422)**, never stored and never corrected: correcting it would silently re-key a blob
|
|
782
|
+
* the caller believes it stored under a different name.
|
|
783
|
+
*
|
|
784
|
+
* ## Fleet scoping
|
|
785
|
+
*
|
|
786
|
+
* `putFleetBlob`/`getFleetBlob` reuse `scopeFleetId`, the same rule task dispatch obeys. A blob written by one
|
|
787
|
+
* fleet is **not readable by another**, and the read refusal is a 404 rather than a 403: a distinguishable
|
|
788
|
+
* "not yours" tells a prober that the hash exists here.
|
|
789
|
+
*/
|
|
790
|
+
const blobMatch = /^\/v1\/fleet\/blobs\/([0-9a-f]{64})$/.exec(path);
|
|
791
|
+
if (blobMatch !== null) {
|
|
792
|
+
const hash = blobMatch[1];
|
|
793
|
+
if (method === 'PUT') {
|
|
794
|
+
const received = rawBody ?? Buffer.alloc(0);
|
|
795
|
+
const actual = createHash('sha256').update(received).digest('hex');
|
|
796
|
+
if (actual !== hash) {
|
|
797
|
+
throw new ProtocolError('BLOB_HASH_MISMATCH', 'the body does not hash to the key it was sent under', {
|
|
798
|
+
expected: hash,
|
|
799
|
+
actual,
|
|
800
|
+
bytes: received.length,
|
|
801
|
+
});
|
|
802
|
+
}
|
|
803
|
+
const stored = this.state.putFleetBlob(device, received);
|
|
804
|
+
if (stored.ok !== true) {
|
|
805
|
+
throw new ProtocolError(stored.reason ?? 'BLOB_WRITE_FAILED', stored.detail ?? 'the blob was refused', {
|
|
806
|
+
hash,
|
|
807
|
+
});
|
|
808
|
+
}
|
|
809
|
+
return sendJson(res, 201, {
|
|
810
|
+
protocol_version: PROTOCOL_VERSION,
|
|
811
|
+
ok: true,
|
|
812
|
+
hash: stored.hash,
|
|
813
|
+
bytes: stored.bytes,
|
|
814
|
+
evicted: stored.evicted,
|
|
815
|
+
});
|
|
816
|
+
}
|
|
817
|
+
if (method === 'GET') {
|
|
818
|
+
const bytes = this.state.getFleetBlob(device, hash);
|
|
819
|
+
if (bytes === null) {
|
|
820
|
+
// Same code and shape for "no such blob" and "not your fleet's" -- deliberately indistinguishable.
|
|
821
|
+
throw new ProtocolError('NOT_FOUND', 'no such blob is readable by this device', { hash });
|
|
822
|
+
}
|
|
823
|
+
res.writeHead(200, {
|
|
824
|
+
'content-type': 'application/octet-stream',
|
|
825
|
+
'content-length': bytes.length,
|
|
826
|
+
// Same reason as `/metrics`: a memory document must not be cached by an intermediary.
|
|
827
|
+
'cache-control': 'no-store',
|
|
828
|
+
});
|
|
829
|
+
res.end(bytes);
|
|
830
|
+
return true;
|
|
831
|
+
}
|
|
832
|
+
throw new ProtocolError('BAD_REQUEST', 'a blob is addressed with PUT or GET', { method });
|
|
833
|
+
}
|
|
834
|
+
|
|
764
835
|
if (method === 'GET' && path === '/v1/devices') {
|
|
765
836
|
return sendJson(res, 200, {
|
|
766
837
|
protocol_version: PROTOCOL_VERSION,
|
package/src/relay/state.mjs
CHANGED
|
@@ -17,6 +17,9 @@ import { createHash, randomBytes } from 'node:crypto';
|
|
|
17
17
|
// exhaustive tests — three lines inside this object would be untestable without a live relay.
|
|
18
18
|
import { targetsFor } from '../chatops/fleet-wiring.mjs';
|
|
19
19
|
import { deviceFleetFields } from '../chatops/fleet-wiring.mjs';
|
|
20
|
+
// v1.0.7: the one home for "which fleet does this requester act as", reused by the blob channel so it cannot
|
|
21
|
+
// drift from the rule the task routes already obey.
|
|
22
|
+
import { scopeFleetId } from '../chatops/fleet-wiring.mjs';
|
|
20
23
|
// v0.8.1: the account-derived fleet key, so a machine can join with no pairing code.
|
|
21
24
|
import { planJoin } from '../chatops/fleet-join.mjs';
|
|
22
25
|
|
|
@@ -56,6 +59,21 @@ export const ERROR_STATUS = Object.freeze({
|
|
|
56
59
|
SIGNATURE_EXPIRED: 401,
|
|
57
60
|
SIGNATURE_REPLAY: 401,
|
|
58
61
|
SIGNING_NOT_CONFIGURED: 500,
|
|
62
|
+
|
|
63
|
+
/* v1.0.7: the content-addressed blob channel.
|
|
64
|
+
*
|
|
65
|
+
* `BLOB_HASH_MISMATCH` is **422**, not 400: the request is well-formed and the bytes are fine, but they do not
|
|
66
|
+
* hash to the key they were sent under, so storing them would break content addressing for every later reader.
|
|
67
|
+
* That is a semantically invalid request, not a malformed one -- 422 is the status that says so.
|
|
68
|
+
*
|
|
69
|
+
* `PERSISTENCE_DISABLED` is **503**: the relay was started without a state directory, so the channel cannot
|
|
70
|
+
* exist. Retrying will not fix a missing directory, but a caller is still right to treat it as "this relay cannot
|
|
71
|
+
* serve this right now" rather than as its own mistake. */
|
|
72
|
+
BLOB_HASH_MISMATCH: 422,
|
|
73
|
+
BLOB_TOO_LARGE: 413,
|
|
74
|
+
BLOB_EMPTY: 400,
|
|
75
|
+
BLOB_WRITE_FAILED: 500,
|
|
76
|
+
PERSISTENCE_DISABLED: 503,
|
|
59
77
|
});
|
|
60
78
|
|
|
61
79
|
/** §5.1 required envelope fields — missing any one means `unverifiable`. */
|
|
@@ -510,6 +528,57 @@ export class RabbitState {
|
|
|
510
528
|
/** Inject/replace the persistence backend (also used by tests). */
|
|
511
529
|
setPersistence(persistence) { this._persist = persistence ?? null; return this._persist; }
|
|
512
530
|
|
|
531
|
+
/* ---------------- v1.0.7: the content-addressed blob channel (#1) ---------------- */
|
|
532
|
+
|
|
533
|
+
/**
|
|
534
|
+
* Which fleet a device acts as when it writes or reads a blob.
|
|
535
|
+
*
|
|
536
|
+
* **Reuses `scopeFleetId` rather than re-deriving it.** The rule "the requester's fleet comes from the device
|
|
537
|
+
* record the relay wrote at pairing time, never from the request" is already stated, tested and enforced for task
|
|
538
|
+
* dispatch; a second copy here would be a second place for it to drift, and the blob channel is exactly where a
|
|
539
|
+
* drift would leak one fleet's memory into another's.
|
|
540
|
+
*
|
|
541
|
+
* @param {{machine_id?: string, fleet_id?: string|null}|null} device
|
|
542
|
+
* @returns {{ok:true, fleetId:string|null, legacy:boolean}|{ok:false, reason:string, text:string}}
|
|
543
|
+
*/
|
|
544
|
+
scopeBlobFleet(device) {
|
|
545
|
+
return scopeFleetId({ device, relayFleetId: this.fleetId ?? null });
|
|
546
|
+
}
|
|
547
|
+
|
|
548
|
+
/**
|
|
549
|
+
* Store bytes under their SHA-256, owned by the writer's fleet.
|
|
550
|
+
*
|
|
551
|
+
* Delegates to `Persistence` -- this class never touches the filesystem (v0.1.2 §7), and that constraint is what
|
|
552
|
+
* keeps the durable path in one place.
|
|
553
|
+
*
|
|
554
|
+
* @param {object} device the authenticated device
|
|
555
|
+
* @param {Buffer} bytes
|
|
556
|
+
*/
|
|
557
|
+
putFleetBlob(device, bytes) {
|
|
558
|
+
if (!this._persist) {
|
|
559
|
+
return { ok: false, reason: 'PERSISTENCE_DISABLED', detail: 'the relay has no persistence backend' };
|
|
560
|
+
}
|
|
561
|
+
const scope = this.scopeBlobFleet(device);
|
|
562
|
+
if (scope.ok !== true) return { ok: false, reason: scope.reason, detail: scope.text };
|
|
563
|
+
return this._persist.putBlob(bytes, { fleetId: scope.fleetId });
|
|
564
|
+
}
|
|
565
|
+
|
|
566
|
+
/**
|
|
567
|
+
* The bytes for a hash, or `null`.
|
|
568
|
+
*
|
|
569
|
+
* `null` covers three cases on purpose -- no such blob, a blob owned by another fleet, and no persistence -- so a
|
|
570
|
+
* caller cannot use the relay to enumerate which hashes exist elsewhere on it.
|
|
571
|
+
*
|
|
572
|
+
* @param {object} device the authenticated device
|
|
573
|
+
* @param {string} hash
|
|
574
|
+
*/
|
|
575
|
+
getFleetBlob(device, hash) {
|
|
576
|
+
if (!this._persist) return null;
|
|
577
|
+
const scope = this.scopeBlobFleet(device);
|
|
578
|
+
if (scope.ok !== true) return null;
|
|
579
|
+
return this._persist.getBlob(hash, { fleetId: scope.fleetId });
|
|
580
|
+
}
|
|
581
|
+
|
|
513
582
|
/** Snapshot the device table; failures must never break the request path. */
|
|
514
583
|
_saveDevices() {
|
|
515
584
|
if (this._restoring || !this._persist) return false;
|