@uuaid/pillar 2.2.0 → 2.4.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/HOSTING-LIMITS.md CHANGED
@@ -29,42 +29,45 @@ envelope to every carrier, so each carrier counts all of your sends.
29
29
 
30
30
  | Limit | Value | Source | When you hit it | What to do |
31
31
  |---|---|---|---|---|
32
- | Long-poll `wait` | 30 s. A larger value is cut to 30 without an error | :63 | 200 with `envelopes: []` after 30 s | Ask for 25 (§3) |
33
- | Re-check while waiting | every 400 ms | :426 | n/a | Mail reaches a waiting poll about 0.4 s after it is stored, whatever `wait` is |
34
- | Rows per reply | 200, oldest first | :418 | 200 with 200 rows | Poll again at once, `since` = highest `seq` |
35
- | Open long-polls (`wait` > 0) | 48 for community; 64 in all, so 16 are kept for priority grants | :82, :74 | 503 `busy` | Wait at least 2 s, with jitter, and retry, or use `wait=0` for a while |
36
- | `x-pillar-ts` clock skew | ±5 min | :66 | 401 `clock-skew` | Fix your clock (NTP). Retrying does not help |
37
- | `x-pillar-delegation` size | 8,192 bytes | :67 | 401 `delegation-too-large` | Send a smaller delegation document |
38
-
39
- **Authentication** (:330-361). Send `x-pillar-pubkey`, `x-pillar-ts` (Unix ms)
32
+ | Long-poll `wait` | 30 s. A larger value is cut to 30 without an error | :77 | 200 with `envelopes: []` after 30 s | Ask for 25 (§3) |
33
+ | Re-check while waiting | every 400 ms | :672 | n/a | Mail reaches a waiting poll about 0.4 s after it is stored, whatever `wait` is |
34
+ | Rows per reply | 200, oldest first | :664 | 200 with 200 rows | Poll again at once, `since` = highest `seq` |
35
+ | Open long-polls (`wait` > 0) | 48 for community; 64 in all, so 16 are kept for priority grants | :96, :100 | 503 `busy` | Wait at least 2 s, with jitter, and retry, or use `wait=0` for a while |
36
+ | `x-pillar-ts` clock skew | ±5 min | :80 | 401 `clock-skew` | Fix your clock (NTP). Retrying does not help |
37
+ | `x-pillar-delegation` size | 8,192 bytes | :81 | 401 `delegation-too-large` | Send a smaller delegation document |
38
+
39
+ **Authentication** (:578-607). Send `x-pillar-pubkey`, `x-pillar-ts` (Unix ms)
40
40
  and `x-pillar-sig`, which is Ed25519 over
41
41
  `GET /v1/inbox/<uuaid>?since=<seq>` + `"\n"` + ts. `wait` is not signed. The
42
42
  uuaid must be derived from the key. To read another identity's inbox, also
43
43
  send `x-pillar-delegation`: a delegation document signed by that identity that
44
- names your key and covers `inbox` or `*` (:376-392). Every auth failure is a
44
+ names your key and covers `inbox` or `*` (:622-638). Every auth failure is a
45
45
  401 with the reason in `error`. Reads have no rate limit. Only the open
46
46
  long-polls above are capped, and `wait=0` reads are not capped at all.
47
47
 
48
48
  ### Sending: `POST /v1/envelopes`
49
49
 
50
- The carrier runs these checks in order (:267-313). Each budget is spent when
50
+ The carrier runs these checks in order (:486-576). Each budget is spent when
51
51
  its check passes. So a send refused later for being stale or for a full store
52
52
  has still used budget, and so has a duplicate.
53
53
 
54
54
  | Check | Value | Source | When you hit it | What to do |
55
55
  |---|---|---|---|---|
56
- | Raw body read | 2 MiB + 8 KiB | :65 | Connection dropped, no JSON body | Stay under your tier's body limit |
57
- | Envelope signature and shape | n/a | :268-274 | 400 with the reason (403 `delegation-invalid:*`) | Fix the envelope |
58
- | Refused sends per client IP | 60 per window | :73 | 429 `rate-limited`, which hides the real reason | Stop and fix the sender |
59
- | Body size | 512 KiB community; a tier grant can raise it (table below) | :64 | 413 `body-too-large` + `maxBodyBytes`, `tier` | Send the payload as a blob (§3) |
60
- | Sends per sender uuaid | 120 per window for community | :71 | 429 `rate-limited` | Back off (§3) |
61
- | Accepted sends per client IP | 300 per window; priority grants use their own bucket of 600 | :72, :83 | 429 `rate-limited` | Everyone behind your NAT shares this budget |
62
- | Envelope `createdAt` | within ±24 h of carrier time | :69 | 400 `stale-envelope` | Seal a new envelope. Never retry an old one |
63
- | Store size | 256 MiB. Community sends are refused above 85% (217.6 MiB) | :75, :81 | 503 `carrier-full` | Your other carriers still take it. Don't retry this one soon |
56
+ | Raw body read | 2 MiB + 8 KiB | :79 | Connection dropped, no JSON body | Stay under your tier's body limit |
57
+ | Envelope signature and shape | n/a | :487-497 | 400 with the reason (403 `delegation-invalid:*`) | Fix the envelope |
58
+ | Refused sends per client IP | 60 per window | :87 | 429 `rate-limited`, which hides the real reason | Stop and fix the sender |
59
+ | Body size | 512 KiB community; a tier grant can raise it (table below) | :78 | 413 `body-too-large` + `maxBodyBytes`, `tier` | Send the payload as a blob (§3) |
60
+ | Sends per sender uuaid | 120 per window for community | :85 | 429 `rate-limited` | Back off (§3) |
61
+ | Accepted sends per client IP | 300 per window; priority grants use their own bucket of 600 | :86, :112 | 429 `rate-limited` | Everyone behind your NAT shares this budget |
62
+ | Envelope `createdAt` | within ±24 h of carrier time | :83 | 400 `stale-envelope` | Seal a new envelope. Never retry an old one |
63
+ | Store size | 256 MiB. Community sends are refused above 85% (217.6 MiB) | :89, :95 | 503 `carrier-full` | Your other carriers still take it. Don't retry this one soon |
64
64
 
65
65
  Sending the same envelope id again is safe. It returns 202 with
66
- `duplicate: true` (:322-324). On a carrier with a receipt key (below), the
67
- same id with a different body is refused 409 `envelope-id-conflict`.
66
+ `duplicate: true` and the `seq` it was stored at (:565, :572). The carrier keys
67
+ envelopes on sender and id together (from 2.4.0), so another sender's envelope
68
+ under the same id is a different envelope and is stored. On a carrier with a
69
+ receipt key (below), the same id from the same sender with a different body is
70
+ refused 409 `envelope-id-conflict` (:570).
68
71
 
69
72
  **Carrier receipts (from 2.1.0).** Start the carrier with
70
73
  `--receipt-keychain <file>` (`CARRIER_RECEIPT_KEYCHAIN`; passphrase in
@@ -74,9 +77,9 @@ acceptance receipt signed from the stored row (src/net/carrier-receipt.mjs).
74
77
  The carrier never mints this key: a missing or unreadable file refuses to
75
78
  start. Without the option nothing changes.
76
79
 
77
- **Windows, memory, retention.** Every rate window is a sliding 60 s (:70),
78
- held in process memory (:114), so a restart resets every budget; past 10,000
79
- keys the oldest is dropped (:76). Envelopes are kept 14 days by default (:86;
80
+ **Windows, memory, retention.** Every rate window is a sliding 60 s (:84),
81
+ held in process memory (:230), so a restart resets every budget; past 10,000
82
+ keys the oldest is dropped (:90). Envelopes are kept 14 days by default (:156;
80
83
  `--ttl`, bin/carrier.mjs:32, bin/pillar-seed.mjs:66) and expired rows are
81
84
  deleted hourly (bin/carrier.mjs:46-49, bin/pillar-seed.mjs:171-174). Each row
82
85
  keeps the deadline set when it was stored (`retain_until` = acceptance time +
@@ -161,7 +164,7 @@ extra carriers.
161
164
  `<ns>` is one of the self-certifying namespaces `foundation`, `pillar` or
162
165
  `agentnet` (`SELF_CERTIFYING_NAMESPACES`, src/identity/keychain.mjs). Every
163
166
  identity `pillar init` mints is one (`--namespace pillar|agentnet|foundation`;
164
- the default is `foundation` in 2.2.0 and `pillar` from 2.2.1), and it is the rule
167
+ the default is `foundation` through 2.3.0 and `pillar` from 2.4.0), and it is the rule
165
168
  every envelope is accepted on. A device, a service, a registry
166
169
  namespace (`zilligon`, `iaaso`, …), or the same local id under any other
167
170
  prefix is not self-certified and needs a registry badge naming its key. A self-certified identity signs broker requests with keyId
@@ -300,7 +303,7 @@ front of each one.
300
303
  | Proxy rule | Why | pillar-sfo (live) and repo template |
301
304
  |---|---|---|
302
305
  | Carrier read timeout > 30 s + margin | The carrier can hold a request for 30 s plus one 400 ms tick | `/v1/` → 127.0.0.1:8799, 40 s; template: infra/pillar-carrier-fleet/provision-carrier.sh:102 |
303
- | Replace `X-Forwarded-For` with the peer's address | The carrier takes the first XFF entry as the client IP (:208-212). If the proxy appends to XFF, a client can pick its own rate bucket. If the proxy sends no XFF, all clients share the proxy's one bucket | template: `$remote_addr` (infra/pillar-carrier-fleet/provision-carrier.sh:107) |
306
+ | Replace `X-Forwarded-For` with the peer's address | The carrier takes the first XFF entry as the client IP (:395-399). If the proxy appends to XFF, a client can pick its own rate bucket. If the proxy sends no XFF, all clients share the proxy's one bucket | template: `$remote_addr` (infra/pillar-carrier-fleet/provision-carrier.sh:107) |
304
307
  | Body size ≥ largest honored body + 8 KiB | 512 KiB on a seed; 2 MiB + 8 KiB with `--tier-issuers`. Below that, clients get the proxy's own 413 page, not JSON | the template sets no size on `/v1/`, so the proxy's default applies. Set it explicitly |
305
308
  | CORS: allow `x-pillar-pubkey`, `x-pillar-ts`, `x-pillar-sig` | Browser clients send them. Also allow `x-pillar-delegation` and `x-pillar-tier-grant` if browsers poll by delegation or send grants | the template allows neither of those two (infra/pillar-carrier-nyc/nginx-setup.sh:25) |
306
309
  | Relay: a short timeout is fine | No relay route long-polls | `/push/` → 127.0.0.1:8801/, 30 s |
@@ -308,13 +311,40 @@ front of each one.
308
311
 
309
312
  **Under load:**
310
313
  - **`carrier-full` does not clear by itself.** `vacuum()` deletes expired rows
311
- (:239-243) but never runs SQLite `VACUUM`. The size check reads `page_count`
312
- (:308-311), which still counts freed pages. In a local test with the same
314
+ (:436-441) but never runs SQLite `VACUUM`. The size check reads `page_count`
315
+ (:527-530), which still counts freed pages. In a local test with the same
313
316
  schema, deleting all 300 rows of a 30,904,320-byte store left
314
317
  `page_count × page_size` at 30,904,320 bytes, with 7,540 pages free. So once
315
318
  the file passes 217.6 MiB, community sends get 503 until you stop the
316
319
  carrier, run `sqlite3 <db> 'VACUUM;'`, and restart it. On a `pillar-seed`
317
320
  every sender is community, so that means everyone. Watch the file size.
321
+ - **From 2.4.0 a start may run `VACUUM`.** Re-keying the envelopes table (see
322
+ RELEASE-NOTES-2.4.0.md) copies it and frees the old copy, which would double
323
+ the counted size. The carrier reclaims those pages right after the re-key,
324
+ then truncates the WAL (:212-224). It also vacuums on any later start that
325
+ finds more than a quarter of the file's pages free. That retries a first
326
+ start that was killed or whose `VACUUM` failed. It also catches a store
327
+ thinned that much by TTL expiry, where the cap was counting the free pages.
328
+ At a quarter free or less, a start does not vacuum. If a `VACUUM` fails, the
329
+ carrier logs `VACUUM failed` with the free page count and keeps running; the
330
+ next start tries again, or stop it and run `sqlite3 <db> 'VACUUM;'`. A
331
+ `VACUUM` runs before the carrier listens, so the start waits for it and
332
+ connections are refused meanwhile (a 122 MiB store thinned by 30% took
333
+ 0.44 s in a local test; carrier disks are not measured). Stop every other
334
+ process that has the store open (a second carrier, a `sqlite3` shell) before
335
+ restarting: one holding a read transaction makes the WAL truncation wait out
336
+ SQLite's 5 s busy timeout, and the carrier then logs that the WAL was not
337
+ truncated. At peak, under the default secure delete, the re-key used about
338
+ five times the table on disk in a 119 MiB test: the database, a WAL as large
339
+ as it (secure delete zeroes the dropped table through the WAL), and the
340
+ backup. About 1.1 times of that is SQLite's temporary file, in
341
+ `SQLITE_TMPDIR` or `TMPDIR`, which may be another filesystem: leave room in
342
+ both before the first 2.4.0 start.
343
+ - **Re-key backups under secure delete.** A start removes
344
+ `<db>.pre-sender-unique-<ms>-<pid>.bak` files that no running re-key is
345
+ still writing: its process is gone; or it is over an hour old; or it carries
346
+ the carrier's own pid but predates this start (a container restart reuses
347
+ the pid, often 1). After the first 2.4.0 start, check that none remains.
318
348
  - Sends get 429 and 503 as in §1; new long-polls get 503 `busy` once 48
319
349
  community (64 in all) are open.
320
350
  - Nothing limits `wait=0` reads, `/v1/health` or `/v1/seed-info`. Rate-limit
@@ -350,7 +380,7 @@ carrier's code is the `reason` of a refused send or the `error` of a refused
350
380
  inbox read; the broker's is its `code`. `pillar` and `pillar-blob` print a
351
381
  refusal as one line, `<code>: <fix>`: the server's `fix` when it sends one,
352
382
  printed as received (only line breaks and control characters become spaces),
353
- else the client's map (src/net/refusals.mjs:24-73, exported as
383
+ else the client's map (src/net/refusals.mjs:24-76, exported as
354
384
  `@uuaid/pillar/refusals`). A carrier older than 0.2.0-alpha.18 sends no
355
385
  `fix`; the client fills it in. On a thrown error, `err.code` and `err.fix`
356
386
  carry the same pair. Every fix, the client's and both servers', is a fixed
@@ -57,7 +57,10 @@ and put the real verb inside the sealed payload.
57
57
  1. Thread root = the first `request`'s envelope id (known to recipient from
58
58
  the envelope itself; the request payload needs no thread field).
59
59
  2. Every `request` and `reply` SHOULD be acked within 30s. Absence of an
60
- ack = retry (idempotent — carrier dedups on envelope id), not failure.
60
+ ack = retry, not failure. Resend the SAME envelope: the carrier dedups on
61
+ sender + envelope id. A new attempt that re-seals takes a new id: a carrier
62
+ with a receipt key refuses a re-sealed envelope under the same id (409
63
+ `envelope-id-conflict`), and one without keeps the first.
61
64
  3. Multi-turn: same `thread` for the whole conversation, `inReplyTo` =
62
65
  the latest envelope id each time.
63
66
  4. Errors ride the normal channel: `kind: "reply", status: "error",
package/README.md CHANGED
@@ -56,8 +56,8 @@ vars and a JSON summary on stdin, and only fires when there is mail.
56
56
 
57
57
  ### Several readers on one home: `--reader`
58
58
 
59
- Carrier reads are non-destructive and the mailbox is idempotent on envelope
60
- id, so it never matters which process PULLS a message. It used to matter who
59
+ Carrier reads are non-destructive and the mailbox is idempotent on sender +
60
+ envelope id, so it never matters which process PULLS a message. It used to matter who
61
61
  SAW it: `pillar inbox` lists as new only what that run pulled, so when two
62
62
  readers shared a home, whichever ran first was the only one that saw the mail
63
63
  (and with `--mark-read`, the other never saw it at all).
@@ -258,7 +258,7 @@ header (`x-pillar-tier-grant`), not the envelope.
258
258
 
259
259
  On send, `deliver()` replicates the envelope to **every** carrier in your set
260
260
  in parallel — any one accepting it is success. On receive, you poll **all** of
261
- them; duplicates across seeds are deduped by envelope id, so handlers fire
261
+ them; duplicates across seeds are deduped by sender + envelope id, so handlers fire
262
262
  exactly once. Kill any single seed and delivery continues through the
263
263
  survivors.
264
264
 
package/bin/pillar.mjs CHANGED
@@ -3,7 +3,7 @@
3
3
  * pillar — one-command CLI.
4
4
  *
5
5
  * pillar init [--home ~/.pillar] [--namespace pillar|agentnet|foundation]
6
- * first-time setup: mint UUAID + keys (default namespace: foundation in 2.2.0)
6
+ * first-time setup: mint UUAID + keys (default namespace: pillar since 2.4.0)
7
7
  * pillar init --refresh [--directory URL] re-sync carriers + address book (keychain untouched)
8
8
  * pillar init --force --confirm <current uuaid> REPLACE the identity (new UUAID); the old one is archived
9
9
  * pillar id [--home ~/.pillar] print our UUAID + PeerId + addrs
@@ -42,7 +42,7 @@ import { parseLinkOffer, isLinkOfferLive, LINK_GRANT_KINDS } from "../src/net/li
42
42
  import { mintDelegationDoc } from "../src/net/delegation.mjs";
43
43
  import { refusalLine } from "../src/net/refusals.mjs";
44
44
  import {
45
- readerKey, processContext, otherLiveLeases, describeProcess, describeReader, initialCursor, handleInOrder,
45
+ readerKey, processContext, otherLiveLeases, processesWithMailboxOpen, describeProcess, describeReader, initialCursor, handleInOrder,
46
46
  LEASE_TTL_MS, LEASE_HEARTBEAT_MS,
47
47
  } from "../src/readers.mjs";
48
48
  import { extractGroupTag, findGroup, normalizeMembers } from "../src/groups.mjs";
@@ -382,6 +382,17 @@ async function confirmAndArchiveIdentity(home, old, args) {
382
382
  const { Mailbox } = await import("../src/mailbox/mailbox.mjs");
383
383
  const mbPath = join(home, "mailbox.db");
384
384
  if (existsSync(mbPath)) {
385
+ // Two checks, each catching what the other cannot:
386
+ // 1. Lease scan (otherLiveLeases): a recv daemon that has started and is
387
+ // heartbeating. Catches remote hosts and processes that opened the DB
388
+ // but have since closed it.
389
+ // 2. File-open scan (processesWithMailboxOpen via lsof): any process with
390
+ // an open fd on mailbox.db right now — a pre-lease daemon (idle, DB
391
+ // open, no lease yet) or a one-shot `pillar inbox` run.
392
+ // The exclusive-lock probe (BEGIN EXCLUSIVE with busy_timeout=0) was
393
+ // rejected: WAL mode lets concurrent readers proceed, so an idle daemon
394
+ // holding the DB open but not writing would NOT block the probe, giving
395
+ // a false pass.
385
396
  const mb = new Mailbox(mbPath);
386
397
  const live = otherLiveLeases(mb);
387
398
  mb.close();
@@ -389,6 +400,11 @@ async function confirmAndArchiveIdentity(home, old, args) {
389
400
  console.error(`refusing to replace the identity while ${live.map(describeProcess).join("; ")} is reading this home — stop it first`);
390
401
  exit(1);
391
402
  }
403
+ const holders = processesWithMailboxOpen(mbPath);
404
+ if (holders.length) {
405
+ console.error(`refusing to replace the identity while ${holders.map((p) => `${p.command} (pid ${p.pid})`).join("; ")} has this home's mailbox open — stop it first`);
406
+ exit(1);
407
+ }
392
408
  }
393
409
  const stamp = new Date().toISOString().replace(/[:.]/g, "-");
394
410
  const dir = join(home, "rotated", `${stamp}-${(current.split(":")[3] ?? "unknown").slice(0, 8)}`);
@@ -973,7 +989,7 @@ async function cmdRecv(args) {
973
989
  const { env, payload } = viewOfRow(p, row, decrypt);
974
990
  await handle({ ...env, payload });
975
991
  }, {
976
- advance: (row) => p.mailbox.setReaderCursor(rk.key, row.seq),
992
+ advance: (row) => p.mailbox.setReaderCursor(rk.key, row.released_seq ?? row.seq),
977
993
  onError: (row, e) => console.error(`reader ${rk.key}: row ${row.seq} not handled, not retried: ${e.message}`),
978
994
  });
979
995
  } finally { busy = false; }
@@ -1038,7 +1054,7 @@ async function cmdInbox(args) {
1038
1054
  console.log(JSON.stringify(inboxLine(env, payload, { new: true })));
1039
1055
  printed.push({ id: env.id, sender: env.sender, kind: env.kind, payload, groupId });
1040
1056
  }
1041
- if (rows.length && !args.peek) p.mailbox.setReaderCursor(rk.key, rows.at(-1).seq);
1057
+ if (rows.length && !args.peek) p.mailbox.setReaderCursor(rk.key, rows.at(-1).released_seq ?? rows.at(-1).seq);
1042
1058
  const qCount = (p.mailbox.inboxQuarantined?.(500) ?? []).length;
1043
1059
  console.log(`inbox: ${rows.length} new for reader "${rk.key}" (${pulled} pulled from carriers by this run)${args.peek ? ", cursor not moved (--peek)" : ""}${qCount ? `, ${qCount} quarantined (pillar trust quarantine)` : ""}`);
1044
1060
  } else {
@@ -1051,8 +1067,10 @@ async function cmdInbox(args) {
1051
1067
  printed.push({ id: env.id, sender: env.sender, kind: env.kind, payload: env.payload, groupId: extractGroupTag(env.payload)?.id ?? null });
1052
1068
  }
1053
1069
  // Backlog: rows stored earlier (e.g. by a recv daemon) but never surfaced.
1054
- const freshIds = new Set(fresh.map(e => e.id));
1055
- backlog = (p.mailbox.inboxUnread?.(500) ?? []).filter(r => !freshIds.has(r.envelope_id));
1070
+ // Keyed on (sender, id): an id is unique only within its sender.
1071
+ const freshKeys = new Set(fresh.map(e => `${e.sender}\n${e.id}`));
1072
+ const isFresh = (r) => freshKeys.has(`${r.sender}\n${r.envelope_id}`);
1073
+ backlog = (p.mailbox.inboxUnread?.(500) ?? []).filter(r => !isFresh(r));
1056
1074
  if (args.unread) {
1057
1075
  for (const row of backlog) {
1058
1076
  const { env, payload, groupId } = viewOfRow(p, row, decrypt);
@@ -1067,7 +1085,7 @@ async function cmdInbox(args) {
1067
1085
  // other reader already marked read are missing.
1068
1086
  if (prev) {
1069
1087
  const taken = p.mailbox.inboxAfter(prev.cursor, 1000)
1070
- .filter((r) => !freshIds.has(r.envelope_id) && r.pulled_by !== rk.key && (!args.unread || r.read_at));
1088
+ .filter((r) => !isFresh(r) && r.pulled_by !== rk.key && (!args.unread || r.read_at));
1071
1089
  if (taken.length) {
1072
1090
  const by = new Map();
1073
1091
  for (const r of taken) by.set(r.pulled_by ?? null, (by.get(r.pulled_by ?? null) ?? 0) + 1);
@@ -1086,7 +1104,7 @@ async function cmdInbox(args) {
1086
1104
  console.log(`inbox: ${fresh.length} new, ${backlog.length} older unread${args.unread ? "" : " (--unread to print)"}${qCount ? `, ${qCount} quarantined (pillar trust quarantine)` : ""}`);
1087
1105
  }
1088
1106
  if (args["mark-read"]) {
1089
- for (const v of printed) p.mailbox.markInboxRead?.(v.id);
1107
+ for (const v of printed) p.mailbox.markInboxRead?.(v.id, v.sender);
1090
1108
  const r = await p.sendReadReceipts(printed);
1091
1109
  if (r.sent || r.dropped) console.error(`read receipts: ${r.sent} sent${r.dropped ? `, ${r.dropped} not sent (per-run cap)` : ""}`);
1092
1110
  }
@@ -1246,7 +1264,7 @@ async function cmdTrust(args) {
1246
1264
  // pillar trust set <open|directory|allowlist>
1247
1265
  // pillar trust allow|block <uuaid> add to list (removes from the other)
1248
1266
  // pillar trust unlist <uuaid> remove from both lists
1249
- // pillar trust quarantine [--release ID] list quarantined mail / release one
1267
+ // pillar trust quarantine [--release ID [--from UUAID]] list quarantined mail / release one
1250
1268
  const home = homePath(args);
1251
1269
  const tPath = join(home, "trust.json");
1252
1270
  const t = loadTrustPolicy(home);
@@ -1292,8 +1310,22 @@ async function cmdTrust(args) {
1292
1310
  const { Mailbox } = await import("../src/mailbox/mailbox.mjs");
1293
1311
  const mb = new Mailbox(join(home, "mailbox.db"));
1294
1312
  if (args.release) {
1295
- mb.db.prepare("update inbox set quarantined = null where envelope_id = ?").run(args.release);
1296
- console.log(`released: ${args.release} (now visible to pillar inbox --unread)`);
1313
+ // An id is unique only within its sender: with two quarantined senders
1314
+ // under one id, --from picks which one to release.
1315
+ const held = mb.db.prepare("select sender from inbox where envelope_id = ? and quarantined is not null").all(args.release);
1316
+ const from = typeof args.from === "string" ? args.from : undefined;
1317
+ if (!from && held.length > 1) {
1318
+ console.error(`${held.length} senders have ${args.release} quarantined; release one with --from <uuaid>: ${held.map(r => r.sender).join(", ")}`);
1319
+ mb.close();
1320
+ exit(2);
1321
+ }
1322
+ const changed = mb.releaseQuarantined(args.release, from);
1323
+ if (changed === 0) {
1324
+ console.error(`no quarantined row with id ${args.release}${from ? ` from ${from}` : ""} — nothing released`);
1325
+ mb.close();
1326
+ exit(1);
1327
+ }
1328
+ console.log(`released: ${args.release}${from ? ` from ${from}` : ""} (now visible to every reader, including those whose cursor passed its seq)`);
1297
1329
  mb.close();
1298
1330
  return;
1299
1331
  }
@@ -1303,7 +1335,7 @@ async function cmdTrust(args) {
1303
1335
  mb.close();
1304
1336
  return;
1305
1337
  }
1306
- console.error("usage: pillar trust [set <policy>|allow <uuaid>|block <uuaid>|unlist <uuaid>|quarantine [--release ID]]");
1338
+ console.error("usage: pillar trust [set <policy>|allow <uuaid>|block <uuaid>|unlist <uuaid>|quarantine [--release ID [--from UUAID]]]");
1307
1339
  exit(2);
1308
1340
  }
1309
1341
 
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@uuaid/pillar",
3
- "version": "2.2.0",
3
+ "version": "2.4.0",
4
4
  "description": "Pillar (Protocol for Inter-agent Linked Ledger And Relay) — the secure delivery layer of UACP, the Universal Agent Communication Protocol. UACP = UUAID (identity) + Envelope (signed, encrypted container) + Pillar (relay, mailbox, routing, store-and-forward). Reference implementation of the UACP delivery layer; includes VDA-1 (IAASO-3901, in ballot) delivery attestation and a blind escrow for the Zilligon ZGN / EVM rails.",
5
5
  "license": "Apache-2.0",
6
6
  "type": "module",
@@ -24,7 +24,7 @@
24
24
  */
25
25
  import { createHash, createCipheriv, createDecipheriv, hkdfSync, randomBytes } from "node:crypto";
26
26
  import { ed25519, x25519 } from "@noble/curves/ed25519.js";
27
- import { jcs } from "../identity/jcs.mjs";
27
+ import { jcsRfc8785 } from "../identity/jcs.mjs";
28
28
 
29
29
  export const E2E_ALG = "x25519-hkdf-sha256-aes256gcm";
30
30
  const HKDF_INFO = "uuaid-pillar-e2e/1";
@@ -58,7 +58,8 @@ export function sealPayload(recipientEdPubHex, payload, { aad }) {
58
58
  const iv = randomBytes(12);
59
59
  const cipher = createCipheriv("aes-256-gcm", key, iv);
60
60
  cipher.setAAD(Buffer.from(aad, "utf-8"));
61
- const plaintext = Buffer.from(jcs(payload), "utf-8");
61
+ // Sealed payloads carry full RFC 8785 numbers (2026-10-24); see identity/jcs.mjs.
62
+ const plaintext = Buffer.from(jcsRfc8785(payload), "utf-8");
62
63
  const ct = Buffer.concat([cipher.update(plaintext), cipher.final()]);
63
64
  const tag = cipher.getAuthTag();
64
65
  return {
package/src/groups-v2.mjs CHANGED
@@ -995,7 +995,7 @@ export class GroupReceiver {
995
995
  for (const h of due) {
996
996
  const r = this._judgeKnown(h.env, h.g, now, out);
997
997
  const x = r.reason ? { msgId: h.msgId, verdict: r.verdict, reason: r.reason } : { msgId: h.msgId, verdict: r.verdict };
998
- if (h.env.ref !== undefined) x.ref = h.env.ref; // the caller's handle on the held message (pillar: the envelope id)
998
+ if (h.env.ref !== undefined) x.ref = h.env.ref; // the caller's handle on the held message (pillar: { id, sender } of its envelope)
999
999
  out.released.push(x);
1000
1000
  }
1001
1001
  }
@@ -2,16 +2,27 @@
2
2
  * RFC 8785 JCS canonicalization. Same implementation as the pillar-core reference,
3
3
  * kept independent so the pillar-node package has no upward dependency.
4
4
  *
5
- * Numbers: integers only (safe for the wire we design; enforce it). Floats
6
- * throw because deterministic ECMA-262 number serialization requires more code
7
- * than we want in a reference impl and this package's message payloads never
8
- * legitimately contain non-integer numbers.
5
+ * Numbers — two entry points with one byte rule for integers:
6
+ * jcs() INTEGERS ONLY; floats throw. Every transport document (envelopes,
7
+ * delegations, receipts, escrow terms, blobs, VDA) keeps using it, so
8
+ * what the carrier and escrow refuse is unchanged — escrow v1/v2 rely
9
+ * on that throw to reject a hostile offer (src/escrow/terms.mjs).
10
+ * jcsRfc8785() full RFC 8785 §3.2.2.3 numbers (ECMAScript Number::toString, i.e.
11
+ * JSON.stringify(n)); used only for the SEALED PAYLOAD (crypto/e2e.mjs
12
+ * sealPayload) and documents carried inside it (PAM-1, AinFi). Byte-
13
+ * identical to jcs() for every finite integer. Added 2026-10-24 so an
14
+ * agent can send a device a value such as 21.5; byte-matches
15
+ * @uuaid/pillar-client jcsRfc8785.
9
16
  */
10
17
  export function jcs(value) {
11
- return _ser(value);
18
+ return _ser(value, false);
12
19
  }
13
20
 
14
- function _ser(v) {
21
+ export function jcsRfc8785(value) {
22
+ return _ser(value, true);
23
+ }
24
+
25
+ function _ser(v, fractions) {
15
26
  if (v === null) return "null";
16
27
  if (v === true) return "true";
17
28
  if (v === false) return "false";
@@ -19,16 +30,17 @@ function _ser(v) {
19
30
  if (typeof v === "number") {
20
31
  if (!Number.isFinite(v)) throw new Error("JCS: NaN/Infinity not permitted");
21
32
  if (Number.isInteger(v)) return String(v);
33
+ if (fractions) return JSON.stringify(v); // RFC 8785 §3.2.2.3
22
34
  throw new Error("JCS: non-integer numbers not permitted; use string decimal encoding");
23
35
  }
24
36
  if (typeof v === "bigint") return v.toString();
25
- if (Array.isArray(v)) return "[" + v.map(_ser).join(",") + "]";
37
+ if (Array.isArray(v)) return "[" + v.map((x) => _ser(x, fractions)).join(",") + "]";
26
38
  if (typeof v === "object") {
27
39
  // RFC 8785 §3.2.3: keys sort by their UTF-16 code units, which is exactly
28
40
  // JavaScript's default string order. (Comparing UTF-16LE *bytes* is not: it
29
41
  // puts "Ā" (U+0100) before "a", and disagreed with @uuaid/pillar-client.)
30
42
  const keys = Object.keys(v).sort(); // @rule jcs.code-unit-order
31
- return "{" + keys.map(k => JSON.stringify(k) + ":" + _ser(v[k])).join(",") + "}";
43
+ return "{" + keys.map(k => JSON.stringify(k) + ":" + _ser(v[k], fractions)).join(",") + "}";
32
44
  }
33
45
  throw new Error("JCS: unsupported type " + typeof v);
34
46
  }
@@ -72,12 +72,14 @@ export const SELF_CERTIFYING_NAMESPACES = Object.freeze(["foundation", "pillar",
72
72
  * The namespace `pillar init` mints a NEW agent identity in when no --namespace is given.
73
73
  * Expand before contract (owner order 2026-10-24; Uuaid-Codex review K1): 2.2.0 makes every
74
74
  * verifier accept foundation, pillar and agentnet but still mints `foundation`, because a
75
- * `pillar:` identity cannot read its inbox on a pre-2.2.0 carrier. 2.2.1 flips this to
76
- * `pillar` once the carrier fleet, the main relay and the blob brokers run 2.2.x/1.2.0.
75
+ * `pillar:` identity cannot read its inbox on a pre-2.2.0 carrier. 2.4.0 flips this to
76
+ * `pillar` (the contract half; owner order "when i create a new uuaid . it will not be A
77
+ * foundation UUAID"), now that the carrier fleet, the main relay and the blob brokers run
78
+ * 2.2.x/1.2.0. 2.2.1 was never published: it would have moved `latest` below 2.3.0.
77
79
  * Existing keychains keep the UUAID they were saved with. `Keychain.generate`'s own default
78
80
  * stays `foundation` for library callers; CLI mint sites pass `mintNamespace(...)` instead.
79
81
  */
80
- export const NEW_IDENTITY_NAMESPACE = "foundation";
82
+ export const NEW_IDENTITY_NAMESPACE = "pillar";
81
83
 
82
84
  /**
83
85
  * Carrier-infrastructure identities (pillar-seed, pillar-directory, pillar-escrow) stay in
package/src/index.mjs CHANGED
@@ -8,7 +8,7 @@
8
8
  * lives at @uuaid/pillar-browser (later).
9
9
  */
10
10
  export { Keychain, localIdFromKey, SELF_CERTIFYING_NAMESPACES, NEW_IDENTITY_NAMESPACE, SERVICE_IDENTITY_NAMESPACE, mintNamespace, keyDerivedUuaidFor, keyDerivesUuaid } from "./identity/keychain.mjs";
11
- export { jcs } from "./identity/jcs.mjs";
11
+ export { jcs, jcsRfc8785 } from "./identity/jcs.mjs";
12
12
  export { seal, open, decrypt, envelopeSha, keychainSeed, ENVELOPE_VERSION, E2E_ALG } from "./net/envelope.mjs";
13
13
  export { DELEGATION_VERSION, verifyDelegationDoc, delegationCoversKind } from "./net/delegation.mjs";
14
14
  export { sealPayload, openPayload, edPubToX25519, edSeedToX25519 } from "./crypto/e2e.mjs";
@@ -20,6 +20,33 @@ import { createHash } from "node:crypto";
20
20
  import { mkdirSync } from "node:fs";
21
21
  import { dirname } from "node:path";
22
22
  import { jcs } from "../identity/jcs.mjs";
23
+ import { rekeyToSenderUnique } from "../sqlite-rekey.mjs";
24
+
25
+ // An envelope id is unique only within its sender (Annex B5(a),
26
+ // src/sqlite-rekey.mjs). The outbox keeps UNIQUE(envelope_id): every id in it
27
+ // is this agent's own. INBOX_COLUMNS must declare every column _migrate can
28
+ // add to an older inbox, because the re-key copies the old table's columns.
29
+ const INBOX_COLUMNS = `
30
+ seq integer primary key autoincrement,
31
+ envelope_id text not null,
32
+ envelope_sha256 text not null,
33
+ sender text not null,
34
+ received_at text not null,
35
+ envelope_json text not null,
36
+ chain_hash text not null,
37
+ read_at text,
38
+ quarantined text,
39
+ pulled_by text,
40
+ group_id text,
41
+ released_seq integer,
42
+ unique (sender, envelope_id)`;
43
+ const INBOX_INDEX = "create index if not exists inbox_sender on inbox (sender);";
44
+ // Every cursor read (_seqFloor, inboxTipSeq, inboxAfter) orders on a row's
45
+ // position, coalesce(released_seq, seq); without this index each one, and so
46
+ // every append, scans the whole inbox.
47
+ const INBOX_POS_INDEX = "create index if not exists inbox_pos on inbox (coalesce(released_seq, seq));";
48
+ // The re-key drops the old table and with it every index on it.
49
+ const INBOX_INDEXES = [INBOX_INDEX, "create index if not exists inbox_group on inbox (group_id);", INBOX_POS_INDEX];
23
50
 
24
51
  const SCHEMA = `
25
52
  create table if not exists outbox (
@@ -37,17 +64,8 @@ create table if not exists outbox (
37
64
  create index if not exists outbox_recipient on outbox (recipient);
38
65
  create index if not exists outbox_status on outbox (status);
39
66
 
40
- create table if not exists inbox (
41
- seq integer primary key autoincrement,
42
- envelope_id text not null unique,
43
- envelope_sha256 text not null,
44
- sender text not null,
45
- received_at text not null,
46
- envelope_json text not null,
47
- chain_hash text not null,
48
- read_at text
49
- );
50
- create index if not exists inbox_sender on inbox (sender);
67
+ create table if not exists inbox (${INBOX_COLUMNS});
68
+ ${INBOX_INDEX}
51
69
 
52
70
  create table if not exists carrier_cursors (
53
71
  carrier text primary key,
@@ -75,6 +93,15 @@ export class Mailbox {
75
93
  // transaction that re-reads table_info inside it: the second opener waits
76
94
  // on the write lock (busy timeout) and then finds nothing left to add.
77
95
  this.db.transaction(() => this._migrate()).immediate();
96
+ // UNIQUE(sender, envelope_id) (Annex B5(a), user_version 1). It runs after
97
+ // _migrate, so the table it copies already has every column INBOX_COLUMNS
98
+ // declares, and outside it: the backup is VACUUM INTO, which SQLite refuses
99
+ // inside a transaction. The re-key takes its own immediate transaction and
100
+ // re-checks there, so of two openers only the first rebuilds.
101
+ rekeyToSenderUnique(this.db, { table: "inbox", columnsSql: INBOX_COLUMNS, indexSql: INBOX_INDEXES });
102
+ // A store the re-key leaves alone (fresh, or re-keyed by an earlier
103
+ // build) gets the position index here. An index needs no ladder step.
104
+ this.db.exec(INBOX_POS_INDEX);
78
105
  }
79
106
 
80
107
  _migrate() {
@@ -100,7 +127,7 @@ export class Mailbox {
100
127
  if (!have.has(name)) this.db.exec(`alter table ${table} add column ${name} ${type}`);
101
128
  }
102
129
  };
103
- addCols("inbox", [["pulled_by", "text"], ["group_id", "text"]]);
130
+ addCols("inbox", [["pulled_by", "text"], ["group_id", "text"], ["released_seq", "integer"]]);
104
131
  addCols("outbox", [
105
132
  ["kind", "text"], ["thread", "text"], ["group_id", "text"], ["group_msg_id", "text"],
106
133
  ["recipient_ack_at", "text"], ["recipient_ack_id", "text"], ["recipient_read_at", "text"],
@@ -212,26 +239,45 @@ export class Mailbox {
212
239
  }
213
240
 
214
241
  /**
215
- * Append an incoming envelope. Same shape. MUST be idempotent on envelope.id;
216
- * a duplicate delivery does NOT re-append and does NOT advance the chain.
242
+ * Append an incoming envelope. Same shape. MUST be idempotent on
243
+ * (sender, envelope.id); a duplicate delivery does NOT re-append and does
244
+ * NOT advance the chain. A DIFFERENT envelope under a stored (sender, id)
245
+ * is not appended either: it returns `collision: true` alongside
246
+ * `duplicate: true`, so a caller that only checks `duplicate` still fires
247
+ * no handlers. The same id from another sender is a different envelope.
217
248
  * `pulledBy` is the reader key of the storing process (metadata only).
249
+ * `quarantined` is the trust-gate reason string; set it here so insert and
250
+ * quarantine are one commit — a named reader in another process can never
251
+ * see the row unquarantined between the two.
218
252
  */
219
- appendInbound(envelope, { pulledBy = null } = {}) {
253
+ appendInbound(envelope, { pulledBy = null, quarantined = null } = {}) {
220
254
  return this.db.transaction(() => {
221
- const existing = this.db.prepare("select seq, envelope_id, envelope_sha256 from inbox where envelope_id = ?").get(envelope.id);
255
+ const sender = envelope.sender ?? envelope.from ?? "unknown";
256
+ const envSha = Mailbox.envelopeSha(envelope);
257
+ const existing = this.db.prepare("select seq, envelope_sha256 from inbox where envelope_id = ? and sender = ?").get(envelope.id, sender);
222
258
  if (existing) {
223
- return { duplicate: true, seq: existing.seq, envelopeId: envelope.id, envelopeSha256: existing.envelope_sha256 };
259
+ const collision = existing.envelope_sha256 !== envSha;
260
+ return { duplicate: true, ...(collision ? { collision: true } : {}), seq: existing.seq, envelopeId: envelope.id, envelopeSha256: existing.envelope_sha256 };
224
261
  }
225
- const envSha = Mailbox.envelopeSha(envelope);
226
262
  const prev = this._getHead("inbox");
227
263
  const chainInput = prev == null ? CHAIN_SEED + envSha : prev + envSha;
228
264
  const chain = createHash("sha256").update(chainInput).digest("hex");
265
+ // Explicit seq: must exceed every existing seq AND every released_seq, so a
266
+ // released row's cursor position never ties with a future row's seq, and
267
+ // every seq ever issued (the counter), so a deleted row's is never reused.
268
+ const nextSeq = this._seqFloor() + 1;
229
269
  const r = this.db.prepare(`
230
- insert into inbox (envelope_id, envelope_sha256, sender, received_at, envelope_json, chain_hash, pulled_by)
231
- values (?, ?, ?, ?, ?, ?, ?)
232
- `).run(envelope.id, envSha, envelope.sender ?? envelope.from ?? "unknown", new Date().toISOString(), JSON.stringify(envelope), chain, pulledBy);
270
+ insert into inbox (seq, envelope_id, envelope_sha256, sender, received_at, envelope_json, chain_hash, pulled_by, quarantined)
271
+ values (?, ?, ?, ?, ?, ?, ?, ?, ?)
272
+ `).run(nextSeq, envelope.id, envSha, sender, new Date().toISOString(), JSON.stringify(envelope), chain, pulledBy,
273
+ quarantined != null ? String(quarantined).slice(0, 100) : null);
274
+ // Keep sqlite_sequence at least this seq, so an OLD-code daemon on the
275
+ // same mailbox (which lets AUTOINCREMENT pick) never ties with this row.
276
+ // Current SQLite already raises it for an explicit rowid; the bump keeps
277
+ // that from being an assumption.
278
+ this.db.prepare("update sqlite_sequence set seq = max(seq, ?) where name = 'inbox'").run(nextSeq);
233
279
  this._setHead("inbox", chain);
234
- return { duplicate: false, seq: Number(r.lastInsertRowid), envelopeId: envelope.id, envelopeSha256: envSha, chainHash: chain };
280
+ return { duplicate: false, seq: nextSeq, envelopeId: envelope.id, envelopeSha256: envSha, chainHash: chain };
235
281
  })();
236
282
  }
237
283
 
@@ -245,9 +291,16 @@ export class Mailbox {
245
291
  return this.db.prepare("select * from outbox where envelope_id = ?").get(envelopeId) ?? null;
246
292
  }
247
293
 
248
- /** Full inbox row by envelope id (VDA message-delivery instrument). */
249
- getInbound(envelopeId) {
250
- return this.db.prepare("select * from inbox where envelope_id = ?").get(envelopeId) ?? null;
294
+ /**
295
+ * Full inbox row by (envelope id, sender) (VDA message-delivery
296
+ * instrument). Without `sender`, the first row stored under the id, which
297
+ * may be any sender's: pass the sender whenever it is known.
298
+ */
299
+ getInbound(envelopeId, sender) {
300
+ if (sender === undefined) {
301
+ return this.db.prepare("select * from inbox where envelope_id = ? order by seq asc limit 1").get(envelopeId) ?? null;
302
+ }
303
+ return this.db.prepare("select * from inbox where envelope_id = ? and sender = ?").get(envelopeId, sender) ?? null;
251
304
  }
252
305
 
253
306
  markOutboundDelivered(envelopeId, ackSha256) {
@@ -275,9 +328,14 @@ export class Mailbox {
275
328
  return this.db.prepare("select * from inbox where quarantined is not null order by seq asc limit ?").all(limit);
276
329
  }
277
330
 
278
- markQuarantined(envelopeId, reason) {
279
- this.db.prepare("update inbox set quarantined = ? where envelope_id = ?")
280
- .run(String(reason).slice(0, 100), envelopeId);
331
+ /** Without `sender`, every row under the id is quarantined (the safe side). */
332
+ markQuarantined(envelopeId, reason, sender) {
333
+ const r = String(reason).slice(0, 100);
334
+ if (sender === undefined) {
335
+ this.db.prepare("update inbox set quarantined = ? where envelope_id = ?").run(r, envelopeId);
336
+ } else {
337
+ this.db.prepare("update inbox set quarantined = ? where envelope_id = ? and sender = ?").run(r, envelopeId, sender);
338
+ }
281
339
  }
282
340
 
283
341
  // ── pillar-group/2 send queue: what the v2 receiver asked to send ─────────
@@ -319,26 +377,100 @@ export class Mailbox {
319
377
  .run(state, new Date().toISOString(), envelopeId, reason ? String(reason).slice(0, 200) : null, id);
320
378
  }
321
379
 
322
- clearQuarantine(envelopeId) {
323
- this.db.prepare("update inbox set quarantined = null where envelope_id = ?").run(envelopeId);
380
+ /**
381
+ * Release a quarantined row (minor 6). The row keeps its original seq (chain
382
+ * order is immutable) but gets released_seq = tip + 1, so it sorts strictly
383
+ * after every row a cursor could already have passed: inboxAfter matches on
384
+ * coalesce(released_seq, seq) > cursor. Released rows appear in release
385
+ * order. A row quarantined again after a release (a pillar-group/2
386
+ * hold-expired drop) can be released again, at a new position above the
387
+ * tip, so readers see it again; a row that is not quarantined is never moved.
388
+ *
389
+ * One immediate transaction: (a) read the tip, (b) set released_seq,
390
+ * (c) bump sqlite_sequence so an OLD-code daemon's next AUTOINCREMENT
391
+ * insert gets a seq strictly above every released_seq. Without (c) the
392
+ * old daemon's insert takes the same number as this release and a named
393
+ * reader that consumed the release never sees the daemon's row.
394
+ * The immediate transaction also serialises concurrent releases.
395
+ *
396
+ * An id is unique only within its sender (Annex B5(a)). Releasing is the
397
+ * unsafe direction, so without `sender` it refuses when two senders hold
398
+ * the id quarantined rather than release another sender's row with it.
399
+ * Returns the number of rows changed (0 = unknown id or not quarantined).
400
+ */
401
+ releaseQuarantined(envelopeId, sender) {
402
+ let changes = 0;
403
+ this.db.transaction(() => {
404
+ if (sender === undefined) {
405
+ const n = this.db.prepare("select count(*) as n from inbox where envelope_id = ? and quarantined is not null").get(envelopeId).n;
406
+ if (n > 1) throw new Error(`releaseQuarantined: ${n} senders hold envelope ${envelopeId} quarantined; name the sender`);
407
+ }
408
+ const tip = this._seqFloor();
409
+ const where = sender === undefined ? "envelope_id = ?" : "envelope_id = ? and sender = ?";
410
+ changes = this.db.prepare(`update inbox set quarantined = null, released_seq = ? where ${where} and quarantined is not null`)
411
+ .run(tip + 1, envelopeId, ...(sender === undefined ? [] : [sender])).changes;
412
+ if (changes > 0) {
413
+ this.db.prepare("update sqlite_sequence set seq = max(seq, ?) where name = 'inbox'")
414
+ .run(tip + 1);
415
+ }
416
+ }).immediate();
417
+ return changes;
324
418
  }
325
419
 
326
- markInboxRead(envelopeId) {
327
- this.db.prepare("update inbox set read_at = ? where envelope_id = ? and read_at is null")
328
- .run(new Date().toISOString(), envelopeId);
420
+ /**
421
+ * pillar-group/2's name for a release: a held message that settles is
422
+ * released like any quarantined row, so a named reader whose cursor already
423
+ * passed its seq still sees it (minor 6 on the 2.1.0 group path).
424
+ */
425
+ clearQuarantine(envelopeId, sender) {
426
+ return this.releaseQuarantined(envelopeId, sender);
329
427
  }
330
428
 
331
- setInboundGroup(envelopeId, groupId) {
332
- this.db.prepare("update inbox set group_id = ? where envelope_id = ?").run(groupId, envelopeId);
429
+ /** Without `sender`, every row under the id is marked read. */
430
+ markInboxRead(envelopeId, sender) {
431
+ const at = new Date().toISOString();
432
+ if (sender === undefined) {
433
+ this.db.prepare("update inbox set read_at = ? where envelope_id = ? and read_at is null").run(at, envelopeId);
434
+ } else {
435
+ this.db.prepare("update inbox set read_at = ? where envelope_id = ? and sender = ? and read_at is null").run(at, envelopeId, sender);
436
+ }
437
+ }
438
+
439
+ /** Without `sender`, every row under the id is tagged. */
440
+ setInboundGroup(envelopeId, groupId, sender) {
441
+ if (sender === undefined) {
442
+ this.db.prepare("update inbox set group_id = ? where envelope_id = ?").run(groupId, envelopeId);
443
+ } else {
444
+ this.db.prepare("update inbox set group_id = ? where envelope_id = ? and sender = ?").run(groupId, envelopeId, sender);
445
+ }
446
+ }
447
+
448
+ /**
449
+ * The highest number any inbox row occupies or ever occupied: every seq and
450
+ * released_seq, and the AUTOINCREMENT counter (which the B5(a) re-key carries
451
+ * and every explicit-seq write bumps). A new seq or released_seq is above it.
452
+ */
453
+ _seqFloor() {
454
+ return this.db.prepare(`select max(
455
+ coalesce((select max(coalesce(released_seq, seq)) from inbox), 0),
456
+ coalesce((select seq from sqlite_sequence where name = 'inbox'), 0)) as n`).get().n;
333
457
  }
334
458
 
335
459
  inboxTipSeq() {
336
- return this.db.prepare("select coalesce(max(seq), 0) as n from inbox").get().n;
460
+ return this.db.prepare("select coalesce(max(coalesce(released_seq, seq)), 0) as n from inbox").get().n;
337
461
  }
338
462
 
339
- /** Non-quarantined rows after `seq`, oldest first — what a reader has not seen. */
463
+ /**
464
+ * Non-quarantined rows after `seq`, oldest first — what a reader has not seen.
465
+ * A released row is seen at its released_seq (>= every live cursor), not its
466
+ * original seq, so no cursor that already passed it skips it.
467
+ */
340
468
  inboxAfter(seq = 0, limit = 500) {
341
- return this.db.prepare("select * from inbox where seq > ? and quarantined is null order by seq asc limit ?").all(seq, limit);
469
+ return this.db.prepare(`
470
+ select * from inbox
471
+ where coalesce(released_seq, seq) > ? and quarantined is null
472
+ order by coalesce(released_seq, seq) asc limit ?
473
+ `).all(seq, limit);
342
474
  }
343
475
 
344
476
  // ── Readers ────────────────────────────────────────────────────────────────
@@ -10,7 +10,9 @@
10
10
  * POST /v1/envelopes
11
11
  * body: envelope JSON (sealed by sender's pillar)
12
12
  * - carrier verifies the transport signature before accepting (fail closed)
13
- * - idempotent on envelope.id
13
+ * - idempotent on (sender, envelope.id), the store's key (Annex B5(a)):
14
+ * a stored pair again -> 202 { accepted: true, duplicate: true, seq, sha };
15
+ * another sender's envelope under the same id is a different envelope
14
16
  * -> 202 { accepted: true, seq, sha }
15
17
  * With a receipt key (constructor `receiptKeychain`), the 202 also carries
16
18
  * `receipt` + `receiptSig` (uuaid.carrier-receipt.v1, ./carrier-receipt.mjs),
@@ -70,6 +72,7 @@ import { verifyTierGrant } from "../identity/tier.mjs";
70
72
  import { verifyDelegationDoc, delegationCoversKind } from "./delegation.mjs";
71
73
  import { withCarrierFix } from "./refusals.mjs"; // additive `fix` on refusals; reason/error/status unchanged
72
74
  import { carrierReceiptSigningString, carrierReceiptShapeOk, carrierUuaidFromKey } from "./carrier-receipt.mjs";
75
+ import { rekeyToSenderUnique } from "../sqlite-rekey.mjs";
73
76
 
74
77
  const MAX_WAIT_S = 30;
75
78
  const MAX_BODY = 512 * 1024; // 512 KiB per envelope (community tier)
@@ -135,6 +138,20 @@ export function parseSecureDelete(value) {
135
138
  throw new Error(`secure-delete must be on or off, not "${value}"`);
136
139
  }
137
140
 
141
+ // An envelope id is unique only within its sender (Annex B5(a), src/sqlite-rekey.mjs).
142
+ const ENVELOPES_COLUMNS = `
143
+ seq integer primary key autoincrement,
144
+ envelope_id text not null,
145
+ recipient text not null,
146
+ sender text not null,
147
+ sha text not null,
148
+ body text not null,
149
+ created_at text not null default (strftime('%Y-%m-%dT%H:%M:%fZ','now')),
150
+ retain_until text,
151
+ envelope_sha text,
152
+ unique (sender, envelope_id)`;
153
+ const ENVELOPES_INDEX = "create index if not exists idx_envelopes_recipient on envelopes(recipient, seq);";
154
+
138
155
  export class CarrierServer {
139
156
  constructor({ dbPath = "carrier.db", ttlDays = 14, cryptoInventory, trustedTierIssuers = [], keychain = null, publicUrl = null, secureDelete = true, receiptKeychain = null, communityInboxWaits = COMMUNITY_CONCURRENT_WAITS } = {}) {
140
157
  // Every row's retain_until is created_at + ttlDays, so the TTL must be a
@@ -158,18 +175,8 @@ export class CarrierServer {
158
175
  this.db.pragma(`secure_delete = ${this.secureDelete ? "ON" : "OFF"}`);
159
176
  this.lastVacuum = null;
160
177
  this.db.exec(`
161
- create table if not exists envelopes (
162
- seq integer primary key autoincrement,
163
- envelope_id text not null unique,
164
- recipient text not null,
165
- sender text not null,
166
- sha text not null,
167
- body text not null,
168
- created_at text not null default (strftime('%Y-%m-%dT%H:%M:%fZ','now')),
169
- retain_until text,
170
- envelope_sha text
171
- );
172
- create index if not exists idx_envelopes_recipient on envelopes(recipient, seq);
178
+ create table if not exists envelopes (${ENVELOPES_COLUMNS});
179
+ ${ENVELOPES_INDEX}
173
180
  -- Lifetime service counters. Kept OUTSIDE the envelopes table because
174
181
  -- vacuum() deletes envelopes at TTL, and a seed's service history must
175
182
  -- survive its own garbage collection.
@@ -188,6 +195,33 @@ export class CarrierServer {
188
195
  if (!cols.has("envelope_sha")) this.db.exec("alter table envelopes add column envelope_sha text");
189
196
  this.db.prepare("insert or ignore into seed_stats (k, v) values ('first_seen', ?)")
190
197
  .run(new Date().toISOString());
198
+ // Secure delete promises an expired envelope leaves the disk; a kept backup
199
+ // would hold every stored envelope past its deadline, so it goes once the
200
+ // re-key has committed.
201
+ const rekey = rekeyToSenderUnique(this.db, { table: "envelopes", columnsSql: ENVELOPES_COLUMNS, indexSql: [ENVELOPES_INDEX], keepBackup: !this.secureDelete });
202
+ if (rekey.migrated) console.warn(`pillar-carrier: envelopes re-keyed on (sender, envelope_id): ${rekey.rows} rows, seq counter ${rekey.counter}, backup ${rekey.backup ?? (this.db.memory ? "none (in-memory)" : "removed (secure delete)")}`);
203
+ if (rekey.swept) console.warn(`pillar-carrier: removed ${rekey.swept} backup file(s) an interrupted re-key left (secure delete)`);
204
+ // The rebuild copied the table and freed the old one, so the file is about
205
+ // twice its size, and the store cap counts every page, free ones too
206
+ // (page_count, the full-refusal check below). Reclaim them once, here: else
207
+ // a carrier above about half its cap refuses deposits for good.
208
+ // Also on a later start: a run killed (or failed) between the re-key's commit and this VACUUM
209
+ // leaves the freed copy on the file (just under half the pages), and only this check notices it.
210
+ // A quarter of the pages free also means the cap counts a quarter more than is stored, so a
211
+ // VACUUM pays its way then too; below that the start skips it.
212
+ const pages = this.db.memory ? 0 : this.db.pragma("page_count", { simple: true });
213
+ const freePages = this.db.memory ? 0 : this.db.pragma("freelist_count", { simple: true });
214
+ if (!this.db.memory && (rekey.migrated || freePages * 4 > pages)) {
215
+ try {
216
+ this.db.exec("vacuum");
217
+ // Another connection reading this file holds the WAL: the checkpoint
218
+ // waits out the busy timeout (5 s), then leaves the WAL as it is.
219
+ const [cp] = this.db.pragma("wal_checkpoint(TRUNCATE)");
220
+ if (cp?.busy) console.warn(`pillar-carrier: free pages reclaimed, but another process holds ${dbPath} open, so its WAL was not truncated; stop other processes on the store before restarting the carrier`);
221
+ } catch (e) {
222
+ console.warn(`pillar-carrier: VACUUM failed (${e.message}); the store counts its free pages (${freePages} of ${pages}) toward the size cap until a later start reclaims them, or the carrier is stopped and \`sqlite3 ${dbPath} 'VACUUM;'\` is run`);
223
+ }
224
+ }
191
225
  this.ttlDays = ttlDays;
192
226
  this._backfillRetention(); // @rule carrier.retention-backfill
193
227
  this.cryptoInventory = cryptoInventory ?? { ed25519: "active" };
@@ -522,7 +556,11 @@ export class CarrierServer {
522
556
  return { code: 202, body };
523
557
  } catch (e) {
524
558
  if (/UNIQUE constraint/.test(e.message)) {
525
- const row = this.db.prepare(`select ${ROW_COLUMNS}, body from envelopes where envelope_id = ?`).get(envelope.id);
559
+ // The key is (sender, envelope_id), so this lookup names BOTH. By id
560
+ // alone it could find another sender's envelope and answer this one
561
+ // with that seq and sha as if it were stored (Annex B5(a)).
562
+ const row = this.db.prepare(`select ${ROW_COLUMNS}, body from envelopes where envelope_id = ? and sender = ?`).get(envelope.id, envelope.sender);
563
+ if (!row) throw e;
526
564
  // No receipt key: the duplicate answer is exactly the pre-receipt one.
527
565
  if (!this.receiptKeychain) return { code: 202, body: { accepted: true, duplicate: true, seq: row.seq, sha: row.sha } }; // @rule carrier.keyless-unchanged
528
566
  // §12 rule 1 (P1): the same id with a different body is not a
package/src/pillar.mjs CHANGED
@@ -682,7 +682,8 @@ export class Pillar {
682
682
  * dropped: nothing leaves this process from here.
683
683
  */
684
684
  async _receiveGroupV2(envelope, payload) {
685
- const env = { from: envelope.sender, transportKey: envelope.transportSignature?.publicKey, payload, ref: envelope.id };
685
+ // ref is this process's handle on a held message: (id, sender), the inbox key (Annex B5(a)).
686
+ const env = { from: envelope.sender, transportKey: envelope.transportSignature?.publicKey, payload, ref: { id: envelope.id, sender: envelope.sender } };
686
687
  if (envelope.delegation) env.delegation = { rootKey: envelope.delegation.principalPublicKey };
687
688
  const r = this.groupsV2.receive(env, Date.now());
688
689
  this.metrics.inc(`received.group-v2.${r.verdict}`);
@@ -695,15 +696,15 @@ export class Pillar {
695
696
  if (ids.length < r.sends.length) this.metrics.inc("group-v2.sends.queue-full", r.sends.length - ids.length);
696
697
  }
697
698
  for (const x of r.released) await this._settleHeldGroupV2(x.ref, x.verdict, x.reason, groupId);
698
- for (const x of r.dropped) if (x.ref) this.mailbox.markQuarantined(x.ref, `group-v2:${x.reason}`);
699
+ for (const x of r.dropped) if (x.ref) this.mailbox.markQuarantined(x.ref.id, `group-v2:${x.reason}`, x.ref.sender);
699
700
  if (["accepted", "invitation", "recovered"].includes(r.verdict)) {
700
- this.mailbox.setInboundGroup(envelope.id, groupId);
701
+ this.mailbox.setInboundGroup(envelope.id, groupId, envelope.sender);
701
702
  return true;
702
703
  }
703
704
  if (r.verdict === "warning") return true;
704
705
  // Verdict "v1" (a v1 tag for a group the store does not hold) reaches here only
705
706
  // for a marked row whose store file is gone: no verified chain, nothing to file under.
706
- this.mailbox.markQuarantined(envelope.id, `group-v2:${r.verdict === "v1" ? "no-chain" : (r.reason ?? r.verdict)}`);
707
+ this.mailbox.markQuarantined(envelope.id, `group-v2:${r.verdict === "v1" ? "no-chain" : (r.reason ?? r.verdict)}`, envelope.sender);
707
708
  return false;
708
709
  }
709
710
 
@@ -779,12 +780,13 @@ export class Pillar {
779
780
  return out;
780
781
  }
781
782
 
782
- async _settleHeldGroupV2(envelopeId, verdict, reason, groupId) {
783
- if (!envelopeId) return;
784
- if (verdict !== "accepted") { this.mailbox.markQuarantined(envelopeId, `group-v2:${reason ?? verdict}`); return; }
785
- this.mailbox.clearQuarantine(envelopeId);
786
- this.mailbox.setInboundGroup(envelopeId, groupId);
787
- const row = this.mailbox.getInbound(envelopeId);
783
+ async _settleHeldGroupV2(ref, verdict, reason, groupId) {
784
+ if (!ref?.id) return;
785
+ const { id: envelopeId, sender } = ref;
786
+ if (verdict !== "accepted") { this.mailbox.markQuarantined(envelopeId, `group-v2:${reason ?? verdict}`, sender); return; }
787
+ this.mailbox.clearQuarantine(envelopeId, sender);
788
+ this.mailbox.setInboundGroup(envelopeId, groupId, sender);
789
+ const row = this.mailbox.getInbound(envelopeId, sender);
788
790
  if (!row) return;
789
791
  const envelope = JSON.parse(row.envelope_json);
790
792
  let payload;
@@ -913,7 +915,10 @@ export class Pillar {
913
915
  dead.delete(base);
914
916
  for (const { seq, envelope } of r.envelopes) {
915
917
  since = Math.max(since, seq);
916
- if (found.has(envelope.id) || this.mailbox.getInbound(envelope.id)) continue;
918
+ // Keyed like the inbox, on (sender, id): another sender's envelope
919
+ // under this id must not hide this one (Annex B5(a)).
920
+ const key = `${envelope.sender}\u0000${envelope.id}`;
921
+ if (found.has(key) || this.mailbox.getInbound(envelope.id, envelope.sender)) continue;
917
922
  const v = open(envelope, { cryptoInventory: this.cryptoInventory });
918
923
  if (!v.ok || envelope.recipient !== this.uuaid) continue;
919
924
  let payload;
@@ -921,7 +926,7 @@ export class Pillar {
921
926
  if (!this._trustVerdict(envelope, payload).deliver) continue;
922
927
  const view = { ...envelope, payload };
923
928
  if (match && !match(view)) continue;
924
- found.set(envelope.id, view);
929
+ found.set(key, view);
925
930
  }
926
931
  if (found.size) { stop = true; ctl.abort(); }
927
932
  else if (w === 0 && !r.envelopes.length) {
@@ -965,22 +970,33 @@ export class Pillar {
965
970
  console.warn("pillar: envelope decrypt failed (dropping)", envelope.id, e.message);
966
971
  return;
967
972
  }
968
- const persisted = this.mailbox.appendInbound(envelope, { pulledBy: this.reader });
969
- if (persisted.duplicate) {
970
- this.metrics.inc("received.duplicate");
971
- return; // replay: do not re-fire handlers
972
- }
973
- this.metrics.inc(`received.kind.${envelope.kind}`);
974
973
  // Trust gate (recipient consent): quarantine-not-drop. The envelope is
975
- // already persisted (chain-hashed evidence); a gated sender just never
974
+ // persisted either way (chain-hashed evidence); a gated sender just never
976
975
  // reaches handlers. An ack/reply (or a read receipt) passes the gate ONLY
977
976
  // when it answers an envelope in OUR outbox that was sent TO ITS SENDER —
978
977
  // a genuine receipt for our own send. (kind alone is spoofable, and so is
979
978
  // an envelope id: ids are visible to every carrier. Sent-to-this-sender
980
- // is not.)
979
+ // is not.) Compute the verdict FIRST so insert and quarantine are one
980
+ // commit — a named reader in another process can never observe the row
981
+ // unquarantined between the two writes.
981
982
  const verdict2 = this._trustVerdict(envelope, payload);
983
+ const persisted = this.mailbox.appendInbound(envelope, {
984
+ pulledBy: this.reader,
985
+ quarantined: verdict2.deliver ? null : verdict2.reason,
986
+ });
987
+ if (persisted.collision) {
988
+ // Same sender, same id, different envelope: not stored, not handled.
989
+ this.metrics.inc("received.collision");
990
+ // eslint-disable-next-line no-console
991
+ console.warn(`pillar: dropped ${envelope.id} from ${envelope.sender}: a different envelope from that sender already holds this id`);
992
+ return;
993
+ }
994
+ if (persisted.duplicate) {
995
+ this.metrics.inc("received.duplicate");
996
+ return; // replay: do not re-fire handlers
997
+ }
998
+ this.metrics.inc(`received.kind.${envelope.kind}`);
982
999
  if (!verdict2.deliver) {
983
- this.mailbox.markQuarantined(envelope.id, verdict2.reason);
984
1000
  this.metrics.inc(`received.quarantined.${verdict2.reason}`);
985
1001
  // eslint-disable-next-line no-console
986
1002
  console.warn(`pillar: quarantined ${envelope.id} from ${envelope.sender} (${verdict2.reason})`);
@@ -1011,7 +1027,7 @@ export class Pillar {
1011
1027
  const fromContact = !!this.addressBook?.get(envelope.sender) || !!this.trust?.allow?.includes(envelope.sender);
1012
1028
  const before = this.mailbox.getGroup(tag.id);
1013
1029
  const after = absorbGroupTag(this.mailbox, tag, { sender: envelope.sender, self: this.uuaid, text: payload?.text ?? "", fromContact });
1014
- if ((before ?? after)?.members.includes(envelope.sender)) this.mailbox.setInboundGroup(envelope.id, tag.id);
1030
+ if ((before ?? after)?.members.includes(envelope.sender)) this.mailbox.setInboundGroup(envelope.id, tag.id, envelope.sender);
1015
1031
  else this.metrics.inc("received.group.non-member");
1016
1032
  }
1017
1033
  if (this.receipts.delivered && envelope.kind !== "ack") await this._autoAck(envelope, payload, tag);
package/src/readers.mjs CHANGED
@@ -21,6 +21,7 @@
21
21
  */
22
22
  import { createHash } from "node:crypto";
23
23
  import { hostname, userInfo } from "node:os";
24
+ import { execFileSync } from "node:child_process";
24
25
 
25
26
  export const READER_NAME_RE = /^[a-z0-9][a-z0-9._-]{0,63}$/i;
26
27
  export const IMPLICIT_PREFIX = "ctx:";
@@ -78,6 +79,41 @@ export function otherLiveLeases(mailbox, ctx = processContext()) {
78
79
  });
79
80
  }
80
81
 
82
+ /**
83
+ * Processes (other than this one) that have this mailbox file open RIGHT NOW,
84
+ * via lsof. This catches what the lease scan cannot: a pre-lease daemon
85
+ * holding the DB open but not yet writing, or a one-shot `pillar inbox` run
86
+ * with no lease at all. Returns [] when lsof is unavailable (not a mac/linux
87
+ * host) — the lease check still applies as a floor.
88
+ */
89
+ export function processesWithMailboxOpen(mailboxDbPath) {
90
+ let out;
91
+ try {
92
+ out = execFileSync("lsof", ["-F", "pc", mailboxDbPath], { encoding: "utf-8", timeout: 5000, stdio: ["ignore", "pipe", "ignore"] });
93
+ } catch (e) {
94
+ // exit 1 with empty stdout = nobody has the file open (normal case)
95
+ if (e.status === 1 && !e.stdout?.toString().trim()) return [];
96
+ // exit 1 WITH holders on stdout: lsof found holders but also errored on
97
+ // another search item. Parse what it found.
98
+ if (e.status === 1 && e.stdout?.toString().trim()) { out = e.stdout.toString(); }
99
+ else {
100
+ // anything else (lsof missing, timeout, unexpected error): warn, continue on lease floor
101
+ console.error(`warning: the open-file check (lsof) could not run: ${e.message}. init --force will rely on the lease check only.`);
102
+ return [];
103
+ }
104
+ }
105
+ // lsof -F pc output per open file: p<pid>\n c<command>\n f<fd>\n
106
+ // (the f line follows c; we pair p with the c that immediately follows it)
107
+ const procs = [];
108
+ let pid = null;
109
+ for (const line of out.split("\n")) {
110
+ if (line.startsWith("p")) { pid = parseInt(line.slice(1), 10); }
111
+ else if (line.startsWith("c") && pid != null) { procs.push({ pid, command: line.slice(1) }); pid = null; }
112
+ // f<fd> and other field lines: ignored; pid already consumed or not yet seen
113
+ }
114
+ return procs.filter((p) => p.pid !== process.pid);
115
+ }
116
+
81
117
  /** "pillar recv (pid 4242 on mac, /work/bot, since 2026-09-28T…)" */
82
118
  export function describeProcess(row) {
83
119
  const where = [row.pid != null ? `pid ${row.pid}` : null, row.host ? `on ${row.host}` : null, row.cwd ?? null]
@@ -0,0 +1,190 @@
1
+ /**
2
+ * Re-key an envelope table from UNIQUE(envelope_id) to UNIQUE(sender,
3
+ * envelope_id): UACP charter Annex B5(a).
4
+ *
5
+ * WHY. An envelope id is chosen by its sender, so it is unique only within
6
+ * one sender. Keyed on the id alone, a store lets any sender who learns (or
7
+ * guesses) another sender's next id claim it first; the real envelope then
8
+ * reads as a duplicate and is never stored. Keyed on (sender, id), each
9
+ * sender can collide only with itself and its live delegates, of any kind: a
10
+ * delegate's envelope carries its principal as sender (RedTeam G1).
11
+ *
12
+ * HOW. SQLite cannot drop a table constraint in place, so this is its
13
+ * documented rebuild: create the new table, copy every row with its own seq,
14
+ * drop the old one, rename. A naive rebuild loses two things, and this keeps
15
+ * both:
16
+ *
17
+ * 1. The AUTOINCREMENT counter. sqlite_sequence holds the highest seq ever
18
+ * issued, which is above max(seq) whenever rows were deleted (a carrier's
19
+ * TTL vacuum). A rebuild resets it to max(seq), and the next insert would
20
+ * reuse a seq that clients already read past with `since=<seq>`: that
21
+ * mail would never be delivered. The counter is carried over as
22
+ * max(old counter, max(seq)).
23
+ * 2. The data if anything fails: a file-backed database is copied first with
24
+ * VACUUM INTO, and the rebuild is one transaction.
25
+ *
26
+ * GUARD. The table itself: a table that still has an id-only UNIQUE is
27
+ * rebuilt, whatever `user_version` says, because that number is one per FILE
28
+ * and two envelope tables can share a file (an inbox and a carrier's
29
+ * envelopes): the first one's re-key must not skip the second's. A table
30
+ * without it (a fresh one, created with the new key, or one already rebuilt)
31
+ * is left alone. Either way `user_version` ends at least SENDER_UNIQUE_VERSION
32
+ * and is never lowered.
33
+ *
34
+ * BACKUPS. A kept backup (`keepBackup`) is the copy an operator restores
35
+ * from, so it stays when the rebuild throws. Under secure delete
36
+ * (`keepBackup: false`) no copy may outlive the call: it goes whether the
37
+ * rebuild committed, found the work done or threw, and every start sweeps the
38
+ * copies a killed run left behind (staleBackupFiles says which: never one a
39
+ * running re-key is still writing).
40
+ * The sweep matches every re-key backup of the FILE, so a mailbox sharing a
41
+ * file with a secure-delete carrier loses its kept backup to that sweep.
42
+ *
43
+ * THE VERSION LADDER (a convention every later change must follow).
44
+ * `user_version` is one integer for the whole database file, and this re-key
45
+ * now owns it for both envelope databases, carrier.db and mailbox.db. It is a
46
+ * single monotonic ladder starting at 1 (SENDER_UNIQUE_VERSION). Any later
47
+ * schema change to either file MUST:
48
+ * - take the next number (2, 3, …);
49
+ * - guard on `user_version >= <its own number>`;
50
+ * - stamp only after its own change has applied.
51
+ * A change that reused 1, or guarded on `>= 1`, would be skipped silently on
52
+ * every store this re-key has touched. When a second change arrives, a
53
+ * dedicated migrations table becomes worth its keep. Until then this
54
+ * paragraph is the rule (RedTeam, fm-20260928T061937-2986f6).
55
+ */
56
+
57
+ import { chmodSync, closeSync, openSync, readdirSync, rmSync, statSync } from "node:fs";
58
+ import { basename, dirname, join } from "node:path";
59
+
60
+ export const SENDER_UNIQUE_VERSION = 1;
61
+
62
+ /** True if `table` has a UNIQUE index over exactly (envelope_id). */
63
+ function hasIdOnlyUnique(db, table) {
64
+ for (const ix of db.pragma(`index_list(${table})`)) {
65
+ if (!ix.unique) continue;
66
+ const cols = db.pragma(`index_info(${ix.name})`).map((c) => c.name);
67
+ if (cols.length === 1 && cols[0] === "envelope_id") return true;
68
+ }
69
+ return false;
70
+ }
71
+
72
+ /** True if `pid` names a live process (EPERM: alive, not ours to signal). */
73
+ function alive(pid) {
74
+ try { process.kill(pid, 0); return true; } catch (e) { return e.code === "EPERM"; }
75
+ }
76
+
77
+ // A copy is written in seconds (a store is capped at 256 MiB), so one this old
78
+ // is no opener's copy in progress, whatever its pid now names.
79
+ const STALE_BACKUP_MS = 60 * 60 * 1000;
80
+
81
+ /**
82
+ * `<db>.pre-sender-unique-<ms>-<pid>.bak`, and the journal VACUUM INTO may
83
+ * leave beside it, that no running re-key is still writing. A copy another
84
+ * opener is still writing must stay: deleting it fails that opener's start
85
+ * (SQLITE_IOERR_DELETE_NOENT). So a copy is stale when its process is gone, or
86
+ * it is older than STALE_BACKUP_MS, or it carries THIS process's pid but
87
+ * predates this process: a carrier restarted in a container gets the same pid
88
+ * (often 1) as the killed run that left the copy (Uuaid-Codex X-1). A copy
89
+ * with this pid written since this process started is a worker thread's, and
90
+ * stays.
91
+ */
92
+ function staleBackupFiles(dbFile) {
93
+ const prefix = `${basename(dbFile)}.pre-sender-unique-`;
94
+ const name = /^(\d+)-(\d+)\.bak(-journal|-wal|-shm)?$/;
95
+ const now = Date.now();
96
+ const startedAt = now - process.uptime() * 1000;
97
+ return readdirSync(dirname(dbFile))
98
+ .filter((f) => {
99
+ if (!f.startsWith(prefix)) return false;
100
+ const m = name.exec(f.slice(prefix.length));
101
+ if (m === null) return false;
102
+ const ms = Number(m[1]), pid = Number(m[2]);
103
+ if (now - ms > STALE_BACKUP_MS) return true;
104
+ if (pid === process.pid) return ms < startedAt;
105
+ return !alive(pid);
106
+ })
107
+ .map((f) => join(dirname(dbFile), f));
108
+ }
109
+
110
+ function removeBackup(backup) {
111
+ for (const f of [backup, `${backup}-journal`]) rmSync(f, { force: true });
112
+ }
113
+
114
+ /**
115
+ * Rebuild `table` as `create table <table> (<columnsSql>)` if it still has
116
+ * the id-only key. `columnsSql` must declare every column the old table has
117
+ * (the copy names the old table's columns) plus `unique (sender,
118
+ * envelope_id)`. `indexSql` recreates the table's secondary indexes, which
119
+ * the drop removes. `table` is a code constant, never input.
120
+ *
121
+ * Call it OUTSIDE any transaction: the backup is VACUUM INTO, which SQLite
122
+ * refuses inside one. The rebuild then takes its own IMMEDIATE transaction
123
+ * and re-checks the guard there, so when several processes open one file at
124
+ * once only the first rebuilds; a later one finds the work done and removes
125
+ * the backup it took.
126
+ *
127
+ * `keepBackup: false` removes the backup in every outcome and sweeps the ones
128
+ * earlier runs left (BACKUPS above). A carrier that runs with secure delete
129
+ * passes it: a copy would otherwise keep envelopes on disk past their
130
+ * retention deadline.
131
+ *
132
+ * Returns { migrated: false, swept } or { migrated: true, backup, rows,
133
+ * counter, swept }; `swept` counts the stale backup files removed.
134
+ */
135
+ export function rekeyToSenderUnique(db, { table, columnsSql, indexSql = [], keepBackup = true }) {
136
+ const stamped = () => db.pragma("user_version", { simple: true }) >= SENDER_UNIQUE_VERSION;
137
+ const stamp = () => { if (!stamped()) db.pragma(`user_version = ${SENDER_UNIQUE_VERSION}`); };
138
+ const sweep = () => (keepBackup || db.memory ? 0 : staleBackupFiles(db.name).map((f) => rmSync(f, { force: true })).length);
139
+ if (!hasIdOnlyUnique(db, table)) {
140
+ // No write lock on the common path: several processes open one home at once.
141
+ if (!stamped()) db.transaction(stamp).immediate();
142
+ return { migrated: false, swept: sweep() };
143
+ }
144
+ let backup = null;
145
+ if (!db.memory) {
146
+ backup = `${db.name}.pre-sender-unique-${Date.now()}-${process.pid}.bak`;
147
+ // VACUUM INTO fills an empty file it finds, so create it first with the
148
+ // live file's mode: a copy of a 0600 store must not be readable by others.
149
+ closeSync(openSync(backup, "wx", 0o600));
150
+ chmodSync(backup, statSync(db.name).mode & 0o777);
151
+ try { db.prepare("vacuum into ?").run(backup); }
152
+ catch (e) { removeBackup(backup); throw e; }
153
+ }
154
+ const tmp = `${table}__rekey`;
155
+ let result = null;
156
+ try {
157
+ result = db.transaction(() => {
158
+ // Another opener may have finished while this one was copying.
159
+ if (!hasIdOnlyUnique(db, table)) {
160
+ stamp();
161
+ return { migrated: false };
162
+ }
163
+ const cols = db.pragma(`table_info(${table})`).map((c) => `"${c.name}"`).join(", ");
164
+ const oldCounter = db.prepare("select seq from sqlite_sequence where name = ?").get(table)?.seq ?? 0;
165
+ db.exec(`create table ${tmp} (${columnsSql})`);
166
+ db.exec(`insert into ${tmp} (${cols}) select ${cols} from ${table}`);
167
+ db.exec(`drop table ${table}`);
168
+ db.exec(`alter table ${tmp} rename to ${table}`);
169
+ for (const sql of indexSql) db.exec(sql);
170
+ const { rows, maxSeq } = db.prepare(`select count(*) as rows, coalesce(max(seq), 0) as maxSeq from ${table}`).get();
171
+ const counter = Math.max(oldCounter, maxSeq);
172
+ // sqlite_sequence has no key on `name`, so "insert or replace" would add
173
+ // a second row. Delete, then insert the one row.
174
+ db.prepare("delete from sqlite_sequence where name = ?").run(table);
175
+ db.prepare("insert into sqlite_sequence (name, seq) values (?, ?)").run(table, counter);
176
+ stamp();
177
+ return { migrated: true, backup, rows, counter };
178
+ }).immediate();
179
+ } finally {
180
+ // `result` is null only if the rebuild threw: then a kept backup stays,
181
+ // and under secure delete this call's own copy still goes. Copies older
182
+ // runs left are swept below, on a start that gets this far.
183
+ if (backup && (!keepBackup || (result && !result.migrated))) {
184
+ removeBackup(backup);
185
+ if (result?.migrated) result.backup = null;
186
+ }
187
+ }
188
+ result.swept = sweep();
189
+ return result;
190
+ }
@@ -122,9 +122,12 @@ export function attestMessageDelivery({ claim: claimRaw, mailbox, mailboxOwner,
122
122
  });
123
123
  }
124
124
  // Full sight within the recipient's own inbox: absent = failed (§10.6 satisfied).
125
- const row = mailbox.getInbound(claim.envelopeId);
125
+ // By (id, sender): the same id from another sender is another envelope,
126
+ // and must not stand in for (or hide) the claimed sender's.
127
+ const row = mailbox.getInbound(claim.envelopeId, claim.from);
126
128
  if (!row) {
127
- return buildAttestation({ claim, verdict: "failed", reason: "not-in-recipient-inbox", coverage: coverageInbox, keychain });
129
+ const reason = mailbox.getInbound(claim.envelopeId) ? "sender-mismatch" : "not-in-recipient-inbox";
130
+ return buildAttestation({ claim, verdict: "failed", reason, coverage: coverageInbox, keychain });
128
131
  }
129
132
  if (row.sender !== claim.from) {
130
133
  return buildAttestation({ claim, verdict: "failed", reason: "sender-mismatch", coverage: coverageInbox, keychain });