trajectories 0.0.0 → 0.1.1

package/docs/research/ledger/docs/research/2026-09-16-c8-remote-truth.md

@@ -0,0 +1,390 @@
+# C8: remote truthfulness primitives, 2026-09-16
+
+## Outcome
+
+The failure semantics a sync surface must obey now exist as code with an owner:
+`packages/core/src/sync-truth.ts` — a typed outcome vocabulary, a pure state
+machine, the dialect data for transfer error lines, and the stamp/disclosure
+contract — plus `packages/core/tests/sync-truth.test.ts` (25 cases, 435
+assertions). No transport ships, and none is needed to use any of it.
+
+The sentence this cycle exists to make impossible is *"it didn't answer, so it
+must be empty."* Five rules do that work, and each one is enforced by a shape
+rather than by a comment:
+
+1. **`unreachable` and `empty` are different kinds, and the difference is
+   structural.** `empty` is produced only by a probe that exited 0 and found
+   none of the whitelist present; a probe that failed (spawn failure, non-zero
+   exit, signal) settles `unreachable` *before* any content claim is
+   constructible. The mirror action states the same fact mechanically:
+   `keep-untouched` is exactly the no-reached-probe case, so a driver cannot be
+   handed a mirror deletion it did not earn with a probe that reached.
+2. **`partial` is not `ok`.** Exit 24 (a source vanished while the transfer read
+   it) settles `partial`: the mirror keeps what transferred — Wake's rule — and
+   the stamp action is `record-error`, so `last_sync_at` goes on naming the last
+   sync that *completed*. This is the cycle's one deliberate tightening of Wake,
+   which stamps a clean success for 24 (`remote.rs#L407-L419`,
+   `db.rs#L864-L877`); "partial ≠ complete" is only enforceable if the persisted
+   record says it.
+3. **The probe decides what the transfer may touch.** A transfer directive
+   carries exactly the probe-confirmed present set, and absent paths are purged
+   between the two steps. The full whitelist never reaches a transfer — the
+   hazard Wake measured on this host (openrsync truncating its file list on a
+   missing source's parent, exit 23 only; any sender-side I/O error suppressing
+   the delete phase, `remote.rs#L13-L22`).
+4. **Dialects are data.** GNU and openrsync live as rows in `RSYNC_DIALECTS`
+   (error-line shape, vanished-source shape, summary lines); `rsyncStderrShape`
+   is the single dialect-agnostic algorithm over that table, and a shape the
+   table does not know is not evidence either way.
+5. **An empty source is a hypothesis only a successful probe may confirm, and an
+   unreachable source is not a fact about content at all.** This is the whole
+   vocabulary's ordering: reach first, then content.
+
+Everything in the module is pure except `readSyncStamps`, which reads a table
+this build deliberately does not create (decision 6 below). Nothing spawns a
+process, nothing touches a filesystem, no CLI verb, no store write, no
+scheduler, and no new dependency; the only import is a `Database` *type*.
+
+### Evidence, and what kind it is
+
+**OBSERVED** (in this tree's output, this cycle):
+
+- the case suite and the sweep — `bun test tests/sync-truth.test.ts`: 25 pass /
+  0 fail, 435 assertions;
+- the whole suite with this cycle's additions: 626 pass / 0 fail, 4,742
+  assertions, 43 files;
+- `bun run typecheck` and `bun run build`: clean;
+- the api contract test that pins the exported key list: 19 pass / 0 fail.
+
+**INFERRED** (from Wake's source, or from the shapes here — not run):
+
+- the dialect table's coverage limit (an unknown error shape is invisible to
+  `rsyncStderrShape`; no unknown-dialect rsync was executed here);
+- the claim that a future transport driver can satisfy the state machine's
+  contract — no driver exists yet, and the two `INFERRED` items in *Still open*
+  are exactly where that could bite.
+
+**Not done at all:** no ssh or rsync process was spawned, no transport was
+written, no filesystem mirror was created or deleted, no store — production or
+temporary — was opened by any shipped path, and no timing or scale measurement
+was taken. Nothing in this note should be read as one.
+
+## The vocabulary: kind, claim, mirror, stamp
+
+| kind | what it claims | mirror (`cache`) | stamp |
+| --- | --- | --- | --- |
+| `ok` | reached; every probe-confirmed path transferred | `purge(landed)` — possibly empty | `update-stamp` |
+| `empty` | reached; holds none of the whitelist | `purge(landed)` — the round's whole absent set when there was one, its purge having succeeded | `update-stamp` |
+| `partial` | reached; a subset transferred, missing exactly the vanished sources | `purge(landed)` | `record-error` |
+| `unreachable` | **no contact in the round that settled**; that round wrote nothing | `keep-untouched` when no round reached; else `purge(landed)` | `record-error` |
+| `refused` | the attempt stopped short of a mirror state this store can vouch for (bad name, missing tool, failed mirror deletion, non-vanished transfer error) | `keep-untouched` when nothing reached; else `purge(landed)` — possibly empty when the licensed deletion failed | `record-error` |
+
+`landed` is the union of the absent sets of purges that **reported success**:
+the record names mirror entries that are really gone. A purge that failed joins
+nothing — the paths it was licensed to delete stay visible as the outcome's
+`absent` (a reached round's fact, deleted or not), and the account names the
+filesystem failure. A mirror deletion that cannot land is `refused`, never a
+quiet success: the remote no longer holds those sessions, and reporting a clean
+sync would leave them in the cache and the index.
+
+The reasons are a closed set (`host_name_invalid`, `probe_tool_unavailable`,
+`probe_failed`, `source_empty`, `transferred`, `sources_vanished`,
+`mirror_purge_failed`, `transfer_tool_unavailable`, `transfer_failed`), because a
+disclosure has to name why, and free text cannot be counted or translated.
+
+Two invariants are asserted over every scripted run, not merely documented:
+
+- **Reach licenses the mirror.** `cache.kind === "keep-untouched"` exactly when
+  no probe in the attempt reached the source; once one did (detected
+  independently, by the presence of a `purge`/`transfer` directive), the action
+  is `purge` with the deletions that landed, and `present ∪ absent` partitions
+  the whitelist.
+- **Completeness owns the stamp.** `update-stamp` exactly for `ok`/`empty`;
+  `record-error` for everything else, with a non-null account (the reason code is
+  the account of last resort, never an empty error).
+
+## The state machine, step by step
+
+`beginSyncAttempt(source, paths)` → a name that cannot be addressed settles
+`refused/host_name_invalid` with **zero directives** (nothing runs); a valid one
+asks for `probe`. `advanceSyncAttempt(state, observation)` is the only
+transition, and an observation that does not match the stage the attempt waits
+on **throws** — a driver bug must not be absorbed into an outcome the store then
+believes.
+
+| stage | observation | next |
+| --- | --- | --- |
+| `probe` | spawn failure | settle `unreachable`/`probe_tool_unavailable`, mirror untouched |
+| `probe` | exit ≠ 0 (or signal) | settle `unreachable`/`probe_failed`, detail = `<tool> exited with N: <3-line tail>` |
+| `probe` | exit 0, some paths absent | `purge` directive with the absent set (`remote.rs#L345-L349`) |
+| `probe` | exit 0, nothing present | settle `empty`/`source_empty` via the purge, `update-stamp` |
+| `probe` | exit 0, all present | `transfer` directive with the present set |
+| `purge` | removed, present non-empty | `transfer` directive |
+| `purge` | removed, present empty | settle `empty`/`source_empty` |
+| `purge` | failed | settle `refused`/`mirror_purge_failed`, with nothing joining the landed set and the licensed paths left on `absent` (a remote deletion that cannot propagate must not report as a clean sync, `remote.rs#L361-L387`) |
+| `transfer` | exit 0 | settle `ok`/`transferred` |
+| `transfer` | spawn failure | settle `refused`/`transfer_tool_unavailable` |
+| `transfer` | vanished sources, code 23, round 1 | **one** re-probe: back to `probe`, round 2 (`remote.rs#L355-L359`) |
+| `transfer` | vanished sources otherwise | settle `partial`/`sources_vanished` |
+| `transfer` | anything else | settle `refused`/`transfer_failed` |
+
+The round's facts belong to the round: the retry clears `present`/`absent` but
+keeps `reached` and the deletions that already landed, so a round-2 probe that
+cannot reach settles `unreachable` with the first round's landed deletions named
+and no content claim at all.
+
+## The dialects, and where this stops short
+
+GNU and openrsync, pinned from Wake's own fixtures (`remote.rs#L478-L496`):
+
+| dialect | error line | vanished source | summaries (never errors) |
+| --- | --- | --- | --- |
+| GNU | `rsync: [sender] … failed|error …` | `… failed: No such file or directory (2)` | `some files/attrs were not transferred`; `some files vanished before they could be transferred` |
+| openrsync | `rsync(pid): error: …` | `…: (l)stat: No such file or directory` | — |
+
+The summary exclusion is load-bearing, not tidiness: GNU's exit-23 summary
+contains "errors", so without it a vanished transfer would read as an
+unaccounted error and be refused (`remote.rs#L421-L431`).
+
+The exit-code rule is **evidence agreement**, and it differs from Wake in exactly
+one place:
+
+- `0` → complete. Warnings on stderr do not change that.
+- `24` → vanished sources, unless an error line contradicts the code's own claim.
+  Wake accepts 24 unconditionally; the tightening here is that a 24 *with* a
+  non-vanished error line is refused, because a partial-transfer claim the
+  evidence contradicts is precisely what a truthfulness primitive exists to
+  refuse.
+- `23` → vanished sources only on unanimous vanished-source evidence *and at
+  least one error line*. No lines at all is "shape unknown" and stays a failure
+  — Wake's own refusal, kept.
+- Anything else → failure.
+
+Known limit, recorded rather than hidden: only lines the dialect table
+recognizes as errors count as evidence. A permission error printed in a shape
+the table does not know is invisible to the classifier — for exit 23 that still
+lands on `failed`, but for exit 24 it would be read as vanished. Adding the shape
+is one row in `RSYNC_DIALECTS` and nothing else; `INFERRED` (from the table's
+construction; no unknown-dialect rsync was run here).
+
+## Ported mechanics (Wake MIT, `iAmCorey/Wake@d9f702b6df0e85` v0.6.5 — mechanics with attribution, no code copied)
+
+| Ledger | Wake pin | What was kept |
+| --- | --- | --- |
+| probe → purge absent → transfer order; the transfer directive carries only probe-confirmed paths | `crates/wake-core/src/remote.rs#L13-L22` (measured hazard), `#L333-L354` | the order, and the reason: a missing source can silently truncate a transfer's file list (exit 23 only) and any sender I/O error suppresses the delete phase |
+| probe stdout filtered to exact whitelist lines, in whitelist order | `remote.rs#L264-L275`, `#L277-L281`, test `#L594-L604` | MOTD/rc echo/parents/children/absolute paths cannot become sources; a duplicated line is still one path |
+| probe failure fails the whole attempt, mirror untouched | `remote.rs#L333-L354` | "cannot treat cannot-connect as the remote is empty" — here it is the `unreachable` kind plus `keep-untouched` |
+| exit 0 only for the probe; tail-of-stderr account | `remote.rs#L388-L394`, `#L452-L471` | last three lines joined, capped at 500 chars, signal death spelled out |
+| one re-probe when sources vanished between probe and transfer | `remote.rs#L355-L359` | exactly one; the second round's facts settle as-is |
+| vanished-source detection across two dialects; summary lines excluded; an evidence-free 23 not reclassified | `remote.rs#L421-L431`, `#L478-L496` | the shapes, moved into a data table; the unanimity rule |
+| a mirror deletion that fails is a sync failure | `remote.rs#L361-L387`, `#L499-L521` | absence is success, an error is `refused/mirror_purge_failed` |
+| disposable mirror: orphan purge re-asserted after every round, because an in-flight transfer cannot be cancelled and would write a removed tree back | `remote.rs#L202-L222` | `orphanedMirrors(configured, present)` as a pure name-set verdict (the caller owns the root and the deletion) |
+| host names validated before a session key or an argv | `remote.rs#L194-L200`, test `#L546-L558` | the charset, the leading-alphanumeric rule, and the `:` refusal — decomposed into reasons (`colon` because this store keys sessions as `source:native-id`) |
+| `last_sync_at` advances only on a completed sync; a failure records the error and leaves the stamp | `crates/wake-core/src/db.rs#L116-L122`, `#L864-L877` | the pair, and the "None = never successfully synced" reading |
+
+**Adapted:** Wake's per-host row becomes a per-source one (`source` is this
+store's axis everywhere else, and a remote is a source of a particular kind);
+Wake's two soft-success paths (24, and a second all-ENOENT 23 returning `Err`)
+become one `partial` kind with one `record-error` stamp, so the record
+distinguishes "incomplete" from "failed" without either of them claiming
+completeness; Wake's `RsyncError::SourcesVanished(String)` enum distinction
+becomes the typed `TransferResult`.
+
+**Not ported:** the transport and everything around it — `probe_args`' `sh -c`
+loop, `rsync_args`, `SSH_OPTS`, `REMOTE_LAYOUTS` (eighteen agents' remote paths),
+`--exclude` credential guards, `sync_hosts`' parallel thread scope, the mirror
+size accounting for the settings screen, and `cache_bytes`. Those are the cycle
+that ships a surface.
+
+## Persistence: the stamp contract, and why no schema version moved
+
+`SYNC_STAMP_DDL` is schema-ready and **deliberately not executed by this cycle**.
+The decision, stated as the trade it is:
+
+| Option | Cost | Verdict |
+| --- | --- | --- |
+| Take `SCHEMA_VERSION` 16 now, add the table to `CREATE_TABLES_SQL`, enter the ladder from 13/14/15 | Migrates every store this build touches for a table no code path can populate; makes every older installed build refuse those stores (`SCHEMA_MISMATCH`); grows PR #52's ladder obligation from "accept 15" to "accept 15 and 16" on an unaccepted branch; freezes a row shape before the transport that owns it exists | **rejected** |
+| Ship the contract — DDL as data, row type, `stampedSyncRow`, `readSyncStamps`, disclosure — and let the writer cycle take the version | The table exists only in tests this cycle; the future writer's diff is one line in `CREATE_TABLES_SQL` plus its ladder entry, with no rework of the vocabulary | **taken** |
+
+Boring wins here because a schema version is a claim about shape, and a version
+spent on an empty table buys nothing while charging every store and every
+neighbouring branch. The obligation is recorded, not dropped: **the cycle that
+ships a writer claims 16 and enters the ladder from every live shape**, the way
+C7's step entered from both 13 and 14 — and #52's ladder then owes the same
+two-sided treatment for 15 *and* 16.
+
+What the contract does pin, with CHECKs, so a row cannot lie:
+
+- `last_sync_at` is null or a time; a **complete** outcome must carry one and
+  must clear `last_sync_error`;
+- an **incomplete** outcome must carry `last_sync_error` (the reason code is the
+  account of last resort — `stampedSyncRow` never writes a failure with no
+  account) and must not advance `last_sync_at`;
+- `last_outcome` is the closed kind set, so the store can still tell
+  `unreachable` from `empty` a week later;
+- `updated_at` stamps the *attempt*, complete or not, so staleness has two
+  readable ends: when the last attempt ran, and when the last sync completed.
+
+`readSyncStamps` returns `null` for a store without the table — the same
+"unknown is not empty" distinction `StoreStatus.coverage` draws, and the same
+`sqlite_master` probe `TrajectoryDb.getSourceCoverageSummary` uses.
+
+## Coverage honesty
+
+`discloseSyncState(stamps | null)` is the shape a consuming surface renders,
+modelled on the existing coverage envelope (`coverage: {degraded, total,
+oldest_updated_at}` — `SourceCoverageSummary`, the `sync --robot` summary and
+the v2 search envelope):
+
+| state | meaning |
+| --- | --- |
+| `unknown` | the store holds no stamp table: sync state is not knowable here — never "clean", never "never synced" |
+| `never_synced` | the table exists and is empty: no attempt has run (a fact about attempts, not about content) |
+| `recorded` | per-source entries, `total`, `incomplete` (last attempt did not complete — the `degraded` analog), and `oldest_synced_at` (the staleness floor over *completed* syncs) |
+
+Each per-source entry carries a `mirror` claim ceiling: `current` (the last
+attempt completed), `incomplete` (`partial`: the mirror misses exactly what
+vanished), or `not-claimed` (`unreachable`/`refused`: this store says nothing
+about what the mirror holds).
+
+**No status line and no CLI surface ships with this cycle**, and that is the
+honest half of "ship the disclosure if the table ships": the table does not, so a
+rendered line would report an unknown as a fact. The types and their derivation
+are tested; the verb that prints them belongs to the cycle that can write them.
+
+## Decisions
+
+1. **A closed kind set whose members are claims, not severities.** `ok`,
+   `empty`, `partial`, `unreachable`, `refused` — each one names what a
+   consumer may and may not conclude. Alternatives considered and rejected:
+   a boolean `ok`, because it cannot separate "no contact" from "empty";
+   separate `missing`/`error`/`timeout` kinds, because they are all "no contact"
+   to every consumer that matters (the distinction lives in the reason, where it
+   can be counted without widening the claim vocabulary).
+2. **`refused` is one kind, not three.** Invalid name, missing tool, failed
+   mirror deletion, and a non-vanished transfer error are all "the attempt did
+   not establish a mirror state this store can vouch for"; the *reason* carries
+   the distinction. This keeps the disclosure table small and the reasons
+   countable.
+3. **A reached probe is the mirror's only write authority, and the record names
+   deletions that landed.** Wake's own comment says it ("cannot treat
+   cannot-connect as the remote is empty"); here it is a typed cache action, so
+   `keep-untouched` cannot be confused with `purge([])`, and a purge that fails
+   contributes nothing to the deletion record (`deleted`) — the licensed set is
+   still a fact, and it is the outcome's own `absent`. Two facts, two fields,
+   each labeled for what it is.
+4. **The transfer directive is the probe's output, by construction.** Not a flag,
+   not a caller's responsibility: the state machine produces the directive from
+   the probe it just classified.
+5. **Partial records an error.** Wake's 24 is a soft success; here it is
+   `partial` + `record-error`. The cost is a visible account after a partial
+   round (a busy remote source can produce one repeatedly); the alternative is a
+   store that shows a clean timestamp over a mirror that is missing files.
+6. **The stamp table is not created this cycle** (see the table above).
+7. **`orphanedMirrors` decides names, not paths.** The caller owns the mirror
+   root and the deletion; the verdict is set difference against the configured
+   names, re-asserted after every round. Wake's ordering (purge after the round,
+   not only on removal) is the ported semantics — the function is idempotent and
+   cheap, and the contract is that a driver runs it after each attempt settles.
+8. **No status verb, no CLI flag, no env var.** Nothing in this cycle is
+   reachable from a command line, by design: a verb would need a transport to
+   have something to report.
+9. **The primitives are exported** (`index.ts` → `export * from
+   "./sync-truth.js"`), because the module's whole purpose is to be the one
+   owner of a vocabulary three future surfaces would otherwise invent three
+   times. `api.test.ts`'s exact export list and `docs/api-surface.md` §2 are
+   updated with them.
+
+## The ADR-0005 boundary: what this cycle deliberately does not decide
+
+ADR 0005 (push ingestion tiers) is **proposed and unaccepted** (`docs/decisions/`
+does not carry it; it lives as PR #13's draft), and issue #14 stays blocked until
+it is accepted or amended. This cycle ships **no transport and no sync feature**,
+and the boundary is explicit:
+
+- **No transport.** Nothing spawns ssh, rsync, or any other process. The
+  classifiers take reports; the machine returns directives; a driver that does
+  not exist yet runs them.
+- **No push-vs-pull decision.** The vocabulary is deliberately transport-neutral:
+  the probe/transfer/purge steps describe a *mirror*, and the outcome/stamp/
+  disclosure vocabulary would serve a push tier unchanged — but nothing here
+  presumes either, and nothing here depends on ADR 0005's tier-B fidelity rule
+  or its "never a synchronous dependency" constraint.
+- **No whitelist.** Which paths a source contributes, what is excluded, and the
+  rule that credentials are never in the list (Wake's contract test,
+  `remote.rs#L560-L572`) belong to the surface that knows the layouts.
+- **No scheduling, no lease, no concurrency.** A future sync that writes the
+  store is a writer like any other and must take the writer lease; that
+  interaction, and whether a mirror pass runs beside a scan, are the transport
+  cycle's decisions.
+- **No layout or cache root.** `orphanedMirrors` compares names; where mirrors
+  live, how they are laid out, and whether they count toward the store's
+  disclosed size (Wake folds `remotes/` into its storage line) are open.
+- **No schema version.** See decision 6 — the ladder obligation is recorded for
+  the writer cycle, and the ladder itself is not touched here.
+
+## Files
+
+- `packages/core/src/sync-truth.ts` (new, 843 lines with comments) — the
+  vocabulary, the classifiers, the machine, `orphanedMirrors`, the stamp
+  contract and DDL, and the disclosure.
+- `packages/core/tests/sync-truth.test.ts` (new, 622 lines, 25 cases) — names,
+  probe classification, dialects, the machine's every transition, the exhaustive
+  invariant sweep, the mirror lifecycle, the DDL's refusals, the stamp's
+  carry-forward rule, and the disclosure.
+- `packages/core/src/index.ts` — exports the primitives.
+- `packages/core/tests/api.test.ts` — the exact exported-key list.
+- `docs/api-surface.md` — §2.1/§2.3/§2.4 (the new surface), §3.1 (the table that
+  is not created, and the obligation), §4 (the ADR entry).
+- `docs/research/2026-09-16-wake-port-program.md` — C8's row marked done.
+
+## Still open (recorded, not hidden)
+
+- **The writer cycle's ladder obligation.** Claiming 16 later means entering
+  from 13, 14 and 15 (a store at 15 without `sync_stamps` is exactly the shape
+  the ladder must accept), and PR #52's branch then owes 15 *and* 16 the same
+  way. Recorded here because this cycle chose not to spend the version.
+- **A configured name that fails validation.** `orphanedMirrors` decides by
+  configured-name membership, which is Wake's rule; a configured-but-invalid
+  name therefore still "claims" its mirror. A driver should drop such a name from
+  the configured set (or purge explicitly), and the next cycle should decide
+  whether the store refuses to keep it at all.
+- **Dialect coverage is two shapes.** See the limit above; the escape hatch is
+  one row.
+- **`partial`'s account may repeat.** A source that is being written during
+  every sync will report a vanished source every round; the stamp keeps the
+  error until a round completes. That is intended, and it is the cost decision 5
+  accepted.
+- **Nothing consumes the primitives yet.** The stamp table has no writer, the
+  disclosure has no renderer, `orphanedMirrors` has no caller, and the mirror
+  has no location.
+
+## The roadmap row
+
+> **C8** Remote truthfulness primitives (unreachable ≠ empty, partial ≠
+> complete, per-path probe, exit-24 handling, disposable cache lifecycle) —
+> *#14/ADR-0005 blocked on the decision, but the failure-semantics vocabulary has
+> no owner; port it as tested primitives without shipping a transport.*
+> *Open-source case:* every multi-machine user hits this; honest sync failure is
+> a moat vs "it silently showed nothing" — Wake's `remote.rs` is the pinned
+> authority. (`docs/research/2026-09-16-wake-port-program.md`)
+
+Ported: the probe-before-transfer order, exact-whitelist probe filtering, the
+probe-fails-everything rule, the dialect-pinned vanished-source rule, the single
+tolerated re-probe, the failed-mirror-deletion-is-a-failure rule, the disposable
+mirror's orphan purge, host-name validation, and the `last_sync_at` /
+`last_sync_error` pair. Adapted: the outcomes became a closed typed vocabulary
+with explicit cache and stamp actions, `partial` replaced Wake's two soft-success
+paths, and the dialects became a data table. Not ported: everything that talks to
+another machine.
+
+## Verification
+
+- `cd packages/core && bun test` — **626 pass / 0 fail**, 4,742 assertions, 43
+  files (the tree before this cycle was 601 pass / 42 files; this cycle adds one
+  file and 25 cases).
+- `bun run typecheck` — clean (`tsc --noEmit`).
+- `bun run build` — clean (85 modules, 0.55 MB bundle, then `tsc --noEmit`).
+- `bun test tests/sync-truth.test.ts` — 25 pass / 0 fail, 435 assertions.
+- `bun test tests/api.test.ts` — 19 pass / 0 fail, including the exact
+  exported-key list this cycle extended.