brainclaw 1.8.0 → 1.9.0

package/docs/concepts/identity-model-proposal.md

@@ -0,0 +1,371 @@
+# Federated Identity Model — PROPOSAL
+
+> Status: **PROPOSAL** for the architecture loop (Codex schema review + Juan
+> product calls). Not a decision. Produced 2026-06-11 from a two-track
+> identity analysis (local model + federation readiness); full reports held
+> by the coordinator. Couples directly with `event-log-store.md` OPEN
+> QUESTION **C4** (federation conflict primitive) — review them together.
+
+## Why
+
+The federation ambition: several human users, each running one or more
+agents, working on the same project from different machines. Today's
+identity tuple `{agent_name, agent_id, host_id, user}` is locally minted,
+unauthenticated, and name-keyed at every load-bearing point:
+
+- Cloud inbox routed by **bare agent name** (`/api/v1/inbox/claude-code`)
+  — two humans' claude-codes are one identity to the bus.
+- Event cursors, claim ownership comparisons, reputation, trust: keyed by
+  name or per-store registry doc; "Juan's claude-code" is **inexpressible**.
+- `FederationMessage.from` is self-asserted under a single API key; routing
+  uses machine-local `project_path`.
+- ed25519 keypairs are generated per agent and **never used** (private keys
+  currently under `~/.codex/brainclaw/keys`).
+- 2,000+ store files carry the OS username; worktree paths are absolute.
+- No write-authority model exists for same-project-multi-machine (the
+  signaling-vs-execution decision covered cross-*project* only). C4 cannot
+  be answered without one.
+
+Local incidents (2026-06-10) showing the same root: VS Code extension
+inherited agent env → impersonation + cursor consumption; directory
+presence (`~/.config/opencode`) minted a **trusted** identity; three
+simultaneous claude-code instances shared one agent_id (loop slot auth
+distinguishes nothing); a copilot worker committed as the human.
+
+## The model
+
+```
+principal  prn_<id>   a HUMAN (or service account), optionally under org_<id>
+                      (org = the premium billing/management unit)
+actor      agt_<id>   an AGENT INSTANCE = (principal, machine, agent software)
+                      keeps today's agt_ ids; gains principal_id, origin_id,
+                      public key
+origin     orig_<id>  a STORE REPLICA on a machine — minted once at
+                      init/materialize, stored next to project.identity.json.
+                      NOT os.hostname() (hostnames collide; one machine can
+                      hold several replicas). host_id stays as metadata.
+display    name       "claude-code" demoted to a label. Never a key.
+```
+
+### Write authority (answers C4's prerequisite)
+
+- **Execution entities** (claims, sessions, runs, locks): origin-scoped,
+  **single-writer**. Other origins materialize them read-only, tagged with
+  their origin. Claims get eager push-on-write to the bus (piggybacked on
+  the existing claim mutation path — still no daemon) for advisory
+  cross-machine visibility; true atomic arbitration is deferred to the
+  cloud dispatcher (Durable Object, premium) and local-first never blocks
+  on it. **Eager-push messages MUST carry the same record identity as
+  journal slices** — `(origin_id, origin_epoch, seq, entity_rev)` plus the
+  full post-image. Then arrival order vs the periodic pull is irrelevant:
+  the receiver applies any copy through the same never-regress comparison
+  (apply iff incoming rev > materialized rev for that origin), and a push
+  that lands after the pull already delivered it is a no-op. A push lacking
+  this identity would reintroduce ordering bugs; it is not optional.
+- **Memory entities** (decisions, traps, plans…): **highest-rev-wins with
+  origin tiebreak** — not wall-clock LWW; timestamps never order anything
+  (same rule as the event-log spec §2.2). On equal `entity_rev` from two
+  origins, the winner for the materialized view is deterministic
+  (lexicographic `origin_id`) and the loser is surfaced as a conflict
+  candidate, never silently dropped.
+- **The scalar-rev hole, named and fixed**: a bare rev comparison cannot
+  distinguish *descends-from* from *concurrent-with*. Origin A edits
+  rev 5→6→7; origin B concurrently edits 5→6. A's 7 beats B's 6 and the
+  comparison looks like a clean fast-forward — B's edit is lost with **no
+  conflict surfaced**. Scalar `entity_rev` alone therefore cannot honor the
+  "conflicts surface as candidates" promise. The minimal fix (still no
+  vectors): exported memory-entity records carry **`base_rev`** — the rev
+  the write was based on. Receiver rule: incoming wins only if
+  `incoming.base_rev >= current.rev` (true fast-forward); otherwise the
+  write was concurrent → materialize the winner by the tiebreak above AND
+  emit a conflict candidate carrying both post-images. One extra integer
+  per exported record; local writes don't need it (single store, the lock
+  serializes). This is sufficient *because* payloads are full post-images
+  (event-log §2.1) — no merge, just detect-and-surface.
+- **Restore-from-backup breaks `(origin_id, seq)` — origin epochs**: a
+  store restored from backup at seq 700 will re-issue seqs 701… with
+  different content under the same `origin_id`, while peers hold cursors
+  at, say, 1000 — the divergent range is silently ignored and the
+  addressing invariant (one `(origin_id, seq)` = one immutable record) is
+  violated. Fix: slice headers carry **`origin_epoch`** (integer, starts
+  at 0). Detection is cheap and reliable: the cloud keeps a per-origin
+  high-water mark; an origin reconnecting with `local next_seq <=
+  cloud watermark` has been restored → it bumps its epoch (recorded
+  locally as a `seq_repair`-class event) before pushing anything. Peers
+  treat a new epoch as a rebase: drop the cursor for that origin and
+  re-pull from the origin's latest checkpoint. Never-regress stays intact
+  *within* an epoch; epochs make the restore case explicit instead of
+  silent.
+- **Consequence for C4**: with partitioned authority, **scalar
+  `entity_rev` + origin tag + `base_rev` on exported memory records**
+  suffices — no vector clocks. Federation transfers are
+  `(origin_id, origin_epoch, seq)`-headed segment slices.
+
+### Authn — the minimal dumb-bus guarantee
+
+Activate the dormant ed25519 keys. `brainclaw cloud register` (already
+specced in the cli-cloud plan) binds *actor public key → principal* once,
+under the principal's API token (one token per human). The cloud verifies
+each message signature once and stamps `verified_principal`; receivers
+trust the stamp and MUST reject unstamped messages on cloud transport. No
+PKI web. Routing moves from `project_path` to `project_id` + channel.
+(Local cross-project-link transport stays a separate, weaker trust domain:
+same machine, same OS user — signatures optional there.)
+
+#### FederationMessage v2 envelope & signature block (concrete)
+
+```jsonc
+{
+  "schema_version": 2,
+  "id": "msg_...",
+  "from": {
+    "principal_id": "prn_...",
+    "actor_id": "agt_...",
+    "origin_id": "orig_...",
+    "origin_epoch": 0,
+    "origin_msg_seq": 412,        // per-origin monotonic counter (replay watermark)
+    "agent_name": "claude-code"   // display only — never a key
+  },
+  "to": { "project_id": "proj_...", "channel": "signals", "actor_id": "agt_..."? },
+  "type": "signal|handoff|candidate|runtime_note|board_snapshot|claim_advisory|slice",
+  "payload_hash": "sha256:...",   // hash of canonical payload bytes
+  "payload": { ... },             // inline iff <= 64 KB, else payload_ref (see slices)
+  "created_at": "...",
+  "causal_parent": "msg_..."?,
+  "idempotency_key": "...",       // DEDUP ONLY — explicitly not a security control
+  "sig": {
+    "alg": "ed25519",
+    "key_fingerprint": "...",     // selects among the actor's registered keys
+    "sig": "base64..."
+  }
+}
+```
+
+- **What is signed**: the ed25519 signature covers the **RFC 8785 (JCS)
+  canonical JSON** of the envelope with `sig` removed and `payload`
+  replaced by `payload_hash`. Today's `JSON.stringify` of a parsed object
+  is insertion-order-dependent and NOT a stable signing base across
+  writers/runtimes — canonicalization is required, not optional. Signing
+  `payload_hash` instead of inline payload means oversized payloads can
+  move out-of-band (blob transfer, below) without changing the signature
+  scheme.
+- **Replay protection**: `idempotency_key` (content hash, today truncated
+  to 64 bits) is **dedup, not replay defense** — it only helps if the
+  receiver retains seen-keys forever, and it cannot distinguish a
+  legitimate redelivery from a malicious replay. The actual defense is the
+  **per-origin monotonic watermark**: every v2 message carries
+  `(origin_id, origin_epoch, origin_msg_seq)`; receivers keep a high-water
+  mark per origin and reject anything at-or-below it. This unifies signal
+  messages with slice cursor semantics — one mechanism, no retention
+  window to tune. `idempotency_key` survives for content-level dedup but
+  its computation must move to canonical JSON (same JCS bytes as signing).
+- **Key rotation**: actor registration holds a **list** of keys with
+  `created_at` / `retired_at` validity windows, not a single key. Rotation
+  = registering a new key under the principal's API token (same act as
+  initial registration — no cross-signing chain needed because the token,
+  not the old key, is the root of the binding). The cloud verifies against
+  whichever registered key the `key_fingerprint` selects, checking the
+  validity window against `created_at`.
+- **Revocation (the fired-teammate case)**: revocation is enforced at the
+  **single online verification point** — the org admin revokes the
+  principal (dashboard / `members` projection); the cloud immediately stops
+  stamping that principal's messages and rejects its API token. Because
+  receivers trust the stamp rather than verifying raw signatures
+  themselves, **revocation does not need to propagate to offline stores to
+  stop new writes** — an offline store cannot receive new federation
+  messages anyway. What offline stores DO hold: (a) previously
+  materialized data from the revoked principal — that is history, not
+  authority; it stays, attributed; (b) a stale `members` projection that
+  still lists the principal — bounded by re-pull on reconnect. The
+  residual product call: whether a store whose members projection exceeds
+  a max staleness age should degrade to read-only federation (deny new
+  grants by default) — see [JUAN] below.
+
+### Authz
+
+Trust moves from per-local-agent-doc to **per (principal, project)**
+grants, distributed as a signed `members` projection (managed in the
+Phase-2 dashboard, materialized locally like everything else). Local-only
+operation is unchanged: principal defaults to an implicit `prn_local`.
+
+### Slice transfer & event-log coupling
+
+The event-log spec is written for ONE store-global seq space assigned
+under the local lock. Foreign events cannot be appended into it (their
+seqs are foreign); the spec's own §2.2 scope boundary anticipates exactly
+this. The ingestion model, stated so the two documents stop hand-waving at
+each other:
+
+- **Export** is what the proposal already says: slices of the local
+  journal, header `{origin_id, origin_epoch, from_seq, to_seq,
+  slice_hash}` — zero local schema cost, origin appears only on export.
+- **Import — per-origin sidecar journals, not seq renumbering.** Ingested
+  slices land in `events/foreign/<origin_id>/seg-*.jsonl`, preserving the
+  remote `(epoch, seq)` addressing (verifiable against `slice_hash`,
+  re-requestable). The receiver keeps **one cursor per foreign origin**
+  (`{origin_epoch, last_seq}`) — a new mechanism; the spec's `meta.json`
+  per-family watermarks don't cover it and must grow a `foreign_origins`
+  map. Renumbering foreign events into the local seq space was rejected:
+  it destroys the remote addressability that re-sync, audit, and
+  `slice_hash` verification depend on.
+- **Materialization is a local journal event.** Applying a foreign batch
+  appends a local `federation_apply` event (local seq, payload = the
+  per-entity outcomes: applied / regressed-skip / conflict-candidate).
+  Local rebuild therefore replays to the same state WITHOUT consulting
+  foreign journals — checkpoints stay self-contained, and the spec's
+  rebuild/gc/verify story is untouched. Foreign sidecars are cache-class:
+  re-pullable, gc-able by their own retention, never part of local
+  checkpoint verification.
+- **Remote gc floor**: the origin may have archived segments below a
+  requested cursor (spec §2.3 retention). The slice protocol therefore has
+  two response shapes, mirroring §2.5's `{gap: true}`: contiguous events,
+  OR a checkpoint-derived snapshot slice (full post-images at the origin's
+  latest checkpoint) from which the receiver re-bases its cursor.
+  Snapshot slices and the epoch-rebase path (above) are the same code.
+- **Oversized payloads**: the phase-0 falsifier FIRED on handoffs (p50
+  ~110 KB, 15-45× over the 64 KB record threshold). Whatever shape
+  `payload_ref` takes in event-log phase 1, federation inherits it:
+  slices reference blobs by sha256; receivers fetch
+  content-addressed blobs (`blobs/<sha256>`) out-of-band and verify the
+  hash. The envelope/signature design above already accommodates this
+  (signatures cover `payload_hash`, never inline bytes). Note the
+  composable alternative flagged in the measurements doc — a handoff
+  "diet" externalizing `snapshot.diff` — also shrinks the federation
+  privacy surface (the diff is the #1 secret-leak channel) and should be
+  preferred where possible.
+- **Idempotency for slices**: the event-log spec says federation dedup
+  uses `(seq, writer)`. At the transfer layer this becomes
+  `(origin_id, origin_epoch, seq)` — the per-origin watermark — with
+  `slice_hash` as the integrity check. The content-hash `idempotency_key`
+  is for *signal* messages only; do not use it for slices.
+
+### Privacy boundary
+
+The federation export strips or rewrites — verified against
+`src/core/schema.ts`, not just the obvious fields:
+
+- `user` (OS username — Claim, CurrentSessionState) → dropped;
+  `principal_id` is the human attribution.
+- **`author` / `author_id`** (present on every memory entity; in practice
+  the OS username or a human name, occasionally an email) → mapped to the
+  principal's display handle. The original strip list missed this — it is
+  the single most widespread PII field in the store (2,000+ files).
+- **Reviewer/actor name fields**: `HandoffReview.requester/reviewer/
+  reviewed_by`, `Candidate.starred_by[]/resolved_by`,
+  `ActionRequiredResponse.responded_by`, `assignee`, `Sequence.owner` →
+  same principal-handle mapping.
+- **`host_id`** — today raw `os.hostname()` (src/core/host.ts), and
+  hostnames routinely embed personal names ("juans-macbook"). Kept as
+  *metadata* locally, but on export it is hashed (or dropped; `origin_id`
+  already provides replica identity).
+- Absolute paths: `worktree_path`, `related_paths` → repo-relative.
+  **Also**: `AgentRun.command` / `shell` (spawn command lines embed
+  absolute paths, usernames, and potentially tokens) → stripped on export,
+  not relativized — command lines are not reconstructable safely; and
+  `verify_cmd` (decisions/traps) → redaction-pattern pass.
+- pid, session env internals (`CurrentSessionState.pid/user/
+  active_project.path`) → dropped.
+- **Existing leak, fix independent of v2**: `FederationMessage.from.
+  project_path` / `to.project_path` are REQUIRED absolute paths in v1 and
+  are pushed to the cloud **today** whenever cloud_sync is enabled. v2
+  routing by `project_id` removes them; until then this is a live PII
+  exposure, not a future risk.
+- **Narrative content (`text`, `narrative`, `snapshot.diff`,
+  `InboxMessage.payload`, `RuntimeEvent.metadata`) cannot be schema-
+  stripped** — free text can quote anything, including secrets and names.
+  This is policy + redaction territory only: the existing
+  `RedactionConfig` patterns, the explicit `visibility: 'shared'` opt-in
+  (already enforced for handoffs/candidates), and the J1 `doctor redact`
+  erasure path. The proposal claims no schema-level guarantee here, by
+  design.
+
+Pairs with the J1 `doctor redact` decision.
+
+## Identity acquisition (local prerequisite, pln#562)
+
+Detection conflates three questions today; they split as follows:
+
+| Question | Owner | Legitimate evidence |
+|---|---|---|
+| What is installed on this machine? | `agent-inventory` (single source) | disk presence — an **origin** attribute; doubles as the capability advertisement the cloud dispatcher needs |
+| Who is calling? | `detectAiAgent` | process-scoped env markers ONLY — never directory presence; claimed identity sanity-checked *against* inventory |
+| What trust? | explicit acts (human at setup, dispatcher at spawn), capped at contributor | never a read-path side effect |
+
+## Migration (additive where verified; two steps are NOT)
+
+Each claim re-checked against the actual zod schemas and call sites:
+
+1. `AgentIdentityDocument` += optional `principal_id`, `origin_id` —
+   **verified additive** (schema.ts keeps `version: z.literal(1)`; new
+   fields zod-optional; old docs parse). Do NOT bump the version literal.
+2. `origin_id` minted lazily at first federation use, stored next to
+   `project.identity.json`; the v2 event envelope stays agnostic (origin
+   added only to *exported* slices via segment headers — zero local
+   *export* cost). **Import is not zero-cost**: per-origin cursors +
+   foreign sidecars (see slice section) are new mechanisms, landing only
+   with federation itself.
+3. Cursors and cloud inboxes re-keyed name → instance. **Two corrections**:
+   (a) local event cursors (`events/.cursors/<agent>.json`) are being
+   migrated to seq watermarks by event-log phase 1 anyway — do the re-key
+   *in that migration*, not as a separate rename done twice; (b) the cloud
+   inbox re-key (`/api/v1/inbox/<agent_name>` → instance-keyed) is a
+   **coordinated cloud API change, not a local rename** — an old client
+   pulling the name-keyed endpoint against a re-keyed cloud silently gets
+   an empty inbox (the pull error path returns `[]`). Needs an API
+   version window or dual-key serving.
+4. Wire `isDuplicate` + a persisted `since` watermark for cloud pulls now
+   (pre-federation bugfix — **verified real**: session-start.ts pulls with
+   `{limit: 100}`, no `since`, no dedup; every session re-materializes the
+   same signals as new entities). Slightly more than "wire it":
+   `materializeFederationSignal` mints fresh local ids and **discards the
+   incoming `idempotency_key`**, so there is nothing on disk to dedup
+   against — the fix must (a) persist a seen-keys watermark file and
+   (b) stamp `provenance.kind: 'federation'` + `remote_id` on
+   materialized entities (the provenance schema already has the slot).
+5. `FederationMessage` v2 — **the secretly-breaking step**.
+   `FederationMessageSchema` pins `schema_version: z.literal(1)` and
+   `deserializeMessage` hard-parses; a v1-only receiver given a v2 message
+   throws, and `pullSignals`' catch-skip **silently drops it forever**
+   (local-transport inboxes are written by the *sender* into the target
+   project). And v1's required `project_path` fields mean a v2 message
+   that omits them fails v1 parsing by absence too. Mitigation: ship the
+   v1|v2 **union reader first** (accept both, one release ahead), only
+   then switch writers to v2; cloud negotiates per-client version during
+   the window. `idempotency_key` survives as dedup-only, recomputed over
+   canonical JSON (see envelope section).
+
+## Open questions for this review
+
+Resolved by the 2026-06-10 schema review pass (claims now in the body
+above): envelope/signature concretization (JCS canonical form, payload_hash
+detachment, per-origin replay watermark, key rotation via token-rooted key
+list); the scalar-rev concurrency hole (`base_rev` fix); restore-from-backup
+(`origin_epoch`); eager-push ordering (envelope identity + never-regress);
+slice ingestion vs the event-log seq space (per-origin sidecars +
+`federation_apply`); remote-gc snapshot slices; the privacy strip-list gaps
+(author/reviewer fields, raw-hostname host_id, AgentRun.command, the live
+v1 project_path leak); the two non-additive migration steps (3b, 5).
+
+Residual — genuinely needs a second model's eye:
+
+- **[CODEX]** `members` projection wire format: exact signed structure
+  (who signs — the cloud as trust root, or the org-admin principal key?),
+  grant granularity (per-project role enum vs capability list), and the
+  max-staleness/deny-by-default semantics interaction with local-only
+  stores that NEVER reconnect. The revocation *enforcement point* is
+  settled (cloud stamp); the projection's own authenticity chain is not.
+- **[CODEX]** `base_rev` adversarial check: can a malicious or buggy
+  origin *forge* fast-forwards (claim `base_rev = current.rev` to suppress
+  conflict surfacing)? Within-org actors are semi-trusted — decide whether
+  conflict suppression by a hostile insider is in scope, or accepted as
+  "history is signed and attributable, dispute via audit".
+- **[CODEX]** `federation_apply` event schema vs event-log C1 review (it
+  joins the `checkpoint_ref`/`journal_note`/`seq_repair`/`backfill` set):
+  payload shape for per-entity outcomes, and whether a huge foreign batch
+  (10k entities) needs chunked apply events to respect the 256 KB record
+  cap and §2.9 chunked-lock rule.
+- **[JUAN]** org model scope for v1 (implicit single-org? explicit org
+  onboarding?); one API token per human vs per machine; whether claim
+  eager-push is acceptable bus chatter on the free tier; members-projection
+  max-staleness → read-only-federation degradation (availability vs
+  security posture).

package/docs/concepts/memory.md

@@ -30,7 +30,7 @@ Shared project knowledge worth keeping.
 
 Examples: constraints, decisions, traps, completed plans, handoffs.
 
-Shared traps now have a lifecycle too: `active`, `resolved`, or `expired`. Active views such as generated context, status, and `project.md` prioritize only active traps so old machine-setup issues stop polluting the current working set, while the canonical memory still keeps resolved traps for audit and search.
+Shared traps now have a lifecycle too: `active`, `resolved`, or `expired`. Active views such as generated context and status prioritize only active traps so old machine-setup issues stop polluting the current working set, while the canonical memory still keeps resolved traps for audit and search.
 
 These live as individual JSON files under `.brainclaw/memory/` (constraints, decisions, traps, instructions) and `.brainclaw/coordination/` (plans, claims, handoffs). They are versioned in `.brainclaw/.git` and shared via Git.
 
@@ -59,9 +59,10 @@ brainclaw makes this context visible and versionable.
 
 brainclaw keeps:
 
-- canonical structured JSON as the source of truth (individual files under `.brainclaw/memory/` and `.brainclaw/coordination/`)
-- a derived readable view (`project.md`) regenerated best-effort from canonical state
-- native agent instruction files (`CLAUDE.md`, `.cursor/rules/brainclaw.md`, etc.) generated via `brainclaw export`
+- canonical structured JSON as the source of truth (individual files under `.brainclaw/memory/` and `.brainclaw/coordination/`)
+- a legacy derived readable view (`.brainclaw/project.md`) regenerated best-effort from canonical state
+- root `PROJECT.md` as the stable project vision when present
+- native agent instruction files (`CLAUDE.md`, `.cursor/rules/brainclaw.md`, etc.) generated via `brainclaw export`
 
 This balances machine reliability with human readability.