@saasontools/strauss-kb 0.1.8 → 0.1.10

package/ARCHITECTURE.md

@@ -72,6 +72,95 @@ replacement's first log format was `·`-delimited, with a splitter to read it
 back. Both are gone: the log is JSONL and the schema is emitted from Zod, so
 `strauss-kb schema` is the contract rather than a description of one.
 
+## Cross-worktree log safety
+
+`log.jsonl` is append-only and the one artifact nothing can rebuild, so how it
+merges across worktrees matters more than how any record file does. Records
+already don't need this: one concept id is one file, so two writers choosing
+distinct ids never merge at all, and `link`-based publish (above) turns the
+one case where they collide into a 409 rather than a merge. The log has no
+such escape — every writer appends into the same file by design — so it needs
+an actual merge strategy, not just atomicity per write.
+
+The append itself already had it: `record()` uses `appendFile`, which opens
+`O_APPEND` and is one `write(2)` for an entry this small, so two processes
+appending locally interleave whole lines, never a torn one. What was missing
+was git's merge of two worktrees' independently-appended logs — the default
+line-level merge can conflict, or silently keep one side, on lines that were
+never in conflict, since both sides only ever added, never edited, a line.
+
+The fix is `.gitattributes: log.jsonl text eol=lf merge=union`, written by
+`record()` — every path that appends a log line (`write`, `setStatus`,
+`verify`, `supersede`), not `write()` alone — on first use
+(`kb-store.ts#ensureGitattributes`). `union` is a merge driver git ships —
+nothing to configure beyond the attribute — that keeps both sides' added
+lines; `eol=lf` closes a second divergence path, below. The alternative
+considered and dropped was a lock file coordinating writes across worktrees
+the way `mutate`'s CAS coordinates a single record: it would need to span
+process boundaries and survive a crashed holder, which is the same
+stale-lock failure mode rejected above, for a smaller problem than the one it
+solves there.
+
+Decisions worth naming because a later reader could reopen them:
+
+- **A `.gitattributes` that exists but declares no merge strategy for the
+  log gets the line appended, not left alone.** The alternative — leave a
+  user's own file untouched — reads as more conservative, but a bundle that
+  already has a `.gitattributes` in front of it is exactly the bundle most
+  likely to be shared across worktrees or forks, so leaving it without union
+  merge is the worse of the two failure modes. A file that _does_ already
+  declare a merge strategy for `log.jsonl` — this one or a user's own, such
+  as `merge=ours` — is left alone entirely: gitattributes resolves repeated
+  lines for one pattern by "last one wins", so appending a second `merge=`
+  line would silently override rather than coexist. Recognizing "already
+  declared" needed a real tokenizer (`hasMergeDeclaration`) rather than
+  exact-string matching against the line this module writes — the first cut
+  missed a tab or doubled space between tokens (false negative, a harmless
+  duplicate line) and could never recognize a user's own `merge=ours` as a
+  decision already made (the one case where appending anything is wrong).
+- **A `.gitattributes` that fails to _read_ is never treated as "missing".**
+  The first cut folded every `readFile` failure — permissions, a transient
+  `EMFILE`, the path being a directory — into "doesn't exist yet" and took
+  the create branch, which is a truncating write: a real `.gitattributes`
+  hit by a transient read error would be silently replaced with just the
+  union-merge line. Only `ENOENT` means missing; anything else is reported
+  as a failure and the file is left exactly as it was. The create branch
+  also uses `wx` (exclusive create) rather than a plain write, so a
+  concurrent writer that created the file between the read and this write
+  fails loudly into the same best-effort catch instead of the second writer
+  truncating the first one's file.
+- **Union merge does not preserve line order, so `kb_log`'s reader sorts by
+  `at` rather than trusting file order.** Sorting there, once, is cheaper
+  than trying to make every future merge order-preserving. `at` is now
+  validated as an actual ISO-8601 timestamp (`z.iso.datetime()`, matching
+  exactly what `record()` writes) rather than any non-empty string — a value
+  that parses as JSON and matches the schema's shape but isn't really a
+  timestamp would otherwise sort unpredictably instead of failing, and
+  `parseLog` already has a place for "well-formed but wrong" to go: reported
+  as malformed, same as any other schema mismatch, never silently repaired.
+- **A union merge can keep the exact same line twice** — a cherry-pick or
+  rebase that carried one worktree's entry into the other's history before
+  the merge, not two independent writes agreeing by chance: `record()` mints
+  its own `at` per call, so two entries equal on every field including `at`
+  cannot be genuine. `parseLog` dedupes entries that are byte-for-byte equal
+  after parsing and keeps everything else, including two entries that agree
+  on every field except `at` — that pair is two real events. The
+  alternative — leave duplicates visible and call it "genuine repeat
+  ambiguity" — was rejected: there's no ambiguity to preserve, since the
+  only way to produce an exact duplicate is the merge itself.
+- **A read-then-append race across processes on the append branch — two
+  processes both reading a `.gitattributes` without the line, both
+  appending it — is left unguarded, not a reason to lock.** `appendFile` is
+  `O_APPEND`, so the outcome is two copies of the same line, never a torn
+  write, and `hasMergeDeclaration` sees a duplicate declaration as "already
+  declared" on the very next call. Cheap residue, not corruption — the same
+  trade the lock-file alternative above was rejected for, at a smaller
+  scale.
+
+GitHub does not run merge drivers for a PR it merges server-side — see the
+README's "Cross-worktree writes" section. The driver only helps a merge a
+local git client actually performs.
+
 ## Rejected for now: a base registry
 
 Cross-base questions are unaskable by construction — supersession, traces, and

package/README.md

@@ -54,6 +54,7 @@ bundling can `require()` it without depending on its Node version honouring
   <type>.<slug>.md    records
   INDEX.md            index      derived, store-owned
   log.jsonl           history    primary, append-only
+  .gitattributes      merge      store-owned, written on first write
   .index.sqlite       search     derived, gitignored
 ```
 
@@ -76,6 +77,46 @@ Repair-on-read, not coordination, is what lets both exist without a lock. The
 index is _eventually_ correct: a writer whose scan predated another's record
 publishes a briefly stale index, and the next read through the store settles it.
 
+### Cross-worktree writes
+
+A committed base is routinely written from more than one worktree at once —
+each records into the same `log.jsonl`, and a plain git merge of two branches
+that both appended lines resolves that file at the line level, same as any
+other text file. That is the wrong merge for an append-only log: git's default
+picks a side, or conflicts, on lines that both branches only ever meant to add
+to.
+
+So the first write to a base (through `write`, or whichever call happens to
+append the first log line) declares a merge driver for its log: it writes
+`log.jsonl text eol=lf merge=union` into the base's `.gitattributes` if that
+file does not exist yet, and appends the line if the file exists but declares
+no merge strategy for `log.jsonl` yet — a `.gitattributes` a user put there
+first is respected, never overwritten wholesale, and a line that already
+gives `log.jsonl` _any_ merge strategy — this one or the user's own choice
+such as `merge=ours` — is left alone rather than layered under a second,
+possibly conflicting one. `union` is one of git's built-in merge drivers; the
+attribute alone is enough; nothing else needs configuring. `eol=lf` pins line
+endings to `\n` regardless of a checkout's `core.autocrlf`, so a Windows
+checkout normalizing the file on checkout can't leave it with mixed endings
+against the raw `\n` every append writes. With it, a merge of two branches
+that both appended to `log.jsonl` keeps both sides' lines instead of picking
+one.
+
+A union merge does not preserve line order, and can occasionally keep the
+same line twice (a cherry-pick or rebase that carried one side's entry into
+the other's history before the merge). `kb_log`'s reader (`kb-log.ts`) sorts
+entries by `at` and drops exact duplicates before returning them, so neither
+is something a caller has to account for.
+
+**This applies to a local `git merge`, not to GitHub.** GitHub computes pull
+request merges (and the merge/squash/rebase buttons) through its own service,
+which does not read `.gitattributes` merge-driver declarations — a PR that
+merges two branches' `log.jsonl` appends on GitHub gets git's ordinary
+line-level merge (or a conflict) even with the attribute in place. The union
+driver only fires for a merge actually run by a local git client, which covers
+worktrees pulling from and pushing to each other directly, but not a merge
+GitHub itself performs.
+
 ## Records
 
 The filename is the identity. `fact.auth-retries.md` has concept id
@@ -215,7 +256,9 @@ strauss-kb [--bundle PATH] <command> [args]
   supersede <concept-id> <replacement-id>  Mark a record superseded, linking both directions.
   answer <concept-id> <answer...>          Resolve an open question and append the answer.
   verify <concept-id> --note <text>        Append a verified[] event — who checked, when, and what the check found.
-  load [type] [--budget N | --all]         Hand over the whole base, each record with its standing.
+  load [type] [--budget N] [--all]
+                                           Hand over the whole base, each record with its standing.
+  catalog [type]                           Every record in one line — id, type, title, standing, stale flag.
   pack <conceptId> [--hops N] [--max-nodes N] [--budget N]
                                            The bounded neighbourhood around one record, every cut named.
   query <text...>                          Search; every match arrives flagged with its standing.
@@ -240,7 +283,7 @@ strauss-kb [--bundle PATH] <command> [args]
   STRAUSS_KB_ACTOR names the writer in the log
 ```
 
-Results go to stdout as JSON — `index` and `pack` are markdown, which is what
+Results go to stdout as JSON — `index`, `catalog` and `pack` are markdown, which is what
 they are, and `doctor` prints a table unless `--json` asks for the object
 behind it. `--json` is refused rather than ignored on the commands that have
 only one form, since a flag that quietly does nothing reads as one that
@@ -250,6 +293,10 @@ whose exit code is not just "did it run": a check that reports a problem
 succeeded as a command and failed as a check, so it exits 1 with its findings
 on stdout.
 
+A flag accepts either spelling — `--budget 4000` or `--budget=4000` — and a
+flag given no value is an error rather than a silent fallback to the default,
+so a trailing typo cannot look like success.
+
 ```bash
 strauss-kb --bundle .strauss/kb write fact <<'JSON'
 {
@@ -269,9 +316,9 @@ strauss-kb validate || echo "problems above"
 
 `strauss-kb-mcp` speaks stdio and takes no API key and no required environment.
 Every CLI verb is a tool: `kb_write`, `kb_write_decision`, `kb_no_decision`,
-`kb_status`, `kb_supersede`, `kb_answer`, `kb_verify`, `kb_load`, `kb_pack`, `kb_query`,
-`kb_trace`, `kb_list`, `kb_index`, `kb_log`, `kb_validate`, `kb_doctor`,
-`kb_schema`, `kb_types`,
+`kb_status`, `kb_supersede`, `kb_answer`, `kb_verify`, `kb_load`, `kb_catalog`,
+`kb_pack`, `kb_query`, `kb_trace`, `kb_list`, `kb_index`, `kb_log`, `kb_validate`,
+`kb_doctor`, `kb_schema`, `kb_types`,
 `kb_pin`, `kb_unpin`, `kb_pins`, `kb_context`. Most tools take a `bundlePath`;
 `kb_schema` and `kb_types` describe the format rather than any one base, and
 `kb_pins` and `kb_context` read the workspace pin manifests instead. The one
@@ -355,40 +402,66 @@ Read for a question, not for a session: a base loaded at the start of a long
 conversation is summarised away by the end of it, and reloading costs about
 three thousand tokens. Read it again at the point of use.
 
-`load` refuses rather than truncating when a base exceeds its budget (25,000
-tokens by default). A truncated base is indistinguishable from a complete one,
-so a caller would answer "that was never decided" from a slice it did not know
-was a slice. `context` refuses the same way at its own, tighter budget (4,000
-by default). Superseded records come back as name, replacement and date only —
-their bodies no longer hold, and a body read later in a long session outlives
-the qualifier that said so. `trace` still reaches them by id.
-
-`--all` (`all: true` over MCP) is the escape hatch: it bypasses the refusal
-outright and hands back the entire bundle whatever its size. A loaded result
-carries `tokensLoaded`, the same estimate the budget is held against, and
-`budgetTokens: null` marks that no ceiling was applied; `--all` is mutually
-exclusive with `--budget`. That refusal is the guardrail an agent needs so a
-wide base does not silently consume its whole context; `--all` is for a
-deliberate operator who has decided the size is worth the tokens, not a
-setting to reach for by default. A reader that does not actually need every
-record is better served by a narrower `type` filter or a `query` than by
-turning the guardrail off.
-
-**Pack is the middle rung.** Under budget, load the base whole — perfect
-recall beats any ranking. Over budget, when the work centres on a record you
-can name, `pack` hands over that record's bounded neighbourhood instead:
-everything within `--hops` of the root, walked over the base's edges — body
-links (a `relatedConceptIds` entry is stored as one), supersession in both
-directions, shared code anchors, and shared sources — ranked and cut to
-`--max-nodes`. Standing travels with it: superseded neighbours arrive as the
-same name, replacement and date stubs `load` emits. Every record the cut
-dropped is named under Excluded, because a named gap is knowable and a silent
-one is not, and past its own token budget `pack` refuses exactly as `load`
-does — naming what was already cut, so the caller can narrow the walk or
-raise the ceiling. Below the header, the only place a timestamp appears, the
-output is byte-identical across runs over an unchanged base: two packs diff,
-and a changed byte means changed knowledge. With neither a budget problem nor
-a root record in hand, the question is a point lookup, and that is `query`.
+**The three rungs, in one rule.** While the base fits the budget, `load` it
+whole. Once `load` refuses, `catalog` then `pack` the record that matters. For
+a lookup by wording, `query`.
+
+```bash
+strauss-kb load                          # under the budget: everything, with standing
+strauss-kb catalog                       # past it: one line per record, ~30 tokens each
+strauss-kb pack decision.cursor-v2       # then the neighbourhood around the one that matters
+strauss-kb query cursor pagination       # or a point lookup by wording
+```
+
+A whole read gives perfect recall and can say _no record answers this_, which
+no ranker can. `catalog` keeps that at a fraction of the cost by naming every
+record instead of every body; `query` gives up both, returning its nearest hit
+whatever the distance.
+
+`load` refuses rather than truncating past its token budget (`--budget` /
+`budgetTokens`, 25,000 by default, held against the estimated size of what is
+handed back) — a truncated base reads as a complete one, so a caller would
+answer "never decided" from a slice it did not know was a slice. `context`
+refuses the same way at its own, tighter budget (4,000 by default). Superseded
+records come back as name, replacement and date only; `trace` still reaches
+them by id.
+
+A refusal reports `approxTokens` against `budgetTokens` and carries a `message`
+naming the budget and the next calls. A successful load reports `budgetTokens`
+too, so a caller can see how close it came before crossing the line.
+
+`--all` (`all: true` over MCP) bypasses the budget and hands back the entire
+bundle regardless of size. A loaded result carries `tokensLoaded`, and
+`budgetTokens: null` marks that no ceiling applied; `--all` is mutually
+exclusive with `--budget`. It is for an operator who has decided the size is
+worth the tokens — a narrower `type` filter, `catalog`, or `query` fits better
+when it is not.
+
+**Catalog is the rung that keeps the base knowable.** One line per record —
+concept id, type, title, standing, stale flag — sorted by type then title, at
+roughly thirty tokens each, so a base far past `load`'s budget still fits in
+one call. Superseded records show the replacement in place of a body. The
+header sums record counts by standing and reports staleness separately (a
+current record can be stale). Bodies live in `load`, `pack`, and `trace`.
+
+`catalog` alone has no ceiling and never refuses — cost is linear at roughly
+thirty tokens a record (a thousand-record base is about 30k, five thousand
+about 150k); narrow with `type` at that scale. Output is deterministic given
+a fixed clock — no timestamp, ordering total down to the concept id — so two
+catalogs of an unchanged base diff to nothing except a stale flag flipping as
+`stale_after` passes. Pass an explicit `now` (library callers) to hold
+byte-equality across that boundary.
+
+**Pack is the middle rung.** Under the budget, load the base whole. Past it, when the work centres on a record you can name (`catalog` is how you
+name it), `pack` hands over that record's bounded neighbourhood: everything
+within `--hops` of the root, walked over the base's edges — body links (a
+`relatedConceptIds` entry is one), supersession in both directions, shared
+code anchors, and shared sources — ranked and cut to `--max-nodes`. Standing
+travels with it: superseded neighbours arrive as the same stubs `load` emits.
+Every dropped record is named under Excluded, and past its own token budget
+`pack` refuses exactly as `load` does. Output is byte-identical across runs
+over an unchanged base below the header. With neither a size problem nor a
+root record in hand, the question is a point lookup — `query`.
 
 **Flag, never filter.** `query` returns every hit with its standing, because a
 filtered result set is invisible — the caller cannot tell it missed anything.