@ctxr/skill-llm-wiki 1.0.1 → 1.1.0

package/CHANGELOG.md

@@ -4,6 +4,124 @@ All notable changes to `skill-llm-wiki` are documented in this file.
 
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [Unreleased]
+
+### Performance
+
+- **Large-corpus pairwise-sweep speedup (~50-100× on I/O-bound paths).** A 596-leaf deterministic build previously took 2h15m; this release targets the three I/O antipatterns responsible for most of that wall time. Surfaced during a live Phase X.6 rebuild attempt on `skill-code-review/reviewers.src`.
+  - **`decision-log.mjs::appendDecision` now uses POSIX append.** The prior implementation read the WHOLE `.llmwiki/decisions.yaml` on every call (via `readFileSync`), concatenated the new entry, wrote to a temp file, and renamed — O(file-size) per append. On a 45 MB decisions.yaml that's ~22 MB of avg-read per call × 189k calls ≈ 4 TB of I/O. Rewritten to use `appendFileSync` for every call after the first; the header-creation path still goes through temp+rename for atomicity. Cost per append drops from ~20 ms to ~200 µs — a ~100× speedup at scale. The append path is best-effort (not the same atomicity contract as temp+rename): on a crash a torn trailing entry is possible and the YAML parser will reject the truncated scalar; recovery is to drop the last `- ...` block and re-run the op. The decision log is an audit trail rather than a reproducibility artefact — tree SHAs are unaffected. See the docstring in `scripts/lib/decision-log.mjs` for the full durability breakdown (first-call atomicity via temp+rename, subsequent calls best-effort, Windows/concurrency caveats). Also includes a pre-append check that peeks at the log's last byte: if the existing file doesn't end in `\n` (manual edit, prior torn-tail truncation), a leading newline is inserted so the appended entry can't concatenate onto the previous line and corrupt YAML.
+  - **`similarity-cache.mjs` shards cache entries by 2-hex prefix.** The prior flat `<cacheDir>/<sha256>.json` layout put 178k files in a single directory on a 596-leaf build. APFS/ext4/ZFS directory lookups degrade once the VFS dirent cache overflows (~10k entries on typical kernels), so by mid-sweep every `existsSync` / `writeFileSync` paid O(log N)-with-large-N cost. New layout: `<cacheDir>/<2hex>/<rest>.json` → 256 shards, ~700 entries per shard for a 178k-pair corpus. Same pattern as `.git/objects/`. Exported new `CACHE_SHARD_PREFIX_LEN` constant so tuning is explicit. Pre-sharding flat caches are NOT auto-migrated — the similarity cache is pure perf memoisation with no user data at stake, and the next build repopulates the sharded layout. `clearCache` and `cacheSize` walk every shard subdir plus any residual flat entries, so pre-upgrade caches still report size and can be cleared.
+  - **`operators.mjs::detectMerge` batches pairwise `decide()` calls via `Promise.allSettled`.** The O(N²) nested loop previously awaited each pair sequentially. At 178k pairs and ~10 ms per pair (cache-hit avg), that alone was ~30 min of wall-clock time even with all I/O cached. Pairs are now streamed into batches of `DETECT_MERGE_PAIR_BATCH_SIZE = 32` concurrent awaits; the streaming accumulator keeps memory at O(batch) instead of O(N²). Result order is preserved via positional `allSettled[]` semantics so downstream proposal collection stays deterministic. No tiered.mjs changes required — each `decide()` call is independent (its own cache file, its own decision-log append); concurrent calls on Node's cooperative event loop don't race on shared mutable state since microtask-ordered awaits serialize map mutations. Expected speedup on I/O-bound sweeps: 10-20× on modern SSDs; more modest on spinning disks.
+  - **Per-pair timeout + retry guards via `p-timeout` + `p-retry`.** A stuck pair (NFS hang, runaway embed call, filesystem lockup) used to stall the whole convergence phase forever under the serial `Promise.all` semantics, AND one genuine error would abort the whole batch, discarding every successfully-completed pair alongside it. Every `decide()` call is now wrapped with `pTimeout({ milliseconds: DETECT_MERGE_PAIR_TIMEOUT_MS = 30_000 })` for per-call deadline enforcement, and `pRetry({ retries: DETECT_MERGE_PAIR_RETRIES = 2, minTimeout: 500, maxTimeout: 5_000 })` for transient-failure resilience. The outer batch uses `Promise.allSettled` so a pair that exhausts retries logs to stderr and skips instead of aborting — the batch's other 31 pair decisions land normally. Validation errors inside `decide()` are treated as non-retryable by throwing `AbortError` so a structural bug doesn't burn the retry budget on something that would fail the same way every time. Breadcrumb logs on every retry attempt + every final-fail let operators see reliability activity without grepping through the decision log. New runtime deps: `p-retry@^8` and `p-timeout@^7` (both sindresorhus, zero-dep, ESM-only — the canonical Node retry/timeout pair, ~1 KB each gzipped).
+
+### Added
+
+- **CLI progress breadcrumbs (Phase X.9).** Long-running operations (build / rebuild / fix / join) now stream a per-phase breadcrumb to stderr as each orchestrator phase fires:
+  - `[build-20260424-212011-abc123 3] ingest: read 596 source file(s), 589 leaves`
+  - `[build-... 7] operator-convergence: 156 operator(s) applied across 23 iteration(s); ...`
+  - Format: `[<op-id> <phase-index>] <phase-name>: <phase-summary>`
+  - Format matches exactly what the final phase log records in the op-summary payload — no new formatter, just a live mirror of `record()` calls.
+  - Writes to stderr (never stdout) so consumers piping the command's stdout don't conflate phase chatter with the final op-summary payload.
+  - Monotonically-increasing phase index per op, so structured log aggregators can reconstruct phase ordering after multiplexing.
+  - Progress is on by default; suppressed when `--json` is passed (the `skill-llm-wiki/v1` envelope consumer contract requires a clean stderr) or when `LLM_WIKI_NO_PROGRESS=1` is set in the environment (CI / hermetic runs that want the pre-X.9 silent-stderr shape).
+  - Implementation: `scripts/lib/orchestrator.mjs::runOperation` accepts an `onProgress({name, summary, index})` callback that the internal `record()` helper invokes alongside the phase-log push. CLI wires a callback that writes the breadcrumb to stderr; in-process callers can pass their own handler for custom streaming (TUI, structured log shipper, etc.). Progress-hook throws are swallowed — a misbehaving callback never halts the op.
+  - Tests (`tests/unit/cli-progress.test.mjs`, 3 scenarios): stderr breadcrumb shape + monotonic phase index; `LLM_WIKI_NO_PROGRESS=1` suppresses; `--json` suppresses.
+- **Real 11-phase `join` implementation (Phase X.2).** The `join` operation now implements the full pipeline from `guide/operations/ingest/join.md` instead of falling through to the convergence-only stub. The skill can now actually merge N ≥ 2 source wikis into one unified output.
+  - **New module `scripts/lib/join.mjs`** exports `runJoin(sources, target, ctx)` and the per-phase helpers (`ingestWiki`, `validateSources`, `planUnion`, `resolveIdCollisions`, `mergeCategoriesWithSameFocus`, `rewireReferences`, `materialisePlan`), plus the canonical `VALID_COLLISION_POLICIES` / `DEFAULT_COLLISION_POLICY` constants.
+  - **Pipeline phases (per the methodology):**
+    1. `ingest-all` — walk each source wiki's tree, reading frontmatter through `readFrontmatterStreaming` (CRLF-safe, bounded reads) and the full body from `captured.bodyOffset`. Plain markdown files with no frontmatter fence and dot-prefixed files are skipped.
+    2. `source-validate` — `validateWiki` on each source, aggregated into a single error/warning report. Any hard error halts with `JOIN-SOURCE-INVALID` — joining a broken source would produce a broken output.
+    3. `plan-union` — merge per-source leaf + index records into a unified plan with `sourceWiki` provenance on every record.
+    4. `resolve-id-collisions` — three policies: `namespace` (default) prefixes colliding ids with `<source-basename>.` and relies on source-scoped `renameMap` rewrites in phase 6 to resolve inbound references (the original id stays with the keeper, so adding it to `aliases[]` would trip `ALIAS-COLLIDES-ID`); `merge` folds duplicates when frontmatter is compatible (same `focus`/`type`/`depth_role`) and falls back to namespace on incompatibility; `ask` throws `JOIN-COLLISION-ASK` for the caller to surface. Every renamed leaf also has its filename rewritten (via `relPath`) to satisfy the validator's `ID-MISMATCH-FILE` rule. A running `liveIds` Set guards against secondary collisions when two sources share the same basename (e.g. `/a/reviewers.wiki` and `/b/reviewers.wiki`) — `reserveId` appends `-2`, `-3`, ... as needed.
+    5. `merge-categories` — detects top-level categories with matching `focus`. First-cut is DETECT-ONLY: folding is deferred because `runConvergence`'s MERGE operator handles sibling leaves, not entire category subtrees. A directory-MERGE operator is tracked as a follow-up.
+    6. `rewire-references` — `links[].id` and `overlay_targets[]` entries are rewritten via `renameMap` then `mergeMap`. Unresolvable references are left as-is and surface as `DANGLING-LINK`/`DANGLING-OVERLAY` at Phase 9 so the user gets one unified report.
+    7. `apply-operators` — `runConvergence` on the unified materialised tree (same tiered-AI quality mode the user passes to `build`/`rebuild`).
+    8. `generate-indices` — `rebuildAllIndices` so every index's `entries[]` reflects the unified structure.
+    9. `validation` — `validateWiki` on the target; failures throw `JOIN-TARGET-INVALID` and trigger rollback.
+    10. `golden-path-union` — placeholder; fixture-regression gate lands as a follow-up.
+    11. `commit` — the orchestrator tags `op/<op-id>` and appends the op-log entry.
+  - **Source immutability.** Every source wiki is treated as strictly read-only. `runJoin` writes only to the prepared target directory (which the CLI creates empty before the op fires). A consumer passing a `--target` that equals, nests under, or contains any source wiki is refused at the intent layer with `INT-18` — the check runs before the pre-op snapshot, so a stray `--target /path/to/source-a/inside` never writes a byte to source A. A target that is unrelated to every source but happens to already exist non-empty also fails early with `INT-01`.
+  - **Per-phase commits.** `runJoin` takes an optional `onPhaseCommit` callback that the orchestrator wires up to `git add -A` + `gitCommit` between the materialise, convergence, and index-generation phases — the joined wiki's private git log shows the same per-phase granularity a fresh `build` does, so `git diff op/<id>~1..op/<id>` works at phase scope.
+  - **Orchestrator integration.** A new `plan.operation === "join"` early branch in `scripts/lib/orchestrator.mjs::runOperation` takes the pre-op snapshot on the fresh target, delegates the 11 phases to `runJoin`, tags the op on success, and rolls the tree back to the pre-op snapshot on any failure — the same rollback semantics `build`/`rebuild`/`fix` get.
+  - **CLI surface.** `node scripts/cli.mjs join <wiki-a> <wiki-b> [<wiki-c>...] --target <path> [--id-collision namespace|merge|ask] [--quality-mode <mode>]`. Positionals can be any number ≥ 2; each must be an existing skill-managed wiki (has `.llmwiki/git/HEAD`). `--target` is required and must be empty (or not exist yet). New `--id-collision` flag with default `namespace`.
+  - **Intent validation.** New structured error codes: `INT-06` for fewer than 2 sources / non-existent source / source-not-managed, `INT-09b` for missing `--target`, `INT-01` for non-empty `--target`, `INT-17` for invalid `--id-collision` value, and `INT-18` for `--target` that overlaps any source wiki (source-immutability guard). All fire at the intent layer so a typo never reaches the pre-op snapshot.
+  - **Tests.** 13 unit tests in `tests/unit/join.test.mjs` cover each phase's contract: ingestWiki reads frontmatter + body + skips dotfiles; validateSources aggregates findings across sources; planUnion tags provenance; resolveIdCollisions exercises all three policies (namespace prefix, merge-compatible fold, merge-fallback-to-namespace on incompatible frontmatter, ask-throws, invalid-policy-rejects); VALID_COLLISION_POLICIES canonical list pinned; mergeCategoriesWithSameFocus detects; rewireReferences rewrites via map; materialisePlan writes to target.
+- **Root-leaf containment invariant (Phase X.11).** The wiki root must hold only `index.md` plus subdirectories — never a direct `.md` leaf. A new orchestrator phase between soft-DAG synthesis and review (Phase 4.4.5, `scripts/lib/root-containment.mjs::runRootContainment`) walks `wikiRoot`, identifies direct-child outlier leaves (non-index `.md` files at the wiki root itself — depth 0 per `depthOf`, topical singletons whose affinity to every other leaf stayed below clustering thresholds), and moves each into its own semantically-named subcategory derived from its own TF-IDF distinguishing tokens. A stub `<slug>/index.md` is written so the new category is routable; Phase 5's `rebuildAllIndices` populates the stub's `entries[]` on the same pass.
+  - **Why single-member categories instead of a shared "uncategorised" bucket.** Every reviewer leaf has `focus` / `covers` / `tags` that describe some coherent topic, so the honest answer to "where does this belong?" is "in its own tight category named after what it is." A shared bucket label admits defeat about something the data already tells us; a per-outlier slug preserves the semantic signal. If the corpus later grows a topically-adjacent leaf, future builds' convergence + balance may nest both into an existing category — a single-member start state is a valid transient, not a permanent scar.
+  - **Slug derivation.** Reuses Phase X.3's `generateDeterministicSlug([outlier], siblings)` and `deterministicPurpose([outlier])` so every outlier gets the same byte-stable naming any clustered leaf would. Sibling corpus is recomputed per outlier so the newly-added subcategory from the previous iteration is included in the IDF ranking — the second outlier picks a slug that discriminates against the first, not a near-duplicate. Uniqueness is enforced via `resolveNestSlug` + the full-wiki forbidden-id index from PR #5: a generated slug that happens to collide with an existing subcategory basename, leaf id, or alias elsewhere in the tree gets the `-group` / `-group-N` fallback treatment. The shared `wikiIndex` is mutated after each resolve so subsequent outliers can't reuse a just-assigned slug.
+  - **parents[] rewrite.** The leaf's new direct parent (`<slug>/index.md`) is same-dir-as-leaf, so a primary `"index.md"` entry stays byte-identical across the move even though its semantic target moves from root-index to subcategory-index. Same convention `applyBalanceFlatten` leveraged when moving a subtree up one level (PR #8). Non-primary entries that DON'T already start with `"../"` gain a `"../"` prefix because paths that were relative to the old leaf-dir (wiki root) are now one level too shallow. Example: `["index.md", "beta/index.md"]` → `["index.md", "../beta/index.md"]`. Entries that already start with `"../"` are preserved byte-identical — a root-level leaf has no legitimate parent above wikiRoot to reference, so the input is already a depth-contract violation; prepending another `"../"` would only escape the root outright (`"../foo"` → `"../../foo"`). The malformed entry stays as-is and validation surfaces it under its normal parent-path rules.
+  - **Determinism.** Outlier iteration is lex-sorted by filename, so two runs on the same outlier set produce byte-identical slug assignment order (matters for `-group-N` collision tie-breaks). Files whose frontmatter fails to parse are skipped silently — the validator surfaces them separately under `PARSE`.
+  - **LIFT guard.** `detectLift` refuses to emit a proposal when the lift destination would be the wiki root (`dirname(dir) === wikiRoot`). Without this, Phase 4.4.5 and LIFT would oscillate forever on the single-member X.11 subcategories that containment itself creates — LIFT would move the leaf back to root, containment would move it back into a subcategory, next run repeats. Every deeper single-child passthrough is still fair game.
+  - **Validator.** New `LEAF-AT-WIKI-ROOT` error in `scripts/lib/validate.mjs` declaratively enforces the invariant: any non-index `.md` at the wiki root is flagged with remediation guidance to run `fix`. `heal.mjs::FINDING_ACTIONS` maps the code to `fix` so automated remediation picks the right command. Even a hand-edit that re-introduces a root-level leaf gets caught on the next validate pass.
+  - **Integration.** Phase 4.4.5 runs BEFORE Phase 4.5 review so the containment commit participates in the `--review` diff — users can drop/abort individual containment moves like any other tree-mutating phase's commits. The `anyMutation` gate that decides whether to fire review now includes `containmentDidCommit` so review is surfaced even when containment is the only phase that changed the tree. Runs BEFORE Phase 5 so `rebuildAllIndices` sees the final tree shape and the stub indexes get populated as part of the regular pass. Runs for every top-level operation that reaches convergence + review (build / rebuild / fix) — containment is an invariant the tree must satisfy, not a one-shot migration. Zero outliers → no `mkdir`, no writes, no commit; the phase is a strict no-op for clean wikis.
+  - **Tests** (8 new scenarios): zero-outliers no-op, single outlier creates folder with deterministic slug, two distinct outliers get two distinct folders, slug collision with existing subcategory triggers `-group-N` fallback, determinism across runs (byte-identical slug assignment), parents[] rewrite correctness for primary + soft-parent entries, already-`../`-prefixed parents preserved byte-identical, dotfiles / frontmatter-less root files are skipped silently.
+  - **Unblocks.** The skill-code-review Phase X.6 build landed 2 outlier leaves (`footgun-bidi-rtl-locale-collation.md`, `footgun-file-path-cross-platform.md`) at the wiki root — their TF-IDF cosine to every other leaf fell below the 0.35 soft-DAG threshold AND the 0.63 HAC threshold AND the coarse-K-means MIN_CLUSTER_SIZE=3 cutoff. Running `fix` on the existing `reviewers.wiki/` now contains them into their own subcategories and completes the invariant.
+- **Deterministic coarse-partition pre-pass for flat large-diverse roots** (`scripts/lib/cluster-detect.mjs::detectCoarseClusters`). Phase X.10 ships the engineering fix for a class of inputs that the v1.0.2 HAC path couldn't structure: a flat directory with hundreds of topically-diverse leaves. On a 596-leaf hand-authored corpus, the HAC's 3-8-size shape-score optimum produced zero valid NESTs during convergence under `--quality-mode deterministic` — the best partition at every candidate threshold was dominated by one giant component plus many singletons, and the 3-8-size clusters that emerged were too few to pass `partitionShapeScore > 0`. Balance enforcement then tried to carve the 576-leaf root linearly (3-5 leaves per sub-cluster iteration) and hit its 20-iter convergence cap far short of the `--fanout-target 6` threshold. The defense-in-depth guard from PR #8 fired, rolled back to the pre-op snapshot, and the whole 2h15m build was wasted.
+  - **Trigger.** `detectClusters` now dispatches to `detectCoarseClusters` when `leaves.length > COARSE_PARTITION_THRESHOLD` (default 50). Below that, the existing HAC path runs unchanged.
+  - **Algorithm.** Deterministic K-means with farthest-first seed init. K = `ceil(N / COARSE_TARGET_CLUSTER_SIZE)` (target 8 avg per cluster). First seed is always `leaves[0]` (lex-first by caller's ordering); each subsequent seed maximises its minimum similarity-distance (`1 - max(sim-to-existing-seeds)`). Assignment uses mean-member-similarity to each cluster so we reuse the existing NxN affinity matrix without exposing per-leaf vectors. Converges in a handful of iterations; `COARSE_KMEANS_MAX_ITERS = 20` cap is defensive against pathological oscillation.
+  - **Filters.** Clusters smaller than `MIN_CLUSTER_SIZE = 3` or larger than `MAX_COARSE_CLUSTER_SIZE = 30` are rejected. Small clusters aren't worth nesting; giant clusters are noise-floor concentration that balance enforcement can refine in a second pass if `--fanout-target` is tight.
+  - **Determinism.** All ordering uses lex-first tie-breaking: first seed index 0, subsequent farthest-first with index-ascending tie-break, member iteration in leaf-array order, post-hoc sort by (average_affinity desc, first-member-path asc). Two runs on the same corpus produce byte-identical cluster membership.
+  - **Integration.** Coarse proposals match the existing `buildNestProposal` output shape with `source: "math-coarse"` and `threshold: null` (K-means rather than HAC-at-threshold). `operators.mjs::tryClusterNestIteration` and `balance.mjs::runBalance` consume them unchanged. Tier 2 escalation for coarse paths is deliberately disabled: the LLM would be asked to partition 500+ leaves in one shot, which is both a huge token cost and typically produces worse structure than the deterministic K-means — so the coarse path returns empty on failure instead of emitting a `propose_structure` marker.
+  - New exports: `detectCoarseClusters`, `COARSE_PARTITION_THRESHOLD`, `COARSE_TARGET_CLUSTER_SIZE`, `MAX_COARSE_CLUSTER_SIZE`, `COARSE_KMEANS_MAX_ITERS`.
+  - **Tests** (5 new scenarios): constants-pin sanity, dispatch fires above threshold, byte-identical membership across runs (determinism), size filters reject tiny/giant clusters, post-sort invariant (affinity-desc, path-asc tie-break).
+  - **Unblocks.** The skill-code-review Phase X.6 rebuild of `reviewers.src/` (596 flat hand-authored leaves) can now run under `--quality-mode deterministic` without hitting the no-NEST-then-balance-stall failure mode.
+- **`--soft-dag-parents` — post-convergence DAG soft-parent synthesis.** A new Phase 4.4 hook between balance-enforcement and review synthesises soft-parent pointers for each routable leaf by scoring the leaf's TF-IDF vector against every candidate category directory's aggregate vector. Directories whose cosine similarity meets `SOFT_PARENT_AFFINITY_THRESHOLD` (0.35) become soft parents; up to `SOFT_PARENT_MAX_PER_LEAF` (3) are kept per leaf, ranked by cosine desc with POSIX-path ascending as a deterministic tie-break. Each leaf's `parents[]` frontmatter is rewritten with the primary parent first (`"index.md"` by the same convention Phase X.5's flatten preserves) and the chosen soft parents after as POSIX-relative paths to each claimed `index.md`. A companion Phase 5.1 pass, `applySoftParentEntries`, runs after `rebuildAllIndices` and propagates each leaf's soft-claim into the corresponding index's `entries[]` array — so the DAG view materialises in every claimed parent's index.md. Idempotent: records are deduped by id, and repeated runs produce byte-identical output. Build/rebuild only; intent rejects the flag elsewhere via `INT-16a`.
+  - **Signal source.** `entryText` from `scripts/lib/similarity.mjs` (focus + covers + tags + domains, focus doubled) is used verbatim so cosine scores sit on the same TF-IDF basis as Phase X.3 / tiered-AI's pairwise comparisons. Threshold calibration transfers across phases. A new companion helper `buildComparisonModelFromTexts(texts)` is exported from similarity.mjs and used here — it skips the `entryText` roundtrip `buildComparisonModel` performs, so pre-aggregated category text (already run through `entryText` per contributor) isn't double-weighted.
+  - **Traversal.** Filesystem-native (`readdirSync` + dot-skip) rather than `listChildren` for leaf discovery, so pre-bootstrap category dirs from Phase 3 draft are visible for candidate-category enumeration. Leaf routability is still validated via frontmatter-must-have-id discipline to match `listChildren`'s "movable leaf" semantics — Phase X.5 round-16 learning applied.
+  - **Determinism.** POSIX-normalised lex sort throughout (leaf order, candidate order, tie-break), deterministic frontmatter serialisation via `renderFrontmatter`. Build twice on the same tree produces byte-identical `parents[]` arrays and `entries[]` appends on every platform.
+  - **Defense-in-depth.** Intent layer rejects the flag outside `build` / `rebuild` (`INT-16a`); orchestrator gates the phase on `plan.operation ∈ {build, rebuild}`. A programmatic caller constructing a plan directly can't accidentally trigger soft-DAG synthesis on a `fix` op. Path-traversal guard in `applySoftParentEntries`: two-layer containment check. Layer 1 is a lexical guard on `path.resolve(leafDir, rel)` + wikiRoot prefix — rejects pure `..`-traversal without touching the filesystem. Layer 2 is a symlink-aware `realpathSync` containment check — resolves the full symlink chain (including intermediate symlinked directories) and rejects any resolved path that sits outside the wiki root's realpath. A hostile leaf's crafted `"../../../external/index.md"` OR `"../trap/index.md"` where `trap/index.md` symlinks out, both get silently rejected instead of mutating arbitrary filesystem paths. Per-index `try/catch` around frontmatter parse so one malformed index doesn't abort the propagation pass.
+  - **Performance.** `applySoftParentEntries` uses bounded `readFrontmatterStreaming` reads (via a new `collectAllLeaves(wikiRoot, withBody=false)` mode) rather than full `readFileSync` — O(frontmatter bytes) not O(total leaf bytes) on large corpora. One `collectAllLeaves` walk feeds a `groupLeavesByDir` map that both `collectCandidateDirs` (routable-leaf check) and `buildCategoryText` (aggregate text) consume directly; neither calls `listChildren` anymore — so the soft-DAG phase does one frontmatter pass over the tree regardless of dir count. `buildCategoryText` reads each candidate `index.md` via `readFrontmatterStreaming` rather than a full `readFileSync` — authored orientation bodies don't inflate the TF-IDF pass. `scoreCandidates` threads `ctx.threshold` through the filter rather than hard-coding the constant, so overrides work correctly.
+  - **CRLF-fence compatibility.** The `withBody=true` path in `collectAllLeaves` now also routes through `readFrontmatterStreaming` (which normalises CRLF→LF on the frontmatter payload), instead of using `readFileSync` + `parseFrontmatter` directly. `parseFrontmatter` only recognises an LF opening fence, so leaves with CRLF frontmatter (common from Windows editors) used to be silently skipped. The body slice now anchors at `captured.bodyOffset` from a raw buffer read so multi-byte characters at the fence boundary can't corrupt the body.
+  - **Actual-write stats.** `applySoftParentEntries` returns `indicesTouched` and `softEntriesAdded` from counters incremented per successful write, not from the planned-appends map. Pre-round-2 stats over-reported on idempotent reruns (every claim deduped → zero writes but `indicesTouched` still counted) and on parse-failure skips.
+  - **Review gate tracks commit reality.** The orchestrator's Phase 4.5 `anyMutation` gate uses a `softDagDidCommit` boolean tracked at commit time, not the `softParentsAdded` counter. A rerun that removes previously-synthesised soft parents now below threshold, or a canonical-order frontmatter rewrite that leaves no net soft-parent change but still alters bytes, produces `softParentsAdded === 0` but still dirties the tree and commits — the review gate correctly surfaces the diff in both cases. The commit subject is `"parents[] synthesis across N leaves"` with no added-count in the message at all — the per-phase `record()` line still surfaces `softParentsAdded` (labelled "selected", since it's the count of pointers chosen this run, not a delta) and the private-git diff is the byte-exact source of truth for what actually changed.
+  - **Non-index safety check.** `applySoftParentEntries` skips any target `index.md` whose frontmatter lacks `type: index` or a non-empty `id` — defense against a leaf claiming a path that happens to resolve to a same-named-but-non-managed markdown file. Soft-DAG propagation never mutates arbitrary content under wikiRoot.
+  - **Atomic writes.** Both leaf rewrites (`rewriteLeafParents`) and index appends (inside `applySoftParentEntries`) go through a local `atomicWriteFile` helper: write to `<path>.tmp`, then `renameSync` into place. Matches `indices.mjs::atomicWriteFile`'s discipline — a crash / SIGKILL between write and rename leaves either the old file intact or the `.tmp` orphaned, never a half-written target. Same durability guarantee the rest of the index-generation pipeline provides.
+  - New module `scripts/lib/soft-dag.mjs` exports `runSoftDagParents`, `applySoftParentEntries`, `SOFT_PARENT_AFFINITY_THRESHOLD`, `SOFT_PARENT_MAX_PER_LEAF`.
+  - `contract.mjs::SUBCOMMANDS.build` + `.rebuild` include `--soft-dag-parents` in their flag list. Extend remains a contract stub and the flag is NOT declared there.
+- **`--fanout-target=N` and `--max-depth=D` — post-convergence balance enforcement.** A new `balance-enforcement` phase between operator-convergence and index-generation iterates until fixed point applying two deterministic transform classes:
+  - **Sub-cluster overfull directories.** Any directory whose *movable* (leaf-only) fan-out exceeds `fanout-target × 1.5` is a candidate; sub-clustering extracts coherent clusters out of leaves, so subdir-heavy dirs with few leaves are un-actionable here and correctly ignored. The math cluster detector carves out the strongest coherent cluster, the Phase X.3 deterministic slug + purpose helpers name it, and `applyNest` applies it. The fanout pass walks the full lex-sorted overfull list until it finds a parent whose leaves yield a live proposal, so one un-actionable candidate never stalls the whole iteration. The "× 1.5" slack (`FANOUT_OVERLOAD_MULTIPLIER`) avoids thrashing on directories that sit one or two children above target. `computeFanoutStats` now returns both `perDir` (combined leaves+subdirs, the Claude-routing-cost view) and `leafCounts` (the movable-fanout view) from a single traversal so `detectFanoutOverload` doesn't re-walk the tree. `buildWikiForbiddenIndex` is built once at `runBalance` entry — but ONLY when `--fanout-target` is set, since `--max-depth`-only runs never call `resolveNestSlug` and would pay a full-tree walk for nothing. The index is mutated (`wikiIndex.add(resolvedSlug)`) after each successful apply, mirroring `operators.mjs::tryClusterNestIteration`'s amortisation pattern — a previous draft rebuilt it per apply, quadratic on the 596-leaf target corpus.
+  - **Flatten overdeep single-child passthroughs.** Any branch exceeding `max-depth` whose terminal segment holds exactly one subdir and zero leaves is lifted up one level. Descendants' `parents[]` paths are left unchanged — they are relative to the direct parent's `index.md`, so promoting the whole subtree up one level preserves every relative path by construction. Multi-child subcategories are left alone. `applyBalanceFlatten` preflights the passthrough dir's raw `readdirSync` entries against the allowed set `{child-basename, "index.md"}` BEFORE any mutation — catches stray non-`.md` content (assets, orphan `README.txt`, subdirs lacking `index.md`) that `listChildren` doesn't enumerate, refusing the flatten rather than silently deleting unexpected data. Dot-prefixed entries (`.DS_Store`, editor backups, `.shape/` internals) are treated as noise — not grounds for refusal, but cleaned before the rename so the final `rmdirSync` succeeds. This matches the blanket dot-skip rule the rest of the pipeline already uses (`listChildren`, `buildWikiForbiddenIndex`, `collectEntryPaths`). Final `rmdirSync` refuses non-empty dirs natively as a second safety layer (e.g., against mid-flight writes between preflight and remove).
+  - Phase runs only when at least one flag is set; otherwise it is a strict no-op. Deterministic in the inputs — POSIX-normalised lex-sorted dir iteration (sort key is relative path with `\` normalised to `/`, so ordering matches across POSIX + Windows), lex-sorted cluster-member iteration, Phase X.3 deterministic naming. Two runs on the same tree produce identical output on either OS. Balance's tree traversal (`computeDepthMap`, `computeFanoutStats`, `detectDepthOverage`) walks the filesystem directly via `readdirSync` rather than through `listChildren`'s index.md-requiring discipline — balance runs at Phase 4.3, BEFORE Phase 5's `bootstrapIndexStubs`, so category dirs created in Phase 3 draft that have leaves but no `index.md` yet would otherwise be invisible to the rebalance pass.
+  - **Hard-fail on non-convergence.** The orchestrator's Phase 4.3 hook now throws when `runBalance` reports `converged: false` (iteration cap hit without reaching a fixed point). The pre-op snapshot restores and the user sees a clear error instead of a silently half-balanced tree — an enforcement phase owes the caller a guarantee, not an advisory best-effort.
+  - New module `scripts/lib/balance.mjs` exports `runBalance`, `computeDepthMap`, `getMaxDepth`, `computeFanoutStats`, `detectFanoutOverload`, `detectDepthOverage`, `applyBalanceFlatten`, and `FANOUT_OVERLOAD_MULTIPLIER`.
+  - New intent errors: `INT-14` (invalid `--fanout-target`, must be a positive integer in [`FANOUT_TARGET_MIN`, `FANOUT_TARGET_MAX`] = [2, 100]), `INT-14a` (`--fanout-target` used on a subcommand other than build / rebuild), `INT-15` (invalid `--max-depth`, must be a positive integer in [`MAX_DEPTH_MIN`, `MAX_DEPTH_MAX`] = [1, 10]), and `INT-15a` (`--max-depth` used on a subcommand other than build / rebuild). All four surface before the orchestrator runs, so a typo never triggers a pre-op snapshot, and the orchestrator also gates the phase on `plan.operation ∈ {build, rebuild}` in defense-in-depth.
+  - `contract.mjs::SUBCOMMANDS.build` and `.rebuild` now list the two new flags so consumers gating on the contract know they're available. `extend` is intentionally NOT in that list — the operation is a stub that throws "not yet implemented", so advertising the flags on it would be a contract lie.
+- **`--quality-mode deterministic`** — a new quality mode that produces byte-reproducible wiki builds with zero LLM/sub-agent calls in the structural decision path. Complements `tiered-fast` / `claude-first` / `tier0-only`:
+  - **Pairwise decisions** (`scripts/lib/tiered.mjs::decide`): Tier 0 decisive paths fire as-is; Tier 0 mid-band escalates to Tier 1 (MiniLM embeddings, already deterministic); Tier 1 mid-band is resolved by a static threshold (`TIER1_DETERMINISTIC_THRESHOLD`, derived from the midpoint of the Tier 1 decisive bounds) instead of escalating to Tier 2. No Tier-2 escalation and no mid-band "undecidable" outcome — Tier 1 always produces a concrete same/different, and `tier2Handler` is never invoked. (Tier 0's "insufficient-text" undecidable on empty-frontmatter pairs is a separate path that predates this mode and is unchanged by design: an empty text pair can't be discriminated by any tier, regardless of quality mode.)
+  - **Cluster NEST** (`scripts/lib/operators.mjs::tryClusterNestIteration`): the `propose_structure` Tier 2 request is skipped entirely; math-only candidates bypass the `nest_decision` gate (auto-approved — the partition-shape score + metric regression gate already provide an algorithmic equivalent); math-only candidates also bypass the `cluster_name` request and receive a deterministic slug from `generateDeterministicSlug()` + a deterministic purpose from `deterministicPurpose()`.
+  - **Deterministic slug algorithm** (`scripts/lib/cluster-detect.mjs::generateDeterministicSlug`): TF-IDF over member frontmatters with the siblings' corpus as the IDF context, ranked `(weight desc, term asc)` for lex tie-breaking, top 1–2 valid tokens joined with `-`. Falls back to `cluster-<7-hex-fnv1a>` when no token survives the slug regex — still deterministic, still member-derived. Byte-stable across member shuffles.
+  - **Use case**: the mode to pair with an upcoming `--fanout-target` / `--max-depth` balance pass and soft-parent synthesis for large hand-authored corpora where reproducible builds matter more than the extra nuance an LLM adds at Tier 2.
+
+### Removed
+
+- **`--quality-mode tier0-only`** — removed entirely. The mode was a narrow "hermetic CI / no-Claude" path that returned `"undecidable"` for every Tier 0 mid-band pair, forcing manual resolution via the interactive review flow. With `--quality-mode deterministic` (Phase X.3, shipped above) covering the same "no LLM / no sub-agent call" use case AND resolving mid-band pairs via the static Tier 1 threshold (no interactive-review fallback required), keeping tier0-only around was pure maintenance tax. Removal touches `scripts/lib/intent.mjs::VALID_QUALITY_MODES`, `scripts/lib/tiered.mjs::QUALITY_MODES`, the mid-band-undecidable branch in `decide()`, the `INT-13` validation message, the CLI help, methodology.md / README.md / guide sections, and their tests. `--quality-mode tier0-only` now raises `INT-13` at the intent layer — users should migrate to `deterministic` (the path for every tier0-only use case) or `tiered-fast` (the default). Note: "deterministic" eliminates runtime Claude calls but the Tier 1 MiniLM model is still downloaded on first use by `@xenova/transformers` if the local cache is cold — fully air-gapped use requires pre-warming `~/.cache/huggingface` on a networked host. A companion `tests/unit/intent-resolve.test.mjs::"VALID_QUALITY_MODES is in sync with tiered.mjs::QUALITY_MODES"` test guards the mode-list duplication between intent.mjs and tiered.mjs so future drift fails loud.
+
+### Fixed
+
+- **`LLM_WIKI_QUALITY_MODE` env var now actually works.** The env var was documented and tested through `tiered.mjs::resolveQualityMode`, but the orchestrator bypassed that helper with `plan.flags?.quality_mode || "tiered-fast"` — so the env var silently had no effect on CLI runs. Orchestrator now calls `resolveQualityMode(plan.flags)` at both the convergence phase and the balance phase, wiring the env var through end-to-end. The flag still wins when both flag and env are set. Invalid values raise `INT-13` at the intent layer on BOTH paths with identical valid-values suggestions (flag and env share the structured-error shape), so a stale `LLM_WIKI_QUALITY_MODE=tier0-only` in a shell profile fails loud on the next invocation rather than surfacing as a generic exit-1. Env-var validation is gated to subcommands that actually consume quality mode (`build` / `extend` / `rebuild` / `fix` / `join`) so a stale env var doesn't lock the user out of recovery paths like `rollback` or `validate`.
+- **Convergence cap starved cluster-NEST on large-merge-heavy corpora.** `operators.mjs::MAX_CONVERGENCE_ITERATIONS` was 20. `runConvergence` applies at most ONE pairwise operator (DESCEND/LIFT/MERGE) per outer iteration, and pairwise ops always outrank cluster NEST (which only runs when no pairwise op fires in a given iteration). Once cluster NEST IS reached, `tryClusterNestIteration` can apply multiple NEST commits in that single pass — the scarcity is at the outer iteration level, not per-NEST. On the 596-leaf `skill-code-review/reviewers.src/` fixture — now reproducibly re-run as a post-merge X.6 smoke test — Tier 1 similarity finds ~20 viable MERGE pairs. Iterations 1-20 all apply MERGE, convergence hits the 20-cap, and the Phase X.10 coarse-partition (which emits ~75 top-level NESTs for a flat 596-leaf root) never runs even once. The downstream balance phase then tried to linearly carve a 576-leaf root, hit its own 20-iter cap, and rolled back. Bumped to `MAX_CONVERGENCE_ITERATIONS = 200` — enough for 20 pairwise ops + at least one cluster-NEST pass (which internally lands all ~75 NESTs) + follow-up MERGEs after NESTs reveal new overlaps. Small wikis exit early via the "no ops fired this iteration" break; the raise is effectively free for them. Caller override via `runConvergence({ maxIterations })` is unchanged. (Follow-up: the real architectural fix is to let cluster NEST interleave with pairwise ops in the same iteration rather than run only as a fallback — tracked separately, not blocking X.6.)
+- **Cross-depth slug collision guard in `resolveNestSlug`.** The v1.0.0 collision resolver checked only the cluster's immediate parent directory, missing collisions with leaf ids or subdirectory basenames elsewhere in the tree. On real-world multi-branch wikis (first observed during a 596-leaf novel-corpus build in the consumer skill `skill-code-review`), Tier 2's `propose_structure` picked slug `event-patterns` for a cluster under `design-patterns-group/` — colliding with an existing leaf at `arch/event-patterns/index.md` (id: `event-patterns`) in a completely different branch. The parent-dir-only walk missed it; validation caught `DUP-ID` post-apply and forced a rollback. The resolver's API now: `resolveNestSlug(slug, proposal, wikiRoot, opts)` gains an optional `wikiRoot` third argument and an `opts.wikiIndex` escape hatch. When `wikiRoot` is supplied, the internal `collectForbiddenIdsPredicate` returns a predicate backed by a local parent-dir set PLUS either the caller's precomputed index (via `opts.wikiIndex`) or a fresh full-tree walk (via the new private `walkWikiIds`). A new exported helper `buildWikiForbiddenIndex(wikiRoot)` materialises the wiki-wide id + directory-basename set once per convergence iteration — `operators.mjs::tryClusterNestIteration` precomputes it when at least one proposal is picked, mutates it by `wikiIndex.add(resolvedSlug)` after each successful apply, and passes it through `opts.wikiIndex` so each slug-resolve call runs in O(parent-dir) instead of O(full-tree). Total cost across a multi-NEST iteration: O(#files + #applies) instead of O(#applies × #files). Dot-prefixed entries (directories AND files — `.llmwiki/`, `.work/`, `.git/`, `.github/`, stray `.DS_Store` / `.foo.md`) are skipped under a blanket rule matching `scripts/lib/chunk.mjs::collectEntryPaths` discipline. Per-file frontmatter is read via `readFrontmatterStreaming` for bounded reads at the ~600-leaf scale. Legacy callers that don't pass `wikiRoot` continue to get the parent-dir-only behaviour, so the change is backward-compatible. Fixes issue [#4](https://github.com/ctxr-dev/skill-llm-wiki/issues/4) bug 2.
+- **Pre-apply alias collision guard in MERGE.** `applyMerge` in `scripts/lib/operators.mjs` now walks the full wiki (skipping every dot-directory) to collect every live entry id (every `.md` file's frontmatter `id:`, including `index.md` entries) before it writes the keeper's new `aliases[]`. If any of the new aliases (absorbed's id or any of absorbed's own aliases) would collide with a live id elsewhere in the tree, the merge is refused pre-apply with a clear error — nothing is written, nothing is deleted, the convergence iteration can continue with the next proposal. Before this fix, such collisions were caught downstream at validation as `ALIAS-COLLIDES-ID`, forcing a full pipeline rollback. The guard is defensive and targets the multi-operator-per-iteration reach-state that produced the collisions during the consumer-skill `skill-code-review` 596-leaf novel-corpus build (3 MERGE pairs hit this during Bundle A2: `pattern-eip-messaging↔pattern-eip-endpoint`, `smell-data-class↔antipattern-anemic-domain-model`, `smell-duplicate-code↔antipattern-copy-paste`). The new helper `collectLiveIds(wikiRoot, excludePaths)` is exported so future operators (e.g. the real `join` implementation) can reuse it. Fixes issue [#4](https://github.com/ctxr-dev/skill-llm-wiki/issues/4) bug 3.
+
+### Changed
+
+- **`VALID_QUALITY_MODES`** (`scripts/lib/intent.mjs`) and **`QUALITY_MODES`** (`scripts/lib/tiered.mjs`) now include `"deterministic"`. The `intent-resolve` and `QUALITY_MODES` canonical-allow-list tests follow.
+- **NEST audit-trail semantics for deterministic mode.** `appendNestDecision` in `scripts/lib/decision-log.mjs` previously hard-coded `tier_used: 2` for every NEST entry — correct before deterministic mode existed, since every NEST touched Tier 2 via `propose_structure` or `nest_decision`. Under `--quality-mode deterministic` no sub-agent is ever consulted for math candidates, so entries from that path now record `tier_used: 0` and a new `confidence_band: "deterministic-math"` distinguishes them from `"math-gated"` (which still means "math candidate that passed a Tier 2 gate" under the other quality modes). Tooling and tests that filter `decisions.yaml` by `tier_used` to reason about sub-agent costs now see accurate zeros on the deterministic path. Call sites that don't supply `tier_used` still default to 2, so every existing non-deterministic entry is byte-identical.
+
+### Tests
+
+- `tests/unit/soft-dag.test.mjs` — 17 new scenarios covering `runSoftDagParents` (no-op empty wiki, zero soft parents on unrelated topics, multiple-category overlap adds soft parents, `maxPerLeaf` cap honoured, determinism across runs, primary-parent preservation at `parents[0]`, CRLF-fence leaf recognition) and `applySoftParentEntries` (append into claimed parent's entries[], idempotency across runs, ignore primary-only leaves, path-traversal rejection via lexical + symlink-realpath defense-in-depth guard, tolerate malformed `parents[]` entries, stats reflect actual writes not planned appends, end-to-end synthesis + propagation). Plus `tests/unit/intent-resolve.test.mjs` — 2 new scenarios covering `INT-16a` subcommand-scoping rejection on `fix`/`extend`/`validate`/`join`/`rollback` and strict `status === "ok"` acceptance on `build`.
+- `tests/unit/balance.test.mjs` — 18 new scenarios: depth-map (plus non-wiki-node dirs skipped under the `index.md`-only discipline), max-depth, fanout-stats (including `leafCounts` return), overload detection (with and without `nestedParents` exclusion, plus a leaf-metric regression guard against flagging dirs overfull only via subdir count, plus a POSIX-sort-key regression guard for cross-platform ordering), depth-overage detection (only single-child passthroughs), flatten happy path + refuse on multi-child, flatten refuses when passthrough holds stray non-`.md` content (defensive emptiness check), flatten tolerates + cleans dot-prefixed noise (`.DS_Store`) under the blanket dot-skip rule, `runBalance` no-op when flags absent, fanout-only pass, depth-only pass, fanout pass skips un-actionable `overfull[0]` and acts on a later candidate, multiplier constant pin.
+- `tests/unit/intent-resolve.test.mjs` — 7 new scenarios covering INT-14 / INT-15 accept/reject boundaries plus INT-14a / INT-15a subcommand-scoping rejection on `fix`/`extend`/`validate` and acceptance on `build`/`rebuild`.
+- New scenarios in `tests/unit/nest-applier.test.mjs` for cross-depth collision, subdir-basename collision, `-group-N` chain fallback, dot-prefixed skip, clean-tree no-op, same-depth regression guard, `buildWikiForbiddenIndex` snapshot shape, `opts.wikiIndex` short-circuit semantics, caller-mutation round-trip, and wiki-root `index.md` id capture (in both the legacy walk and the precomputed-index path). All pre-existing tests pass unchanged.
+- 4 new scenarios in `tests/unit/operators.test.mjs` covering the guard-trips-on-collision path, the guard-permits-clean-merge regression, and two `collectLiveIds` helper tests (skips `.llmwiki/`/`.work/`, honours `excludePaths`).
+- New deterministic-mode coverage across three files:
+  - `tests/unit/tiered.test.mjs` — `QUALITY_MODES` canonical allow-list including `"deterministic"`, a constant-derivation pin for `TIER1_DETERMINISTIC_THRESHOLD` (midpoint of the Tier 1 decisive bounds), a joint Tier 0 + Tier 1 mid-band sweep that exercises the `confidence_band === "deterministic-mid-band"` branch empirically, byte-stability across runs, non-escalation to Tier 2, and Tier 0 decisive-path preservation under the new mode.
+  - `tests/unit/cluster-detect.test.mjs` — `generateDeterministicSlug` (distinguishing-token selection, order invariance, multi-run stability, hash fallback determinism, precomputed-IDF equivalence) and `deterministicPurpose` (most-shared cover, lex tie-break, focus fallback, plain-frontmatter input equivalence).
+  - `tests/e2e/determinism.test.mjs` — new full-build test: two independent `build --quality-mode deterministic` runs on a 6-leaf two-theme corpus must produce byte-identical tree SHAs; the cluster-nest path is in-frame (not skipped); audit-trail check hard-asserts ≥1 NEST entry carrying `tier_used: 0` + `confidence_band: "deterministic-math"`.
+- `tests/unit/intent-resolve.test.mjs` — extended the `--quality-mode` acceptance test to cover `"deterministic"` alongside the existing three modes.
+- 3 skipped (unchanged from prior baseline; all opt-in gates). 0 failing on `ubuntu-latest` / `windows-latest` (CI) and locally.
+
 ## [1.0.0] — 2026-04-16
 
 First stable release. The semantic-routing substrate landed in v0.4.0, multi-NEST convergence landed in v0.4.1, and 1.0.0 closes the remaining sharp edge — a DUP-ID collision path discovered during the v0.4.1 deferred novel-corpus validation — plus the Windows CI parity gap. The v0.4.1 "Known remaining gaps" novel-corpus validation item is **resolved**: the combined `skill-code-review/reviewers/` + `overlays/` corpus (45 leaves) now builds end-to-end on the first try, `validate` returns 0 errors / 0 warnings, and multi-NEST applies atomically in a single convergence iteration. Semver commitments are now in effect: the six public operations (Build, Extend, Validate, Rebuild, Fix, Join), the CLI exit-code surface, the layout-mode contract, and the private-git history shape are stable and will not break in 1.x.

package/README.md

@@ -62,7 +62,7 @@ Every phase is a git commit in the wiki's private history, so you can inspect, d
 - **Stable sibling layout.** `<source>.wiki/` is the one folder a wiki ever lives in. No more `.llmwiki.v1`/`.v2`/`.v3` directory proliferation — prior states are reachable as git tags (`pre-op/<id>`, `op/<id>`) in the private repo.
 - **Three layout modes, never guessed.** `sibling` (default), `in-place` (source IS the wiki), and `hosted` (user-chosen path with a `.llmwiki.layout.yaml` contract). Ambiguous invocations refuse and prompt — see the "Ask, don't guess" rule.
 - **User-repo coexistence.** An auto-generated `.gitignore` hides the private metadata from any ancestor user git. The skill's isolation env block (`GIT_DIR`, `GIT_CONFIG_NOSYSTEM`, `core.hooksPath=/dev/null`, …) keeps the two gits from leaking into each other.
-- **Tiered AI strategy.** TF-IDF (free) → local MiniLM embeddings (required, ~23 MB one-time model download, zero-API) → Claude (only for mid-band ambiguity and decisions requiring natural-language judgment). `--quality-mode tiered-fast|claude-first|tier0-only` selects the escalation policy.
+- **Tiered AI strategy.** TF-IDF (free) → local MiniLM embeddings (required, ~23 MB one-time model download, zero-API) → Claude (only for mid-band ambiguity and decisions requiring natural-language judgment). `--quality-mode tiered-fast|claude-first|deterministic` selects the escalation policy.
 - **Deterministic slug collisions.** NEST operator auto-resolves slug-vs-member-id collisions with a deterministic `-group` suffix before apply. Your convergence loop never needs manual retries for DUP-ID.
 - **Optional interactive review.** `skill-llm-wiki rebuild <wiki> --review` prints the post-convergence diff and commit list, lets the user approve / abort / `drop:<sha>` specific iterations, and re-runs validation + index regen on the reverted tree.
 - **Windows parity.** The CI matrix runs the smoke suite on both `ubuntu-latest` and `windows-latest`; the isolation env switches `/dev/null` to `NUL` and enables `core.longpaths=true` on Windows.
@@ -463,7 +463,7 @@ Quality modes select the escalation policy:
 
 - `tiered-fast` (default) — full Tier 0 → 1 → 2 ladder.
 - `claude-first` — skip Tier 1; mid-band Tier 0 escalates straight to Claude.
-- `tier0-only` — air-gapped mode; mid-band becomes an "undecidable" marker resolved via the interactive review flow.
+- `deterministic` — no LLM in the loop; Tier 1 mid-band resolved by a static threshold, cluster naming produced from member frontmatter. Byte-reproducible builds for air-gapped / hermetic CI use.
 
 Tier 1 uses `@xenova/transformers` running `Xenova/all-MiniLM-L6-v2` locally via ONNX (~23 MB one-time model download, ~50 ms per text on CPU, zero API cost). It is a **required** runtime dependency since v0.4.0 — the dependency preflight at CLI startup verifies it is resolvable, and will offer to `npm install` it on a fresh checkout if it is missing.
 

package/SKILL.md

@@ -1,6 +1,7 @@
 ---
 name: skill-llm-wiki
 description: Use when the user explicitly asks to build, extend, validate, repair, rebuild, or merge an LLM-optimized knowledge wiki from markdown notes, documentation, source code, or mixed folders. Default output is a single stable sibling `<source>.wiki/` with full history in a private git repo under `.llmwiki/git/`; `--layout-mode in-place` transforms the source folder itself, and `--layout-mode hosted --target <path>` honours a user-provided `.llmwiki.layout.yaml` contract. SKILL.md is the entry point; detailed operation instructions are loaded on demand from `guide/` per the routing procedure below.
+format_version: 1
 ---
 
 # skill-llm-wiki
@@ -41,6 +42,12 @@ Every operation is a git sequence: `preflight → pre-op snapshot → phase comm
 
 **Ambiguous invocations refuse and prompt.** If the user's request could mean two things (a default sibling would stomp on a foreign directory, a hosted target has no contract, `--layout-mode in-place` is combined with `--target`, …), the CLI exits with code 2 and a structured `INT-NN` error rather than guessing. See `guide/ux/user-intent.md` for the full list.
 
+## Integrating another skill or agent as a consumer
+
+If the user is building a skill, agent, or CI job that calls this skill programmatically (rather than asking you to build a wiki for them in this session), route them to `guide/consumers/index.md`. That subtree answers: how to gate on `format_version`, how to `init` a topic wiki in one command, how to `heal` after every leaf write, how to detect the skill-absent case, how to write consumer tests against the shipped `scripts/testkit/` helpers. Every consumer recipe is dispatched from `guide/consumers/index.md`; do not re-derive the integration path from SKILL.md alone.
+
+Quick probe summary: `skill-llm-wiki contract --json` returns `format_version` + the full CLI + frontmatter schema, and `skill-llm-wiki where --json` returns absolute install paths. Both are exempt from the runtime-dep preflight so consumers can probe before the skill is fully set up.
+
 ## Non-automation contract
 
 This skill has **no hooks, no filesystem watchers, no PostToolUse listeners, no install-time wiring, no background processes**. Every action happens only in direct response to an explicit user request against an explicit target directory. If the user did not ask, do nothing. Do not propose automation. Do not create hooks. Do not schedule anything.

package/guide/cli.md

@@ -11,8 +11,8 @@ covers:
   - "hidden-git plumbing: log (+ --op), show, diff (+ --op), blame, reflog, history"
   - "remote mirroring: remote add/list/remove, sync with tag-only default refspec"
   - "layout mode flags (--layout-mode sibling|in-place|hosted, --target)"
-  - "tiered-AI flags (--quality-mode tiered-fast|claude-first|tier0-only)"
-  - "UX flags (--no-prompt, --json-errors, --accept-dirty, --accept-foreign-target, --review)"
+  - "tiered-AI flags (--quality-mode tiered-fast|claude-first|deterministic)"
+  - "UX flags (--no-prompt, --json, --json-errors (legacy alias), --accept-dirty, --accept-foreign-target, --review)"
   - "internal helpers: ingest, draft-leaf, draft-category, index-rebuild, index-rebuild-one, shape-check"
   - "exit code summary (0 ok, 1 usage, 2 validation/ambiguity/review-abort, 3 resolve miss, 4 node too old, 5 git missing/too old, 6 wiki corrupt, 7 NEEDS_TIER2 suspend-and-resume, 8 DEPS_MISSING runtime dependency missing)"
 tags:
@@ -230,12 +230,14 @@ All top-level operations accept:
 
 ## Tiered-AI flags
 
-- `--quality-mode tiered-fast|claude-first|tier0-only` — select the escalation policy. Default `tiered-fast` (TF-IDF → MiniLM embeddings → Claude). `tier0-only` never calls Claude, never loads Tier 1; mid-band pairs become "undecidable" markers the user resolves interactively. Unknown values raise `INT-13`.
+- `--quality-mode tiered-fast|claude-first|deterministic` — select the escalation policy. Default `tiered-fast` (TF-IDF → MiniLM embeddings → Claude). `deterministic` never calls Claude; Tier 1 mid-band pairs are resolved by a static threshold so repeated runs on the same inputs are byte-reproducible. Unknown values raise `INT-13`.
 
 ## UX flags
 
 - `--no-prompt` / env `LLM_WIKI_NO_PROMPT=1` — fail loudly on any ambiguity instead of prompting; emits `INT-12` if the skill would otherwise ask a TTY question.
-- `--json-errors` — emit `INT-NN` ambiguity errors as JSON on stderr instead of numbered-options text.
+- Env `LLM_WIKI_NO_PROGRESS=1` — suppress per-phase progress breadcrumbs that the CLI otherwise streams to stderr during long-running operations (build / rebuild / fix / join). Useful for CI jobs that already capture their own progress or that want the pre-X.9 silent stderr shape. `--json` implicitly sets this (the `skill-llm-wiki/v1` envelope consumer contract requires a clean stderr).
+- `--json` — canonical machine-output flag. Enables the `skill-llm-wiki/v1` envelope on subcommands that emit one (validate, init, heal, rollback) and switches `INT-NN` ambiguity errors to JSON on stderr. The consumer-facing probe subcommands `contract` and `where` always emit JSON when this flag is present.
+- `--json-errors` — legacy alias for `--json`, kept for consumers that adopted the flag before the envelope shipped. Triggers identical behaviour. New code should pass `--json`.
 - `--accept-dirty` — operate on a source inside a dirty user git repo (escape hatch for `INT-08`).
 - `--review` — enable the `rebuild` interactive review cycle. See `guide/operations/rebuild.md`.
 

package/guide/consumers/index.md

@@ -0,0 +1,106 @@
+---
+id: consumers
+type: index
+depth_role: subcategory
+depth: 1
+focus: "Integrating another skill or agent as a consumer of skill-llm-wiki"
+parents:
+  - ../index.md
+tags:
+  - consumers
+  - integration
+  - agent
+  - skill
+  - recipes
+activation:
+  keyword_matches:
+    - consumer
+    - consume
+    - integrate
+    - integration
+    - my skill uses
+    - my agent uses
+    - wrap
+    - depend on
+    - hard dependency
+  tag_matches:
+    - consumers
+    - integration
+  escalation_from:
+    - build
+    - init
+    - heal
+    - contract
+    - where
+orientation: |
+  Route here when a user is building another skill, agent, or CI job
+  that calls skill-llm-wiki programmatically. The recipes under
+  recipes/ cover the canonical patterns: seed a dated or subject
+  wiki, run heal after every leaf write, gate CI on validate, handle
+  the skill-absent case, and wire up a consumer test suite. Each
+  recipe names the exact commands and the envelope fields consumers
+  read.
+
+  Not for: end users who are building a wiki for themselves (see
+  SKILL.md + guide/operations/ instead).
+
+entries:
+  - id: consumers-quickstart
+    file: quickstart.md
+    type: primary
+    focus: "Ten-minute integration path: detect the skill, init a topic, heal after writes"
+    tags: [quickstart, integration]
+  - id: recipe-dated-wiki
+    file: recipes/dated-wiki.md
+    type: primary
+    focus: "Init a dated topic (reports, sessions, regressions) with one command"
+    tags: [dated, init, hosted]
+  - id: recipe-subject-wiki
+    file: recipes/subject-wiki.md
+    type: primary
+    focus: "Init a subject topic (runbooks, adrs) with nested categories"
+    tags: [subject, init, hosted]
+  - id: recipe-post-write-heal
+    file: recipes/post-write-heal.md
+    type: primary
+    focus: "The canonical after-every-leaf-write heal invocation"
+    tags: [heal, envelope, post-write]
+  - id: recipe-ci-gate
+    file: recipes/ci-gate.md
+    type: primary
+    focus: "Gate CI on skill-llm-wiki validate --json"
+    tags: [ci, validate, gate]
+  - id: recipe-skill-absent
+    file: recipes/skill-absent.md
+    type: primary
+    focus: "Detect the skill is missing and surface an upgrade path without silently degrading"
+    tags: [skill-absent, preflight, install-hint]
+  - id: recipe-testing
+    file: recipes/testing.md
+    type: primary
+    focus: "Use the shipped testkit in consumer test suites"
+    tags: [testing, testkit, fixtures]
+  - id: recipe-format-gate
+    file: recipes/format-gate.md
+    type: primary
+    focus: "Gate consumer CI on skill-llm-wiki contract --json format_version"
+    tags: [contract, format-version, compatibility]
+
+generator: "skill-llm-wiki/v1"
+---
+
+# Consumers
+
+Every consumer recipe in this subtree follows the same shape:
+
+1. **Trigger** — when a consumer should activate this recipe.
+2. **Commands** — the exact CLI invocations with every flag.
+3. **Envelope fields** — which fields the consumer reads from the
+   `--json` envelope to decide what to do next.
+4. **Minimum checked-in code** — the consumer-side shell or script
+   that the recipe maps to, kept to the smallest honest example.
+5. **Failure modes** — what happens when each step goes wrong and
+   how the recipe tells the consumer to react.
+
+Consumers should read [quickstart.md](quickstart.md) first, then
+pick the recipes matching their integration shape.

package/guide/consumers/quickstart.md

@@ -0,0 +1,96 @@
+---
+id: consumers-quickstart
+type: primary
+depth_role: leaf
+focus: "Ten-minute integration path for any consumer that shells out to skill-llm-wiki"
+parents:
+  - index.md
+tags:
+  - quickstart
+  - integration
+  - consumers
+activation:
+  keyword_matches:
+    - quickstart
+    - getting started
+    - first integration
+    - how do i call
+    - new consumer
+  tag_matches:
+    - quickstart
+
+generator: "skill-llm-wiki/v1"
+---
+
+# Quickstart
+
+You are building a skill, agent, or CI job that wants to manage docs via `skill-llm-wiki`. Here is the shortest path that gets you working without reinventing the common scaffolding.
+
+## 1. Declare the dependency and gate on format_version
+
+`skill-llm-wiki` ships a machine-readable version contract. Read it in your preflight:
+
+```bash
+skill-llm-wiki contract --json | jq -r '.format_version'
+```
+
+Fail fast if that number is below your required minimum. See [recipes/format-gate.md](recipes/format-gate.md) for the canonical gate.
+
+## 2. Locate the install path once
+
+Every other consumer concern (templates, testkit, SKILL.md) hangs off one canonical probe:
+
+```bash
+skill-llm-wiki where --json
+```
+
+Fields you will actually use: `skill_root`, `skill_md`, `templates_dir`, `testkit_dir`. Stop hard-coding `~/.claude/skills/ctxr-skill-llm-wiki/...`; read it from the probe.
+
+## 3. Init a topic wiki with one command
+
+Instead of copying a layout contract and then running build with the right flags, use:
+
+```bash
+skill-llm-wiki init .development/shared/reports --kind dated --template reports --json
+```
+
+The skill seeds the contract from its own shipped templates and prints the exact `build` command to run next. See [recipes/dated-wiki.md](recipes/dated-wiki.md) and [recipes/subject-wiki.md](recipes/subject-wiki.md).
+
+## 4. After every leaf write, run heal
+
+Your consumer writes a leaf. Immediately call:
+
+```bash
+skill-llm-wiki heal .development/shared/reports --json
+```
+
+Parse the envelope. Switch on `verdict`:
+
+- `ok`: nothing to do.
+- `fixable`: run `skill-llm-wiki fix <wiki>`.
+- `needs-rebuild`: run `skill-llm-wiki rebuild <wiki>`.
+- `broken`: surface the diagnostics to the user; do not auto-mutate.
+
+See [recipes/post-write-heal.md](recipes/post-write-heal.md) for the full envelope handling.
+
+## 5. Gate CI on validate
+
+Your CI job runs `skill-llm-wiki validate <wiki> --json` against every wiki the project ships. See [recipes/ci-gate.md](recipes/ci-gate.md).
+
+## 6. Handle the skill-absent case explicitly
+
+If your consumer is a hard dependency on `skill-llm-wiki`, refuse to run when the skill is missing and point the user at the install command. See [recipes/skill-absent.md](recipes/skill-absent.md) for the canonical message and exit shape.
+
+## 7. Use the shipped testkit in your test suite
+
+Hand-rolled stubs and fixtures drift over time. Import from `scripts/testkit/` (path discoverable via `where --json`). See [recipes/testing.md](recipes/testing.md).
+
+## What this replaces
+
+- Hand-written layout YAML files duplicated across every consumer.
+- `validate → fix → rebuild` ladders reimplemented per consumer.
+- Three-step path discovery for SKILL.md.
+- Drift-detection tests against SKILL.md prose.
+- Hand-rolled presence stubs.
+
+The envelope schema is stable across `format_version` 1; consumers gate on that integer and bump when the skill does.

package/guide/consumers/recipes/ci-gate.md

@@ -0,0 +1,125 @@
+---
+id: recipe-ci-gate
+type: primary
+depth_role: leaf
+focus: "Gate CI on skill-llm-wiki validate --json"
+parents:
+  - ../index.md
+tags:
+  - ci
+  - validate
+  - gate
+  - consumers
+activation:
+  keyword_matches:
+    - ci gate
+    - ci validation
+    - github actions
+    - gitlab pipeline
+    - pre-commit wiki
+  tag_matches:
+    - ci
+    - gate
+
+generator: "skill-llm-wiki/v1"
+---
+
+# Recipe: CI gate
+
+## Trigger
+
+Your consumer wants CI to reject PRs that break wikis shipped in the repository.
+
+## Commands
+
+```bash
+skill-llm-wiki validate .development/shared/reports --json
+```
+
+Exit code `0` → clean. Exit code `2` → errors found; CI should fail.
+
+## Envelope fields
+
+```json
+{
+  "schema": "skill-llm-wiki/v1",
+  "command": "validate",
+  "target": "/abs/.../reports",
+  "verdict": "ok" | "broken",
+  "exit": 0 | 2,
+  "diagnostics": [
+    { "code": "IDX-01", "severity": "warning", "path": "...", "message": "..." },
+    { "code": "PARSE",  "severity": "error",   "path": "...", "message": "..." }
+  ],
+  "timing_ms": 412
+}
+```
+
+CI should fail when `exit !== 0` AND any diagnostic with `severity: "error"` is present. A warning-only run exits 0.
+
+## GitHub Actions example
+
+```yaml
+# .github/workflows/wiki-validate.yml
+name: wiki-validate
+on: [pull_request]
+jobs:
+  validate:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-node@v4
+        with: { node-version: "20" }
+      - run: npm i -g @ctxr/skill-llm-wiki
+      - name: contract gate
+        run: |
+          FV=$(skill-llm-wiki contract --json | jq -r '.format_version')
+          if [ "$FV" -lt 1 ]; then
+            echo "skill-llm-wiki format_version=$FV is below required 1" >&2
+            exit 1
+          fi
+      - name: validate reports
+        run: skill-llm-wiki validate .development/shared/reports --json | tee validate.json
+      - name: fail on error diagnostics
+        run: |
+          errs=$(jq '[.diagnostics[] | select(.severity == "error")] | length' validate.json)
+          if [ "$errs" -gt 0 ]; then
+            jq '.diagnostics[] | select(.severity == "error")' validate.json
+            exit 1
+          fi
+```
+
+## Pre-commit / husky example
+
+```json
+// package.json
+"scripts": {
+  "wiki:validate": "skill-llm-wiki validate .development/shared/reports --json | node scripts/wiki-validate-check.mjs"
+}
+```
+
+```js
+// scripts/wiki-validate-check.mjs
+import { readFileSync } from "node:fs";
+const env = JSON.parse(readFileSync(0, "utf8"));
+const errors = env.diagnostics.filter((d) => d.severity === "error");
+if (errors.length > 0) {
+  for (const e of errors) console.error(`[${e.code}] ${e.path}: ${e.message}`);
+  process.exit(1);
+}
+```
+
+## Failure modes
+
+- Validate exits `2`: error-severity findings present in the wiki; `.diagnostics` contains one entry per finding with its code, severity, path, and message.
+- Validate exits `4`: Node.js too old on the CI runner. Upgrade to Node 20 or the version pinned in the skill's `.nvmrc`.
+- Validate exits `5`: git missing or too old on the runner.
+- Validate exits `7`: impossible for `validate` alone; only reachable from build/extend/rebuild (Tier 2 suspend-and-resume).
+- Validate exits `8`: runtime deps missing on CI runner. Install `@ctxr/skill-llm-wiki` (and its transitive deps) in the CI image.
+- For wiki-substrate corruption (missing `.llmwiki/git/`, divergent refs), run `skill-llm-wiki heal <wiki> --json` instead — `heal` is the subcommand that classifies substrate state and surfaces verdict `broken` with exit 6.
+
+## Do not
+
+- Parse `stderr` for errors. The envelope on stdout is the contract; stderr is free-form logging.
+- Silence errors with `|| true`. Validate failures are real; a broken wiki in `main` propagates to every downstream consumer.
+- Rerun validate in a loop until it passes. If it fails once, it fails deterministically.