unknown-knowledge 2.1.0 → 3.0.0-rc.1

package/README.md

@@ -14,6 +14,12 @@ branched and merged by your normal PRs.
 
 ## Quickstart
 
+The 3.0 pilot is available explicitly with
+`npx unknown-knowledge@3.0.0-rc.1 init`. The stable `latest` channel remains
+2.1.0. Existing installations should follow the
+[migration guide](https://github.com/doterodesign/unknown-knowledge/blob/main/docs/migration-3.md);
+init refuses existing roots and does not overwrite client-owned records.
+
 ```bash
 cd your-repo
 npx unknown-knowledge init          # seeds unknown-knowledge/ and an agent wrapper
@@ -52,13 +58,15 @@ wrong parse is a false all-clear. What it could not read is recorded in
 
 ## The engine
 
-Ten command-line surfaces. JavaScript with JSDoc types, zero build step, one
+Twelve command-line surfaces. JavaScript with JSDoc types, zero build step, one
 dependency (D-022).
 
 | Command | Answers |
 | --- | --- |
 | `validate.js` | is the store structurally sound? |
 | `validate-values.js` | do the Concepts still match the code they point at? |
+| `commit-check.js` | do both whole-store validators pass the commit gate? |
+| `reverse-staged.js` | what governed each staged path before and after this commit? |
 | `preflight.js` | which Concepts may this agent trust, right now? |
 | `resolve.js` | what does the store know about these terms or paths? |
 | `survey-map.js` | what is in this repo, and what could not be surveyed? |
@@ -84,6 +92,12 @@ exit `1`, that agent would walk straight past a check that never happened. So a
 crash always exits `2`, and a test enumerates every surface, forces a bug into
 each, and proves it.
 
+The resolver also accepts repeatable `--path` values for lossless filename
+transport, for example `--path 'src/a,b.ts' --path 'src/my file.ts'`. Legacy
+comma-separated `--paths` remains supported; mixing the forms fails with
+exit 2. See the [complete-filename and safe programmatic invocation guide](payload/docs/README.md#reverse-lookup-for-complete-filenames).
+This is an additive MINOR CLI surface change under D-021.
+
 The reverse audit is advisory: its findings are proposals for human review, and
 never a gate. A human may opt in with `--fail-on-findings`, and that is never a
 shipped CI default.
@@ -91,8 +105,8 @@ shipped CI default.
 ## Guarantees
 
 - **The engine never executes your code** (D-014). No `eval`, no importing your
-  modules, no spawning your build. Parsing is lexical; the one subprocess it
-  runs is `git ls-files`.
+  modules, no spawning your build. Parsing is lexical; subprocesses invoke
+  Git only, to list tracked files and read the proposed commit snapshot.
 - **No network, ever.** Nothing is uploaded, and nothing is fetched.
 - **Deterministic.** Same tree in, byte-identical output out. Dates are
   injected, never read from the wall clock, so a report is reproducible from
@@ -102,13 +116,49 @@ shipped CI default.
 
 ## Hooks — the protocol, enforced mechanically
 
-Two POSIX-sh hooks seed under `hooks/`: `pre-commit` runs the blocking
-validator before a commit exists, and `reverse-lookup` runs the `--paths`
-lookup over your staged diff, so the knowledge governing the files you touched
+Two POSIX-sh hooks seed under `hooks/`: `pre-commit` runs both whole-store
+validators through `engine/commit-check.js` before a commit exists, and `reverse-lookup` invokes `engine/reverse-staged.js`
+over your staged diff, so the knowledge governing the files you touched
 surfaces without anyone remembering to ask. Each is a thin wrapper — it invokes
 one engine command and exits with its code, unchanged — and neither reads a
 bypass variable, because a hook with an off switch enforces nothing.
 
+The gate reports each check by name. It exits 0 only when both checks pass;
+findings exit 1, and a failed or never-run check exits 2 even if the other check
+is clean or has findings. Both checks always run over the whole store (D-012).
+Reverse lookup is advisory attribution; its results never restrict validation
+or prove that the agent updated the store. It reads NUL-delimited Git records
+and includes additions, modifications, type changes, deletions and both paths
+of detected copies and renames. Detection uses 50% similarity and a fixed
+1000-candidate exhaustive-search limit; above it, Git may report additions and
+deletions instead of a rename/copy relationship. Local Git limits cannot
+change these settings. Each complete path is passed to the resolver
+with `--path=value`, preserving spaces, commas, quotes, tabs and newlines.
+
+Attribution prints a `staged attribution: candidate <tree-id>` section followed
+by resolver JSON, then a `before <tree-id>` section for HEAD when it exists.
+Both sections use the same complete path set; each reports only that snapshot's
+own pointers. A staged pointer repair therefore cannot erase the old path's
+previous governance. Before evidence is historical navigation, not a current
+trust verdict. An empty staged diff produces no output and exits 0, including
+an unborn repository with nothing staged. Git, resolver and snapshot failures
+exit 2; attribution may be incomplete and must not be treated as a clean lookup.
+These lexical checks detect store and vocabulary drift. Application behavior
+remains the application's test suite's responsibility.
+
+The gate validates an isolated copy of the Git index. Source, stores and
+repo-relative rules come from the same candidate, so unstaged repairs cannot
+hide broken staged bytes and unstaged edits cannot introduce findings. It
+never stashes, resets or restages local work. Installed engine code, schemas
+and runtime dependencies stay on the host; untracked evidence is never copied
+into the candidate. Snapshot preparation and cleanup failures block with exit 2.
+Real installed-hook tests check partial staging, committed bytes and cleanup.
+
+Git submodules, symlinks that escape the snapshot, and non-UTF-8 path names
+or symlink targets are refused explicitly when their evidence cannot be represented faithfully. It reads raw blobs without
+checkout filters or archive attributes. Install required runtime dependencies
+before committing; keep whole-store checks on the actual merge candidate in CI.
+
 They **seed but do not install**: `init` never writes `.git/`, so wiring them is
 your act, not the kit's. Git runs a hook only if it is executable, and the copy
 engine seeds bytes rather than modes — so set the bit when you wire it:

package/cli/kit.manifest.yaml

@@ -60,9 +60,8 @@ unconditional:
     # Git hooks that enforce the protocol mechanically (UCS-1157): blocking
     # validation before a commit exists, and automatic reverse lookup over
     # the staged diff. They are THIN WRAPPERS — each invokes one engine
-    # command and exits with its code, unchanged — so the tested surface is
-    # the wrapped command, and the wiring is reviewed the way the per-IDE
-    # wrappers are.
+    # command and exits with its code, unchanged. Real installed-hook Git
+    # tests exercise the pre-commit gate as well as the engine commands.
     #
     # They SEED but do not INSTALL. init never writes .git/ (the client's own
     # config is the client's), so these land at <root>/hooks/ and the client

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "unknown-knowledge",
-  "version": "2.1.0",
+  "version": "3.0.0-rc.1",
   "description": "Free OSS kit that stands up self-improving knowledge-base + ontology structures in any codebase: three governed YAML stores, a deterministic engine, and an agent protocol.",
   "license": "Apache-2.0",
   "repository": {

package/payload/docs/README.md

@@ -27,6 +27,40 @@ paths arrived from the kit and which your own loop produced.
 
 ## Running the gates
 
+### Following replacements from an old result
+
+Resolver JSON leaf projections include `superseded-by`, a stable array that
+is empty when no loaded leaf directly supersedes the match. It appears on
+query leaves, concept knowledge entry points, path knowledge, scope exclusions
+and document gather rows. Each reference has these fields:
+
+| Fields | Meaning |
+|---|---|
+| `id`, `notation`, `heading`, `file` | Accession and navigational metadata; notation is only a legacy display label |
+| `stage`, `time`, `downranked`, `demotions` | The target's normal lifecycle and freshness metadata; `time.verdict` checks freshness only, not trust |
+| `applies` | Declared jurisdictions, sorted; an empty array means universal |
+
+This is an additive output contract under D-021. Existing fields and scores
+retain their meaning. Authors write only `relates.supersedes` on the successor;
+the loader derives the inverse from the store on every load, without reading
+or storing reciprocal edges in `knowledge/derived/`.
+
+References are deduplicated and sorted by accession. They contain no body,
+excerpt, score or nested edges. Follow one hop at a time, keeping visited IDs
+to avoid revisiting a cycle. Multiple successors remain explicit; order never
+selects a winner. The predecessor remains available for historical requests.
+
+Successor scope is declared metadata, not an applicability verdict. Compare
+it with the request's jurisdiction; without a jurisdiction, establish whether
+the target applies before selecting an answer. A scoped-out successor stays
+visible as navigation and does not enter the ranked results merely because
+it supersedes a match. Draft and stale successors keep their normal flags.
+Run `preflight.js --leaves <IDs>` for consulted targets and read their cited
+sources. Existing health gates report dangling references; structural
+`ref-cycle` findings quarantine every leaf in a supersession cycle.
+
+### Validation commands
+
 The engine is plain Node (≥ 22, no build step) with one library
 dependency, `js-yaml`, resolved from your repo like any other package: if
 your repo does not already carry it, run `npm install --save-dev js-yaml`
@@ -48,6 +82,51 @@ proposes draft concepts for anchors the map does not cover yet; a human
 reviews every draft. Run it on the steward cadence, never as a gate
 (`docs/steward-guide.md`).
 
+## Reverse lookup for complete filenames
+
+Use one `--path` per filename to find the concepts and knowledge leaves that
+govern it:
+
+```sh
+node unknown-knowledge/engine/resolve.js --root . --path 'src/a,b.ts' --path 'src/my file.ts' --json
+```
+
+Both `--path value` and `--path=value` work. Use the equals form for a name
+beginning with `--`, for example `--path=--draft.ts`. Each value stays one
+complete path: commas, quotes, tabs, newlines, Unicode, leading/trailing
+whitespace and POSIX backslashes are literal data. Paths still use POSIX
+normalization for `.`, `..`, repeated `/` and absolute paths relative to
+`--root`. Complete normalized paths are deduplicated and sorted.
+
+Programmatic callers should pass an argument array with no shell. This
+example assumes `repoRoot` and `changedPaths` contain the root and filename
+strings already obtained by the caller:
+
+```js
+import { spawnSync } from 'node:child_process';
+import { resolve } from 'node:path';
+
+const result = spawnSync(process.execPath, [
+  resolve(repoRoot, 'unknown-knowledge/engine/resolve.js'),
+  '--root', repoRoot, '--json',
+  ...changedPaths.map((path) => `--path=${path}`),
+], { encoding: 'utf8', shell: false });
+if (result.error) throw result.error;
+if (result.status !== 0) throw new Error(result.stderr || 'Reverse lookup did not complete');
+const attribution = JSON.parse(result.stdout);
+```
+
+Do not join filenames into a shell command or split them on commas or
+newlines. Skip the invocation when there are no changed paths. An empty or
+missing `--path` value, a repo-root-only path, or mixing `--path` with
+`--paths`, query terms or `--doc` fails with a usage message and exit 2.
+An unmatched path is a normal result: exit 0 with empty attribution.
+
+Legacy `--paths a.ts,b.ts` remains supported, including its comma splitting,
+whitespace trimming and backslash conversion. For equivalent ordinary path
+sets, both inputs return the same attribution and deterministic output.
+The repeatable flag is an additive MINOR surface change under D-021.
+
 ## How the loop works
 
 The runtime contract — `RESOLVE → PREFLIGHT → GATHER → ACT → RECORD` — lives
@@ -68,6 +147,62 @@ authoring README seeded beside the packs) — there is no update channel. The
 governed path for teaching the engine a new anchor shape is
 `protocol/new-kind-pipeline.md`.
 
+## Commit gate (opt-in)
+
+The seeded `hooks/pre-commit` invokes `engine/commit-check.js`, which runs both
+whole-store structural and value validation. Each check reports its name and
+status. Exit 0 means both pass; 1 means findings; 2 means a check failed or
+could not run. Failure dominates findings and success. There is no bypass
+variable. Validators inspect source lexically and never execute application
+code; application behavior still needs its own tests.
+
+Install from the repo root when you choose (init never edits `.git/`):
+
+```sh
+chmod +x unknown-knowledge/hooks/pre-commit unknown-knowledge/hooks/reverse-lookup
+ln -s ../../unknown-knowledge/hooks/pre-commit .git/hooks/pre-commit
+ln -s ../../unknown-knowledge/hooks/reverse-lookup .git/hooks/prepare-commit-msg
+```
+
+Use your chosen kit directory name if different; `KIT_DIR` selects that name.
+Preserve existing hooks when integrating this invocation into your setup.
+`hooks/reverse-lookup` invokes `engine/reverse-staged.js --root .` for advisory
+attribution. Git runs it through the event-named `prepare-commit-msg` link;
+putting `reverse-lookup` beside `pre-commit` under `core.hooksPath` alone does
+not run it. Preserve your existing hooks and call it explicitly from an event
+hook when using that configuration.
+
+The engine parses NUL-delimited Git status/path records, including deletions
+and both paths of detected renames and copies. Detection uses 50% similarity
+and a fixed 1000-candidate exhaustive-search limit. Beyond it, Git may emit
+additions/deletions instead of rename/copy relationships; whole-store checks
+remain authoritative. It passes complete filenames
+to the resolver as repeated `--path=value` arguments, with no shell expansion.
+It prints `staged attribution: candidate <tree-id>` followed by resolver JSON,
+then a `before <tree-id>` section when HEAD exists. Both sections attribute the
+same deduplicated path set against their own snapshot. Previous pointers stay
+visible even when the staged store removes or repairs them. Historical results
+are navigation evidence, not current trust verdicts or proof of a store update.
+
+No staged changes means quiet exit 0, including an empty unborn repository.
+A failed Git read, resolver or snapshot operation exits 2, never a zero-hit
+success. Unsupported history can also refuse attribution even when candidate
+validation passes. Attribution never selects a subset for the whole-store
+gate. Real installed-hook commits test this behavior, including pointer repairs,
+partial staging and unusual filenames.
+
+The gate checks an isolated Git index snapshot. Both validators read staged
+source, stores and repo-relative rules; unstaged and untracked evidence cannot
+affect their findings. Local work is never stashed, reset or restaged. Installed
+engine code, schemas and dependencies run from the host and are not linked into
+the evidence snapshot. Missing dependencies, snapshot preparation failures and
+cleanup failures block with exit 2. Git submodules and escaping symlinks are
+unsupported and refused explicitly, as are non-UTF-8 path names or symlink targets. Raw blob reads
+do not invoke checkout
+filters or apply archive attributes. Run from the repository root and install
+the runtime dependencies before committing. Check the actual merge candidate
+in CI as described below.
+
 ## CI
 
 Session-level preflight is a sufficient gate for a small team, not for

package/payload/docs/ci-wiring.md

@@ -33,6 +33,31 @@ never executes your code and never touches the network
 (`docs/boundaries.md`) — so the jobs need no secrets, tokens, or extra
 permissions.
 
+## Commit gate and CI evidence
+
+The opt-in pre-commit hook calls `engine/commit-check.js --root .`. This runs
+both whole-store validators, reports each check, and returns 2 for any failed
+or never-run check, otherwise 1 for findings, otherwise 0. It reads an isolated snapshot of the Git index, including staged source
+files and layout selection. An unstaged repair cannot hide a broken candidate.
+The installed engine and dependencies remain separate runtime inputs.
+
+In CI, check out the actual committed merge candidate and run both validators
+against that checkout, with no generated or repaired governed files between
+checkout and validation. `node unknown-knowledge/engine/commit-check.js --root .`
+also runs both checks even if one fails. Reverse lookup remains advisory;
+changed-path attribution never limits the whole-store correctness gate.
+Application behavior tests remain separate from these lexical checks.
+
+## Existing application store names
+
+If root-level application `knowledge/` or `ontology/` coexists with the seeded
+kit, commit `.unknown-knowledge.json` at the repository root containing
+`{"kitRoot":"unknown-knowledge"}`. Use `{"kitRoot":"."}` to select root-level
+stores instead. These are the two supported layouts. Keep `--root` at the
+repository root: source pointers are repository-relative. Missing selection in
+an ambiguous layout or invalid configuration stops the check. Stage selection
+with the migration; each candidate/before tree uses its own tracked choice.
+
 ## GitHub Actions
 
 ```yaml

package/payload/engine/commands/commit-check.js

@@ -0,0 +1,49 @@
+/** Whole-store commit gate over one isolated proposed Git snapshot. */
+import process from 'node:process';
+import { existsSync } from 'node:fs';
+import { join } from 'node:path';
+import { parseArgs as parseFlags, rethrowIfBug, runCli } from '../lib/cli.js';
+import { EXIT_CODES } from '../lib/exit-codes.js';
+import { withCommitSnapshot } from '../lib/commit-snapshot.js';
+import { locateKitRoot } from '../lib/kit-root.js';
+
+export const USAGE = 'usage: commit-check [--root <repo-root>]';
+
+/** @param {string[]} argv @returns {Promise<number>} */
+export async function main(argv) {
+  const { options } = parseFlags(argv, { value: ['root'] });
+  return withCommitSnapshot(options.root ?? process.cwd(), ({ candidate }) => checkCandidate(candidate.root));
+}
+
+async function checkCandidate(root) {
+  let kitRoot;
+  try {
+    kitRoot = locateKitRoot(root);
+  } catch (error) {
+    rethrowIfBug(error);
+    // Layout refusals describe candidate evidence, not the random directory
+    // used to hold it. Operational cleanup failures still name their artifact.
+    process.stderr.write(`commit-check: snapshot: ${error.message.replaceAll(root, '<candidate>')}\n`);
+    return EXIT_CODES.FAILURE;
+  }
+  if (!['ontology', 'knowledge', 'decisions'].some((store) => existsSync(join(kitRoot, store)))) {
+    throw new Error('snapshot: no governed stores in the candidate; stage the kit before checking a commit');
+  }
+  const args = ['--root', root];
+  let outcome = EXIT_CODES.CLEAN;
+  // Literal imports load only versioned engine code (D-005/D-014). Each check
+  // loads independently, so a broken validator cannot hide the other's result.
+  for (const [name, load] of [
+    ['validate', () => import('./validate.js')],
+    ['validate-values', () => import('./validate-values.js')],
+  ]) {
+    const status = await runCli(name, async (argv) => {
+      const command = await load();
+      return command.main(argv);
+    }, { usage: USAGE, argv: args });
+    process.stderr.write(`commit-check: ${name}: ${status === EXIT_CODES.CLEAN ? 'clean' : status === EXIT_CODES.FINDINGS ? 'findings' : 'failure'} (exit ${status})\n`);
+    // Failure (2) dominates findings (1), which dominate clean (0).
+    outcome = Math.max(outcome, status);
+  }
+  return outcome;
+}