knodin 0.10.8 → 0.12.0

package/docs/releases/0.12.0.md

@@ -0,0 +1,281 @@
+# knodin 0.12.0
+
+Ten open defects worked as one batch, plus three more found while investigating
+them. Nearly all are the same failure in different clothing: a result that is
+*complete* and a result that is *empty* arriving in the same shape, so the reader
+cannot tell which one they got. One is worse than that, and it goes first.
+
+No schema change and no re-index. An earlier draft of this release carried both,
+for an embedding change that measured well on the test embedder and turned out to
+do nothing on the real model. That story is told below, because the way it was
+nearly shipped is more instructive than the fix — and because the same mistake,
+caught a second time, is what led to the retrieval benchmark this release adds.
+
+## `review` answered a different question and said nothing about it
+
+The MCP gateway accepted `auditRange`, `auditBase` and `auditHead` on `review`,
+discarded them, and reviewed `HEAD~1..HEAD` instead. The response reported
+`base: HEAD~1` truthfully — but carried `staleness: fresh`, `availability:
+healthy` and a full freshness envelope, so unless the caller happened to inspect
+`base`, it read as authoritative for the range they had asked about.
+
+Measured against a real branch, the difference was one changed file versus
+twenty. That is not a near-miss; it is a review of the wrong thing, presented
+with the same confidence as a review of the right thing. It was found while
+running a product validation, where the conclusion would otherwise have been
+"review does not catch this defect" — a false negative manufactured entirely out
+of discarded input, and reported as a finding.
+
+Those three parameters belong to `prs`, and only to `prs`; the schema had always
+said so and nothing enforced it. They are now refused on every other operation,
+and the refusal on `review` names the parameter that would have worked
+(`diffScope: "compare"` with `from`/`toRevision`). An unused-parameter error is
+always better than a confident answer to an unasked question.
+
+## `count: 0` could not tell a typo from dead code
+
+On a healthy, fresh graph, `knodin query callers_of someMisspelling` returned
+`count: 0`, `staleness: fresh`, exit 0 — byte-identical to the answer for a real
+symbol that genuinely has no callers.
+
+The uninitialized-graph case was fixed earlier and now fails loudly, which made
+this one worse rather than better: before that fix an empty result at least felt
+untrustworthy, and afterwards it arrived with a certificate. The freshness stamp
+certifies the **graph**; it says nothing about whether the **target** resolved.
+Those are separate axes and are now reported separately. A query for a symbol
+that exists in no open repository answers `targetResolution: "not-found"`,
+renders as `target-not-found` rather than `no-match`, and exits non-zero, since
+a misspelling is operator error rather than a negative result. A real symbol
+with no callers still answers `resolved` and exits 0 — that is the honest
+negative, and it is now distinguishable from the typo.
+
+This matters most for the symbol it is easiest to lose: a seam is *by
+definition* a well-documented symbol with no callers yet, so `count: 0` was
+destroying exactly the signal that would have said "use this one". One session
+nearly hand-rolled a duplicate of a blessed seam for that reason.
+
+## `callers_of` counted callers and read as call sites
+
+`callers_of` answers "which symbols call X", and its output — `count: N` above
+one line per row — reads as "N call sites, here they are". When a function calls
+the target twice, only the first line appeared and the count was lower than the
+number of call sites. Ground truth of four calls reported as three.
+
+Anyone sizing an edit from that count under-sizes it. Worse, the reporting
+session initially scored a correct answer as a bug, because cross-checking with
+grep found a call the tool had "missed" — it had not missed it, it had merged
+it. Each row now carries `callSiteLines` with every line for that caller, and
+the result carries `callSiteCount` alongside `count`, which is documented as
+counting rows. Renaming the field alone would not have been enough: it would
+have left the reader doing arithmetic they had no data for.
+
+## A team repo with dead hooks had no safe way out
+
+Every exit mutated something. Plain `repair` fixed graph rows and left the
+background indexer dead, so the schema drifted again. `init` rewrote tracked
+`AGENTS.md`, `GEMINI.md` and `.mcp.json` — churn every teammate diffs, on a
+branch that has nothing to do with it. `configure --scope personal` stripped the
+shared block for everyone else. The one clean option, `init --scope personal`,
+is precisely the one refused while tracked team integration is active.
+
+`knodin repair --lifecycle` rewrites the managed Git hooks and the background
+refresh script and touches nothing else — no indexing, no graph rows, and it
+never enters the agent-configuration path at any scope. Fault and remedy now
+match in scope: the fault is in the hooks' contents, so the hooks' contents are
+what changes. Every remediation message that used to say "run `knodin init`" for
+a lifecycle fault now points here instead.
+
+This does not settle the separate question of whether a per-developer scope
+preference should mutate team-shared tracked files. It removes the trap.
+
+## Hooks that predate the interpreter fallback are now named before they break
+
+Scripts generated before the `KNODIN_NODE` fallback pin an absolute Homebrew
+Cellar interpreter with nothing to fall back to. Homebrew deletes that directory
+on upgrade, so every background refresh afterwards dies with exit 127 — in the
+background, where the only trace is a log file — and the graph drifts until
+someone happens to run `status`. One `brew upgrade` did this to three
+repositories on one machine in a day.
+
+The existing check could not see these at all: it keys off a `KNODIN_NODE=`
+line, and these scripts have none, so the most dangerous shape read as healthy.
+Health now reports a script that predates the fallback **while its interpreter
+still exists**, because the whole point is that it is a time bomb rather than an
+already-detonated one. A script it cannot recognise is still left alone; an
+unparseable hook is not evidence of a pinned interpreter, and guessing would
+invent a failure rather than report one.
+
+## Search ranks a depended-upon implementation above its own fresh duplicate
+
+Semantic similarity alone ranked a minutes-old duplicate above the
+implementation it duplicated. The duplicate is short and made almost entirely of
+the query's own words; the canonical version is diluted by an actual body. The
+searcher read their own new leaf as evidence that nothing existed.
+
+Degree is now weighted into rank fusion, capped at about three rank positions.
+It is deliberately a small weight, because the opposite failure is equally real
+and lands on the *same symbols*: a seam is a documented symbol with no callers,
+and a large centrality weight would bury seams to fix duplicates — trading one
+silent miss for another. A connected implementation climbs past neighbours it
+was scoring near; it does not win from anywhere. Symbols with no inbound
+references are excluded from the centrality ranking rather than ranked last, so
+an uncalled symbol is never penalised, it only forgoes a boost it has no
+evidence for.
+
+An accompanying change — bounding the source body inside the embedding budget —
+was written, measured, justified in a comment, given a schema bump, and then
+**reverted before release**. On the deterministic bag-of-words embedder the suite
+runs, it raised the diluted implementation's similarity from 0.163 to 0.212. On
+the real MiniLM model it does nothing: across two body shapes, three query
+phrasings and both budgets, the duplicate wins all twelve times, and the cap
+never moves the winner. Its apparent benefit came from the test fixture, which
+emitted forty copies of one line — truncating identical lines removes a
+redundancy bag-of-words punishes and a transformer largely ignores.
+
+The general lesson is worth more than the change would have been: a short
+document made almost entirely of query tokens beats a long one under cosine
+similarity, and no reweighting of a single vector's inputs changes that. Graph
+degree is the right remedy here precisely because it is not a similarity signal.
+
+## Prose queries got no lexical signal at all, and now do
+
+A query phrased as prose did not surface the symbol whose docstring restates it;
+symbols sharing a word or two in their *identifier* won instead. The ticket
+framed this as docstring prose being under-weighted against code in the
+embedding. That premise turned out to be false — for the reported symbol the
+embedded text is already about 95% docstring, and it still lost.
+
+The actual cause was in rank fusion. The lexical channel selected the AND-matched
+set unless every semantic score was bit-identical, which never happens under a
+real model, so the OR set was computed on every query and discarded on every
+query. And for a query phrased as prose the AND clause is empty by construction:
+it demands one symbol containing every surviving query word, function words
+included. Nothing matches, so the lexical contribution was zero for every
+candidate and "hybrid search" was pure vector search. The docstring's BM25
+evidence existed the whole time, in the set being thrown away.
+
+Both operators are now fused as separate channels, the partial one at 0.75 weight
+and truncated to its top 30. Because the AND set is a subset of the OR set by
+construction, an exact match collects both terms and a partial match collects
+one — so the precision bonus falls out of that containment rather than needing
+machinery of its own. The weight buys a property worth stating: lexical evidence
+can rescue a symbol the embedder under-ranked, and cannot resurrect one the
+embedder judged irrelevant.
+
+A partial match must also clear a quorum: it has to echo at least two of the
+query's words. A single incidental token is not evidence, and this index cannot
+tell the difference unaided — FTS5 here has no stemming and does not split
+identifiers, so for "session authentication and token validations" the class
+that does the work matches *nothing* (its docstring says "sessions" and
+"validation") while a handler containing "session" matches once. Without the
+quorum that handler outranked the authenticator, and no weight fixed it: every
+value from 0.75 down to 0.15 flipped the same way, because the problem was which
+symbol earned credit rather than how much.
+
+Measured on the new benchmark below: **MRR 0.5178 → 0.754, recall@10 0.80 →
+0.925**, with 19 of 40 queries improving, 4 regressing and 17 unchanged — a
+two-sided exact sign test at p = 0.0026. Dev and test halves moved together,
+which is what you want when nothing was tuned against the labels. The largest
+gains were symbols that had been effectively unreachable: rank 24 → 1, 28 → 3,
+14 → 2.
+
+## Doc comments were being lost, and misattributed
+
+Two defects in how a doc comment is matched to the thing it documents. Both fed
+the embedding input and the FTS `summary` column, so both corrupted retrieval
+directly — and they point in opposite directions, which is why neither showed up
+as an obvious symptom.
+
+**Multi-line `//` comments kept only their last line.** tree-sitter gives each
+`//` line its own node and the extractor read a single previous sibling, so the
+loop meant to rejoin lines could never fire for this style. The first line of a
+doc comment is usually the sentence saying what the thing is for — exactly what a
+capability question matches on — so this silently removed the most useful text
+and left a grammatical fragment that still looked like a well-formed summary.
+Block comments are one node and were never affected, which is why the damage
+stayed invisible in codebases that prefer them.
+
+**And a symbol with no doc comment adopted its neighbour's.** The sibling walk
+stepped over as many as five non-comment nodes hunting for a comment, so prose
+written about one symbol was embedded against another. Measured on this
+repository before the fix: **857 of 1,329 documented symbols under `src/` — 64% —
+shared their summary text with another symbol**, and spot checks showed plain
+misattribution rather than genuine repetition, including an interface carrying a
+neighbouring function's docstring. After the fix, 718 symbols are documented and
+**none** share text.
+
+That second one is the worse of the pair. Absent prose merely loses a hit;
+misattributed prose manufactures a confident wrong one.
+
+A related gap is recorded but not fixed: a decorated declaration still never
+receives its doc comment, because the decorator is a sibling of the declaration
+but a child of the exporting statement. It was verified to predate both fixes and
+is pinned by a characterization test.
+
+## A benchmark for retrieval quality, and why it could not be a test
+
+`npm run bench:c103` measures whether a capability question phrased as prose
+finds the symbol implementing it, through the real ranker and the real embedding
+model, over knodin's own source at a pinned commit. Forty hand-written queries,
+15 dev / 25 test, MRR primary.
+
+It exists as an opt-in benchmark rather than a spec because the fast suite
+installs a deterministic bag-of-words embedder, and that embedder produces the
+**opposite** answer on exactly this case: an identifier match beats a docstring
+match on it, while the docstring wins by 2.75x on the real model. Two ranking
+assertions written for this release were deleted for pinning that inversion. A
+test that asserts the reverse of production is worse than no test — it fails
+changes that help users and passes changes that hurt them.
+
+So the fast suite now asserts only ranking properties no choice of embedder can
+change, which in practice means rank arithmetic over graph structure, and
+retrieval quality is measured where it can be measured honestly. The existing
+`c86` evaluation could not host this: it ranks raw text by exact cosine, never
+touching fusion or the ranking weights, and is saturated at recall@5 = 1.0.
+
+## What is not fixed
+
+**Mean-pooling dilution inside long docstrings.** Only one sentence of a
+multi-sentence docstring may be on topic, and averaged over the whole comment it
+becomes a minority of the vector while a short, entirely-on-topic competitor does
+not. Splitting docstrings into sentence chunks and max-pooling was prototyped
+against the real model and moves the reported symbol from rank 4 of 5 to rank 2 —
+but it needs roughly 1.5x the vectors, a composite primary key, and a re-embed
+migration.
+
+It is deliberately **not** in this release. The lexical fix above addressed the
+same failure for no migration at all, and the benchmark now says prose retrieval
+sits at MRR 0.754 with recall@10 of 0.95. Chunking should be re-measured against
+that new baseline before anyone pays for it — which is the whole reason the
+benchmark was built first.
+
+Worth stating plainly: the benchmark corpus is knodin's own source, and the
+defect was reported against a different codebase. The improvement is real and
+significant here; it has not been demonstrated there.
+
+## Smaller things
+
+`repair` now seeds from an indexed sibling worktree when there is **no** database
+at all, reflink-copying a baseline and reconciling only the differing paths
+instead of rebuilding the corpus from scratch. A database that exists but is
+damaged is deliberately excluded — that is the case `repair` exists for, and
+seeding it would replace a verifiable local rebuild with another checkout's
+state.
+
+`readGitHeadFast` is deleted. Its caller invoked it only when `.git` was absent,
+which was its own first bail-out, so the two conditions were exactly
+anti-correlated and the function could only ever return `null` — making the
+`&& headUnchanged` term it fed inert. Coverage confirmed zero hits across the
+whole suite. Git history settled which side was the bug: the fast HEAD read
+landed first, and the later `lease = 0` for git-backed repositories superseded
+it.
+
+Two timing-sensitive specs no longer produce false pre-push failures. Both had
+been widened only for instrumented coverage runs, on the theory that
+instrumentation caused the contention. It does not — the failures reproduce on
+ordinary uninstrumented runs under host load — and the uninstrumented run is
+exactly the one developers gate on. The budgets are now unconditional, matching
+the correction already recorded in `SUBPROCESS_BUDGET_MS`. A timeout that fires
+on a loaded machine reports a failure that did not happen, and a gate whose
+failures have to be re-run in isolation before they can be believed is not
+doing its job.

package/package.json

@@ -1,8 +1,8 @@
 {
 	"name": "knodin",
-	"version": "0.10.8",
+	"version": "0.12.0",
 	"knodin": {
-		"compatibility": "compatible"
+		"compatibility": "breaking"
 	},
 	"description": "knodin — source-evidenced local code intelligence with known bounds. Stable identity, fresh evidence, truthful budgets, and recoverable bounded views.",
 	"license": "MIT",
@@ -63,6 +63,8 @@
 		"docs/releases/0.10.6.md",
 		"docs/releases/0.10.7.md",
 		"docs/releases/0.10.8.md",
+		"docs/releases/0.11.0.md",
+		"docs/releases/0.12.0.md",
 		"docs/releases/0.3.0.md",
 		"docs/releases/0.4.0.md",
 		"docs/releases/0.4.1.md",
@@ -121,6 +123,7 @@
 		"check:hygiene": "node scripts/check-repository-hygiene.mjs",
 		"check:roadmap": "tsx scripts/check-roadmap.ts",
 		"verify:c79": "tsx scripts/verify-c79.ts",
+		"bench:c103": "tsx benchmarks/evaluations/c103-prose-retrieval/runner.ts",
 		"bench:c80": "tsx benchmarks/evaluations/c80-structural-routing/runner.ts",
 		"verify:c80": "tsx scripts/verify-c80.ts",
 		"bench:c81": "tsx benchmarks/evaluations/c81-powered-benchmark/runner.ts",