@agentskit/doc-bridge 1.7.45 → 1.9.0

package/docs/spec/graph-signals-v1.md

@@ -0,0 +1,92 @@
+---
+title: Graph signals v1
+description: What Doc Bridge computes from the repository graph, what each signal means, and what it deliberately does not claim.
+---
+
+# Graph signals v1
+
+The knowledge snapshot is a graph, and some questions are only answerable as graph questions: which
+document is the entry point, which module every change reaches through, which directories move
+together, where the imports run in a circle.
+
+These are computed with [graphology](https://graphology.github.io) from the snapshot, on demand.
+The graph is a working structure: it is never serialised, `DiscoverySnapshotV1` does not change, and
+graphology's own format never reaches disk. Nodes and edges are inserted in sorted order and every
+score is rounded before it is returned, so two runs over the same snapshot agree exactly — including
+after the input order is shuffled.
+
+## The signals
+
+| Signal | Computed from | Over |
+| --- | --- | --- |
+| **Canonicality** | PageRank | `links-to` and `covers` |
+| **Centrality** | betweenness, normalized | `imports` and `re-exports` |
+| **Proximity** | bounded shortest path | imports, documentation, `mentions`, `depends-on` |
+| **Import cycles** | depth-first search | `imports` and `re-exports` |
+| **Area suggestions** | seeded Louvain communities | `imports` and `re-exports` |
+
+External and unresolved endpoints are left out unless a caller asks for them: a dependency on a
+third-party package is a fact about the repository, not a part of its architecture.
+
+**Canonicality** answers where a reader should start. A page many documents link to, or that covers
+many entities, outranks a leaf note nothing points at — however recently the leaf was edited.
+
+**Centrality** answers how much of the dependency structure runs through one module. It is a review
+signal: a change there reaches further than its diff suggests. It is *not* a statement about runtime
+availability, and the diagnostic says so, because "single point of failure" is a claim about
+deployment that a static import graph cannot make.
+
+**Proximity** answers how many hops apart two entities are, bounded — three by default. Unbounded
+proximity is not useful: at ten hops everything is related to everything. `contains` is excluded on
+purpose. It is hierarchy, and including it puts every module in an area two hops from every other
+one, which is true and tells a reader nothing.
+
+**Import cycles** are reported with every edge that forms them. `graphology-dag` answers whether a
+cycle exists; reporting one needs the path, because a diagnostic whose loop a reader cannot trace is
+a claim rather than a finding. The search is bounded in count, and the cycle is rotated so the
+lexicographically smallest node comes first — the same cycle found from two different entry points
+is one finding.
+
+## Area suggestions are suggestions
+
+A Louvain community is a hypothesis: these modules move together, so perhaps they are one unit. It
+never becomes an [area](./config-v1.md#analysisareas-optional) on its own. An area is derived from
+the repository's own structure or declared by a human, and a clustering algorithm is neither.
+
+Suggestions travel as `coverage` entries with `analyzer: graph`, `scope: area-suggestion:<path>` and
+status `not-analyzed` — the honest status, because the clustering ran but whether the cluster is an
+area is a question nobody has answered. A directory that is already an area produces no suggestion.
+Community detection draws from a seeded generator, so a suggestion is reproducible rather than a
+different guess each run.
+
+## Reading the graph as memory
+
+```ts
+import { createDocBridgeGraphMemory } from '@agentskit/doc-bridge'
+
+const graph = createDocBridgeGraphMemory(snapshot)
+await graph.getNode('module:src/mcp/server.ts')
+await graph.findEdges({ label: 'covers', to: 'area:src/mcp' })
+await graph.neighbors('module:src/mcp/server.ts', { depth: 2 })
+```
+
+This satisfies `GraphMemory` from `@agentskit/memory`, so an agent built on AgentsKit walks
+repository structure with the same three calls it uses for its own memory. A second argument layers
+extra entities and relations over the observation, in the snapshot's own vocabulary, so an approved
+enrichment overlay plugs in unchanged.
+
+Writes are accepted and kept in process. The snapshot is an observation and a caller cannot edit it,
+so `upsertNode` lands in a working layer above the projection and `deleteNode` masks rather than
+erases — an agent can annotate what it is exploring without any of it being mistaken for something
+the repository said. `clear()` drops the working layer and leaves the projection intact.
+
+## What consumes them
+
+`centrality-risk` in the rules engine is betweenness now. It used to count how many
+undocumented-relation findings were attached to an entity, which measures documentation debt and
+calls it architecture: a module every import path runs through scored zero if it happened to be
+documented. Without a graph the rule reports nothing at all, which is better than reporting the
+wrong thing under a name people act on.
+
+The rule's threshold reads as a rank when it is 1 or more — `3` means "flag the three most central
+entities" — and as a minimum betweenness when it is below 1.

package/docs/spec/incremental-scan-v1.md

@@ -0,0 +1,102 @@
+---
+title: Incremental scan v1
+description: How Doc Bridge hashes each file-backed entity, when a second scan may reuse one, and what it refuses to reuse.
+---
+
+# Incremental scan v1
+
+Every `module`, `document` and `package` entity carries the hash of its file in its first evidence
+item. An `external` entity carries none: it is a name in a manifest, not a file on disk.
+
+```json
+{
+  "id": "module:src/query/search.ts",
+  "kind": "module",
+  "evidence": [{ "source": "code", "path": "src/query/search.ts", "contentHash": "…" }]
+}
+```
+
+`EvidenceSchema.contentHash` had existed since the first schema and discovery never filled it, so
+every cache and every overlay could be keyed only on "the whole repository changed" — which is true
+between any two commits and therefore useless. With a hash per file, a consumer can expire one
+entry, and a second scan can skip the expensive part: the TypeScript parse and the Markdown parse,
+where nearly all of discovery's time goes.
+
+A document's hash is taken after a leading byte-order mark is stripped. A mark is not content: a
+file that only gained one parses to the same tree, and should not invalidate anything.
+
+## Reuse
+
+```ts
+const cold = discoverRepository({ root, config })
+const fast = discoverRepository({ root, config, previous: cold })
+```
+
+`previous` is an offer, not an instruction. Discovery reuses an entity only when reuse cannot change
+the answer, and two different things can change it:
+
+- **An entity's own fields depend on its own bytes.** A hash match is enough.
+- **A relation depends on what else exists.** A module importing `./new.js` resolved to nothing
+  before that file was added and resolves to a module after; a document mentioning `rank` points at
+  whichever module declares it. So relation reuse also requires that the universe the references
+  resolve against is identical.
+
+Two fingerprints capture that universe, both derived from the previous snapshot rather than stored
+in it — everything they cover is in the snapshot already, and a stored fingerprint is one more
+thing that can be stale or forged.
+
+| Fingerprint | Covers | Gates |
+| --- | --- | --- |
+| Module universe | module paths, packages, compiler options | reuse of a module's relations |
+| Resolution | the module universe, plus document paths, area paths and which module declares each exported symbol | reuse of a document |
+
+A reused entity's relations are replayed against the entity set the scan is producing. An edge
+whose internal target is gone is dropped rather than carried: the file it pointed at was renamed or
+deleted, and a graph that keeps the edge is lying about the repository. An external or unresolved
+endpoint is re-added instead, because such an entity is in the snapshot only because something
+referenced it, and the thing that referenced it is exactly what was reused.
+
+## What is refused outright
+
+A hash says a file has not changed. It says nothing about whether this code would still read it the
+same way — an analyzer that learns to record a document's headings produces different entities from
+identical bytes, and a configuration change moves area boundaries and runtime-wiring detection. So
+the whole snapshot is refused unless it was produced by this `pipelineVersion`, these
+`analyzerVersions` and this `configurationHash`, and a snapshot that does not declare all three is
+refused as well. Trusting an undeclared input with a repository scan is how a cache becomes a
+source of wrong answers.
+
+A cache that is only usually right is worse than no cache. Reuse either produces the snapshot a
+cold scan would produce, or it does not happen.
+
+A reused entity also replays the per-file `coverage` its analyzer produced, because the aggregate
+entries are derived from those rather than stored. An aggregate that cannot be rebuilt from what
+the snapshot carries is an aggregate a fast scan gets wrong: a fact that lives only in a local
+variable during a parse is a fact the next scan cannot replay. `dynamic-imports:<path>` and
+`runtime-wiring:<path>` therefore record every observed load and wiring call — `complete` when the
+target is statically known, `not-analyzed` when it is not.
+
+## The run explains itself
+
+A run that finishes in a tenth of the time has to be able to say why, or nobody can tell a working
+cache from a broken scan. One `coverage` entry reports it:
+
+```json
+{
+  "analyzer": "repository",
+  "scope": "reused-entities",
+  "status": "complete",
+  "reason": "Reused 102 entities and skipped 102 of 102 parse(s). Nothing needed re-parsing."
+}
+```
+
+`status` is `complete` when everything reusable was reused, `partial` when reuse was refused — the
+reason then names what changed — and `not-applicable` when there was no previous snapshot to reuse.
+
+This entry is the one part of a snapshot that describes the *run* rather than the repository. The
+entities, the relations and every other coverage entry are byte-identical to a cold scan's, which
+is what the tests assert; the snapshot's own `contentHash` covers this entry too, so a warm scan
+and a cold scan of the same tree hash differently. That is why the CLI does not yet pass a previous
+snapshot: the artifacts it writes are compared across runs, and the caching layer has to decide
+what a report keys on before a fast scan starts feeding it. `contentHash` and `sourceRevision`
+semantics are otherwise untouched.

package/docs/spec/markdown-analyzer-v1.md

@@ -0,0 +1,73 @@
+---
+title: Markdown analyzer v1
+description: What Doc Bridge reads from a Markdown document, and the relations it observes from prose.
+---
+
+# Markdown analyzer v1
+
+Documentation is parsed with remark (CommonMark plus GFM), not with regular expressions, so a
+document's own prose becomes evidence. Every relation below is `observed` and carries the file and
+line the claim was made on.
+
+| Relation | From → to | Observed from |
+| --- | --- | --- |
+| `links-to` | document → document | a relative link that resolves to a scanned document |
+| `mentions` | document → module or package | inline code or link text equal to a scanned path or a package name |
+| `mentions-symbol` | document → module | inline code equal to an exported name of exactly one module |
+| `covers` | document → anything | a `docbridge` declaration, unchanged |
+
+A symbol resolves to the module that **declares** it, never to a barrel that re-exports it. When
+two modules declare the same name the reference resolves to neither: the tokens and their lines
+are reported as a `markdown` coverage note instead, because sending an agent to one of two
+possible definitions is worse than sending it nowhere. The same rule governs near-misses — an
+unresolved path-shaped reference is matched with Jaro-Winkler and accepted only at 0.92 or above
+*and* with a single candidate, recorded as `metadata.confidence: "fuzzy"`.
+
+Document entities gain what the parser can see: `title`, headings to depth three with their
+lines, a bounded `summary`, `wordCount`, the frontmatter subset (`type`, `audience`, `owner`,
+`lifecycle`, `tier`), any generated regions, and the file's `contentHash` on its evidence. A
+document declaring `audience` overrides the path heuristic that classifies it; `type` overrides it
+only when it names an audience, since in practice `type` names a document kind.
+
+A document referencing more than 64 entities records `evidenceTruncated` and a coverage note. An
+index page's sixty-fifth link is not knowledge, and an unbounded list is not evidence.
+
+## Generated regions
+
+```markdown
+<!-- doc-bridge:generated hash=8f79947 -->
+…generator output…
+<!-- /doc-bridge:generated -->
+```
+
+Mentions inside a generated region are ignored, so Doc Bridge never reads its own output back in
+as evidence about the repository. An unclosed marker owns the rest of the file. The regions are
+recorded on the document entity, which is what lets the audit report a manual edit inside one.
+
+## The `docbridge` block
+
+The block is real YAML validated by a schema, so quoted lists, flow mappings, anchors and
+multi-line strings work as they do in every other tool. Schema violations report the field:
+`docbridge.relations.0: Unrecognized key: note`. Each `DOCBRIDGE_*` code is preserved — a
+repository failing its build on one keeps failing on the same one — and a block YAML cannot read
+at all falls back to the line-oriented scanner, because on a mangled block a diagnostic per line
+helps the author more than a single parser error.
+
+## The cost of a near-miss
+
+Near-miss resolution is the analyzer's only super-linear step: every unresolved path-shaped
+reference is a query against every document, module and area path in the repository. On a monorepo
+of 4 100 documents and 9 240 modules the naive form of that — rebuilding the candidate list per
+document, then computing Jaro-Winkler against each candidate — did not finish.
+
+Two bounds make it cheap, and both are bounds on Jaro's match count `m` rather than heuristics.
+`m` cannot exceed the shorter string, so `jaro ≤ (m/|a| + m/|b| + 1) / 3`; with the prefix bonus
+bounded by `jw ≤ 0.6·jaro + 0.4`, a threshold of 0.92 admits only candidates whose length is
+within 0.6× to 1.67× the query's. And `m` cannot exceed the number of characters the two strings
+share, which one pass over a 128-slot count vector answers, where a similarity computation costs a
+pass over one string per character of the other.
+
+Both only ever over-estimate `m`, so a candidate they skip could not have matched: the index is a
+speed-up with no effect on results, tied scores included. `MarkdownResolution.pathIndex` carries it,
+built once per run; an analyzer called without one builds its own and produces the same output more
+slowly.

package/docs/spec/mcp-knowledge-tools-v1.md

@@ -0,0 +1,147 @@
+---
+title: MCP knowledge tools v1
+description: One search and one lookup that answer inside a declared token budget, findings in the shape the ecosystem consumes, and a doctor that measures what retrieval can see.
+---
+
+# MCP knowledge tools v1
+
+An agent about to edit `src/mcp` used to need four calls — a search, a handoff, the diagnostics,
+the relations — and got back payloads whose size it could only estimate as `contextBytes / 4`.
+Two tools now answer that question in one bounded response, the existing tools stay as they are,
+and the doctor stops reporting a health it did not measure.
+
+## `knowledge.search`
+
+```json
+{ "query": "where do I add a new MCP tool", "kinds": ["document", "area"], "limit": 5, "explain": true, "budgetTokens": 600 }
+```
+
+Ranks the retrieval projection with the same `searchIndex` the CLI uses, so with no `kinds` the
+`results` are what `ak-docs search <query> --json` prints for the same query and index — a test
+compares the two rather than assuming it. Each result adds the entry's `title` and an `excerpt`,
+the opening of the projected body. `kinds` filters after ranking, from a deeper ranking, so the
+order never changes and a filtered call still returns up to `limit`. `explain` attaches the
+matched terms and every score component, as on the CLI.
+
+## `knowledge.lookup`
+
+```json
+{ "id": "area:src/mcp", "depth": 1, "budgetTokens": 900 }
+```
+
+Accepts an entity id, an ownership id, an alias or a `path`, through the same resolution
+`handoff.resolve` uses. One response carries:
+
+| Section | What it is |
+| --- | --- |
+| `entity` | id, kind, path, title, summary, aliases, symbols, tags, provenance, confidence, content hash, ownership, area, package, PageRank |
+| `neighbours` | every entry within `depth` hops, grouped by relation kind (`covers`, `mentions`, `links-to`, `imports`, `contains`, …), each with direction, confidence, distance and the entry it was reached through |
+| `documents` | the documents that `cover` the entity and those that `mention` it, most canonical first |
+| `handoff` | exactly what `handoff.resolve` returns for the entity, `related` included |
+| `diagnostics` | the open diagnostics of the latest reconciliation report that name the entity or point at its file, with the report hash — or `reportHash: null` when no workflow run exists |
+| `evidence` | the entity's path and content hash, then those of the documents about it, each with an `excerpt` |
+
+`contains` is not a projection edge — it is hierarchy — but an agent asking about an area wants its
+modules listed, so the lookup synthesises it from `areaId` and `packageId`. Neighbours are
+bounded per relation kind and visited in sorted order, so two lookups over the same projection
+produce the same response. `depth` is at most 3.
+
+`format: "text"` on either tool renders the same payload as prose through
+`formatRetrievedDocuments`, for clients that prefer it.
+
+## Budgets
+
+When `budgetTokens` is present, both tools and `handoff.resolve` trim through `compileBudget`:
+each droppable section becomes a message, oldest first in the order it may be dropped, the payload
+with every section removed becomes the last message, and `drop-oldest` with `keepRecent: 1` sheds
+sections from the front until the rest fits. The declared order is
+
+```
+evidenceExcerpts → related → neighbours → summaries
+```
+
+and nothing else is ever dropped: the entity, the evidence paths and hashes, the handoff fields an
+agent acts on and the open diagnostics survive every budget. A payload whose undroppable core
+still exceeds the budget reports `fits: false` with every section dropped, rather than truncating
+what it may not drop. A section a payload does not have — search has no `related`, a handoff has
+no `neighbours` — is absent from the report, never reported dropped.
+
+```json
+{
+  "budget": {
+    "budgetTokens": 900,
+    "tokens": { "total": 790, "budget": 900, "core": 684, "sections": { "evidenceExcerpts": 59, "related": 290, "neighbours": 229, "summaries": 106 } },
+    "fits": true,
+    "order": ["evidenceExcerpts", "related", "neighbours", "summaries"],
+    "kept": ["summaries"],
+    "dropped": ["evidenceExcerpts", "related", "neighbours"],
+    "tokenMethod": "approximate"
+  }
+}
+```
+
+Tokens are counted with `approximateCounter` — four characters per token plus two per message,
+over the serialised sections — and reported as `tokenMethod: "approximate"`. `compileBudget` and
+`approximateCounter` are mirrored in `src/budget/compile.ts` because `@agentskit/core` is an
+optional peer and every query surface answers with no peer installed; a test runs the real
+`compileBudget` over the same messages and asserts the two agree on every token count, every
+dropped message and `fits`. On a handoff the droppable sections are `related` and the note that
+repeats the target's summary; `budget` is an optional field of `AgentHandoffV1`, so a budgeted
+handoff is still a valid handoff.
+
+## Every existing tool keeps working
+
+`handoff.resolve`, `doc.search`, `doc.get`, `gate.status`, `retriever.query`, `memory.*`,
+`registry.topology` and `docbridge.*` keep their names, arguments and payloads. `handoff.resolve`
+gains an optional `budgetTokens`; `docbridge.diagnostics` gains an optional `format`. The new tools
+are appended to the advertised list and to the configuration's `surfaces.mcp.tools` default, so a
+configuration that names its tools explicitly is unchanged until it names them.
+
+## Canonical findings
+
+```
+ak-docs check --json --format finding
+docbridge.diagnostics { "format": "finding" }
+```
+
+Both emit every reconciliation diagnostic as a `Finding` from `@agentskit/core/finding`, with
+severities drawn from `SEVERITY_ORDER`, so Code Review, AKOS and dashboards read Doc Bridge with no
+parser of their own. Internal severities map `error → high`, `warn → medium`, `info → low`,
+`off → info`; nothing is `critical`, because a documentation finding never takes a system down.
+`title` is the code as words, `detail` the message, `category` the status, `location` the first
+evidence path and line, `ref` the code, and `metadata` carries the internal code, status,
+severity, evidence and entity ids, so nothing is lost. `confidence` follows the status: an
+observed relation is certain, a stale declaration less so, a coverage gap least of all. Findings
+are ordered most severe first and by id within a severity.
+
+This is a reporter, not a migration: `KnowledgeDiagnostic`, `RuleFinding` and
+`DocumentationAuditFinding` keep their shapes, and the rule verdict still decides the exit code
+of `check`. A test imports the real package and asserts assignability and the severity order.
+
+## The measured doctor
+
+Three dimensions join the score and can lower the grade:
+
+| Dimension | Measures | Points |
+| --- | --- | --- |
+| Reachability | the share of the snapshot's document entities present in the retrieval projection | 15 |
+| Connectivity | the mean of: areas with at least one covering or mentioning document; documents with at least one edge into code | 15 |
+| Benchmark | hit@3 over the golden suite at `retrieval.benchmark.suite` (`docs/bench/retrieval-suite-v1.json` by default) | 10 |
+
+The existing dimensions — index present and fresh, agent docs, human guides, gates — make up the
+other sixty. A repository with no golden suite reports the benchmark as `not-analyzed`, scores
+nothing for it and says so in the issues; it is never silently omitted.
+
+An A requires all three: reachability at 100 percent, connectivity at 80 percent or more, and a
+measured hit@3 of 80 percent or more. A score of 90 that misses one of them is a B, and the text
+report says which. Reachability must be complete because a document retrieval cannot find is a
+document the product does not deliver; the other two have a floor rather than a ceiling.
+
+On this repository every document is in the projection and the benchmark is at 88.3 percent, but
+20 of 39 areas have no document about them and 55 of 100 documents do not point at code —
+connectivity 47 percent, grade B. That is the honest state. A test proves the other direction on
+the same repository: one document out of the projection is enough to lose the A, and an index
+with no projection at all — what the builder produced before the corpus projection — is at zero.
+
+The doctor's `ok` is unchanged: it is still "no error-severity issue and the gates pass", so a
+CI step that runs `ak-docs doctor --text` fails on a stale index, not on a B.

package/docs/spec/measured-enrichment-v1.md

@@ -0,0 +1,229 @@
+---
+title: Measured enrichment v1
+description: What an enrichment run cost, what it invented, whether it improved retrieval at all, and the study measurements — mechanical task expectations, the assisted arm, tokens to first evidence — that decide the same questions for a controlled round.
+---
+
+# Measured enrichment v1
+
+An enrichment stage nobody measures is a stage nobody can defend.
+
+[Enrichment overlay v1](./enrichment-overlay-v1.md) made agent proposals typed, validated and
+bounded. It did not say whether any of it helped. The overlay reported what it accepted, which is
+the one number an agent cannot fail: a curator that proposes a hundred things and has ninety
+rejected looks exactly like one that proposes ten good ones. And the controlled study could not
+answer the question either — its last round recorded zero semantic successes in both arms, because
+a task whose only success criterion is a model's opinion produces no signal, and its third arm has
+been reserved since the first suite without ever running.
+
+This document is the measurement layer: the whole shape of an enrichment run, the retrieval delta
+that can block it, and the study machinery that asks the same questions of a controlled round.
+
+## What the run reports
+
+`EnrichmentStats` in `src/schemas/enrichment.ts` is written by `runEnrichment` and stored on the
+overlay. It is deliberately outside the overlay's content hash — two runs over one unchanged
+repository must agree on their decisions while disagreeing about how long they took.
+
+| Field | Meaning |
+| --- | --- |
+| `byKind` | Per proposal kind: `proposed`, `accepted`, `pending`, `rejected`. Sorted by kind, so two runs produce the same bytes. |
+| `rejectionReasons` | A histogram over the closed list of rejection reasons, sorted by reason. A run's rejections are comparable across runs rather than a log to read. |
+| `inventedReferences` | Rejections that named something the repository does not contain. |
+| `agentRuns`, `inputBytes`, `outputBytes` | What was actually sent and received. Batched packs, so a run is not a call count. |
+| `cacheHits`, `cacheHitRate` | Hits, and hits over hits plus runs, rounded to six places. |
+| `wallTimeMs` | Measured from a monotonic clock, not derived. `runEnrichment` takes a `clock` option so a test can pin it. |
+| `expired` | Entries whose target content hash moved since they were accepted. |
+
+`enrichmentCost(stats)` in `src/enrich/stats.ts` is the same numbers as an object for a reader who
+only wants the bill; `ak-docs enrich --json` reports it under `cost`.
+
+### Invented references
+
+`INVENTED_RELATION_REASONS` is `unknown-endpoint`, `unknown-entity`, `unknown-scope`,
+`unknown-directory`, `unknown-diagnostic`. `inventedReferences` counts exactly those, and it is
+reported next to the rejection total rather than folded into it.
+
+The distinction is not cosmetic. A curator that classifies a document badly is wrong about a
+judgement, and a validator rejecting it is the system working. A curator that proposes a relation
+to a module the repository does not contain is making things up, and that number must trend to
+zero or the agent is unusable. Folding the two together hides the second inside the first.
+
+A schema failure or an unknown kind is a rejection and not an invention: `invalid-kind` is not in
+the list, and a test asserts the counts stay apart.
+
+### Stability
+
+`enrichmentStability(current, previous)` compares a run with the one before it and reports both
+halves of the question at once:
+
+- `overlayHashIdentical` — a deterministic agent over an unchanged repository must reach this. The
+  overlay hash covers decisions, not `acceptedAt` and not `stats`, so a second run that is answered
+  entirely from the cache produces the same hash as the first.
+- `proposalIdShare` — for a live model, which will not reach an identical hash, the share of
+  proposal identifiers present in both runs. The share is over the **union** of the two runs'
+  identifiers, so a run that merely proposes fewer things does not score as more stable.
+
+`sharedProposalIds`, `proposalIds` and `previousProposalIds` are reported next to the share, because
+a share of 0.5 over two identifiers and over two hundred are different facts.
+
+## The retrieval delta
+
+`measureOverlayRetrievalDelta` in `src/bench/overlay-delta.ts` answers the question the rest of the
+statistics cannot: did the overlay make retrieval better?
+
+It runs the golden suite twice over **one snapshot** — once with the accepted overlay projected,
+once with `overlay: 'ignore'` — so the only difference between the two runs is the overlay. Not a
+re-scan, not a different revision, not a different configuration. Neither half needs an index on
+disk: both are projected from the snapshot the caller passes.
+
+The rule is asymmetric on purpose. An overlay may leave retrieval unchanged and it may improve it,
+but it must not lower `hitAt3` (`OVERLAY_BLOCKING_METRIC`): aliases, summaries and rank hints an
+agent proposed exist to help an agent find things. A drop is reported as `regression: true` with
+status `regressed`, and the caller exits non-zero. It is a finding about the agent, never a new
+baseline.
+
+The result carries both metric sets, a per-metric delta with `improved` and `worsened` (lower is
+better for `meanContextBytes`, `meanApproxTokens` and `zeroResultRate`), and — the part that makes a
+regression actionable — `lostCases` and `gainedCases`, the case ids that changed at hit@3.
+
+```bash
+# The overlay on disk, against the configured golden suite.
+ak-docs bench retrieval docs/bench/retrieval-suite-v1.json --overlay --text
+
+# Or as part of the run that produced it.
+ak-docs enrich --retrieval-delta --json
+```
+
+Both exit 1 on a regression. Both report the delta in text and in JSON: `formatOverlayRetrievalDeltaText`
+for a person, `retrievalDelta` (or `overlayDelta`) for a machine. The delta is opt-in because it runs
+the suite twice, which is the right cost for an answer about whether the overlay helped and the wrong
+cost for every routine run.
+
+## Study task expectations
+
+The study's correction is the same idea one level up: state what retrieval is expected to return,
+and check it deterministically.
+
+A task in `docs/study/task-suite-v1.json` may now declare `expectedEntities`, `expectedDocuments`
+and `retrievalQueries`. The references are **opaque** — `primary-entrypoint`, not `docs/alpha.md` —
+because the task suite is publication-bound and a repository path in it is a privacy failure, not a
+convenience. A query with nothing expected is refused by the schema: it would check nothing.
+
+The resolution lives in a second artifact, `StudyExpectationsV1` in `src/study/expectations.ts`,
+which declares `scope: 'local'` and binds to the suite by `taskSuiteHash`. It maps each reference to
+the concrete entity ids or document paths it stands for in one repository on the operator's disk.
+It is never published; putting it under `docs/study/` would fail the privacy gate, which is the
+intended outcome rather than a bug. References resolved against a suite that has moved on are
+refused outright.
+
+`studyRetrievalSuite` turns the pair into an ordinary Open Eval Format suite — one case per task
+query, with the resolved targets as `expectedTargets` and the task, repository and category carried
+in metadata — so the same command, the same ranking and the same metrics that gate this
+repository's retrieval answer the study's mechanical questions. `checkStudyExpectations` runs it:
+
+```bash
+ak-docs study expectations docs/study/task-suite-v1.json \
+  --expectations ./local-study-expectations.json --index .doc-bridge/index.json --text
+```
+
+`ok` requires that every reference resolved **and** every case hit. Two failures are reported rather
+than smoothed over: a task that declares no expectations is listed under `withoutExpectations` as
+unchecked, never counted as a pass, and an unresolved reference fails the check, because an
+expectation nobody resolved is an expectation nobody tested — which is exactly the failure this
+replaces. The command exits 1 when the check does not pass.
+
+`validateStudyTaskSuite(suite, { requireExpectations: true })` names the tasks that cannot be checked
+mechanically. It is off by default: a suite written before expectations existed is still a valid
+suite.
+
+### Where the model adjudicator stops
+
+A rubric item may now be prose, or prose with the mechanical check that decides it
+(`acceptance-checks`, `evidence-coverage`, `retrieval-expectations`). `mechanicalRubricItems` and
+`modelRubricItems` split the rubric on that field, and `adjudicatorRubric(task)` is what the
+adjudicator input carries: `rubric` holds only the items no checker can settle, and the settled ones
+travel next to them as `mechanical` — verdicts, for context, not for review. The instruction says so
+in the same words.
+
+Handing a model an item the runner already measured invites it to disagree with a measurement, which
+is how a study ends up with an opinion where it had a number.
+
+## The assisted arm
+
+`registry-assisted` has been reserved since the first suite and has never executed. It now reports
+its own readiness, and `assistedArmReadiness(plan, providers, suite)` is the only thing that decides
+whether it runs:
+
+| Status | When | Effect |
+| --- | --- | --- |
+| `unavailable` | No assisted scenario in the plan; the scenario names no agent identity and version; no provider CLI for the scenario and a model the suite uses | Its executions are recorded as unavailable observations |
+| `ready` with `undeclared` | The scenario declares no `promptVersion`, or no `agentBudget` | The arm runs; the run says what it could not name |
+| `ready` | Everything declared | The arm runs |
+
+An unavailable arm is **recorded, not skipped**: a scenario absent from a ledger is
+indistinguishable from one that was never planned, and comparing the arms it planned is the study's
+whole purpose. Each such execution lands in the ledger with `execution.status: 'unavailable'`,
+`errorCode: 'registry-unavailable'`, and an automated adjudication of `blocked` carrying the reason.
+A missing Registry is a fact about the environment, not a reason to lose the other two arms —
+`assertRunInputs` skips provider validation for those executions so the rest of the run proceeds.
+
+A missing declaration is the other way round: losing the third arm over an undeclared prompt version
+would be worse than running it without one, so the arm runs and `undeclared` says what a reader
+cannot reconstruct from the ledger — the prompt it used, or the agent's cost apart from the model's.
+The run summary prints the status, the reason and the undeclared fields.
+
+A run plan may declare `promptVersion` and `agentBudget` only on `registry-assisted`: the other two
+arms have no agent to budget.
+
+## Tokens to first evidence
+
+The parent PRD's second claim is that an agent reaches correct grounded evidence for fewer tokens
+than by reading the repository. Total tokens at the end of a task does not test that claim: a run
+that wandered for ten thousand tokens and then found the answer looks the same as one that landed on
+it immediately.
+
+`tokensToFirstEvidence` is a canonical provider measurement — tokens consumed before correct
+grounded evidence was in hand — and the metrics report its p95 per scenario as
+`tokensToFirstEvidenceP95`, rounded up to whole tokens. It is in the improvement list, so a round
+that reaches evidence sooner reads as an improvement, and absent or partial coverage is reported as
+`tokensToFirstEvidence` or `tokensToFirstEvidence-partial` under `missingMetrics` rather than as a
+zero.
+
+The assisted arm's enrichment agent is costed apart from the model: `registryAgentInputTokens`,
+`registryAgentOutputTokens`, `registryAgentCostUsd` and `registryAgentRuns`. `registryAgentCostUsd`
+is added into `totalCostUsd`, so the arm cannot look cheap by charging its work to a line nobody
+adds up, and `missingMetrics` names it when an assisted observation reports no agent cost at all.
+The scenario line in `--text` prints the p95 next to the agent cost and its run count.
+
+## Invariants
+
+- `stats` is outside the overlay's content hash. Cost and timing may differ between two runs that
+  decided identically.
+- `inventedReferences` counts only `INVENTED_RELATION_REASONS`, and is reported separately from the
+  rejection total.
+- `proposalIdShare` is over the union of both runs' identifiers.
+- The retrieval delta builds both indexes from one snapshot. An overlay on disk cannot leak into the
+  baseline half.
+- An overlay that lowers hit@3 is a regression with a non-zero exit, in every mode.
+- A study task's expectations are opaque references. A path or a URL in a publication-bound artifact
+  is a privacy failure, and the privacy gate is what says so.
+- An unresolved reference fails the check; a task without expectations is reported unchecked and is
+  never a pass.
+- A rubric item with a mechanical check never reaches the model adjudicator.
+- An unavailable assisted arm is recorded as unavailable observations, and never fails the study.
+
+## Deviation: the committed study suite
+
+The mechanism above is delivered and tested; the committed 24-task suite, run plan and ledgers are
+**not** rewritten to use it.
+
+Two reasons, both about not making the artifacts worse. The suite's content hash is bound to
+published artifacts — the run plan's `taskSuiteHash`, the observation ledgers, the verification
+binding — and changing it invalidates every one of them. And the concrete targets an expectations
+file would resolve belong to the six study repositories, which are not present in this repository,
+so the references could only be guessed.
+
+An operator adding expectations to a round therefore does three things: add `expectedEntities`,
+`expectedDocuments` and `retrievalQueries` to the tasks, re-seal the suite, and write the local
+expectations file that resolves the references against their checkouts. `ak-docs study expectations`
+then reports which tasks can be answered mechanically and which cannot.