agent-doc-system 0.0.0-stage → 1.1.0

package/docs/EVALS.md

@@ -0,0 +1,166 @@
+# EVALS
+
+Does routing an agent through this system retrieve the *right* context, and less
+noise than exploring the repository unaided?
+
+## What is measured
+
+The question is not "does it compile". A catalog can compile perfectly and still
+route an agent into the wrong corner of the repository. The measurement is
+retrieval quality, on the same scenarios, for two retrieval systems.
+
+**baseline-targeted** — what an agent that has no catalog and looks only where
+the change is: the touched component, its subtree, its relations, its documents
+and its checks.
+
+**baseline-full** — what an agent with no routing ends up holding: every entity,
+every relation, every event subject, every check, every document in the
+repository. This is the honest cost of "just read the repository", and it is the
+size yardstick for reduction.
+
+**catalog** — what `agentdoc query` returns for the touched paths, plus the
+context of the components `impact` says are affected.
+
+No model is involved. Retrieval sets are compared as sets, so the measurement
+is deterministic and cannot be moved by phrasing.
+
+## Metrics
+
+| metric | definition |
+|---|---|
+| recall | fraction of the scenario's ground-truth facts the catalog retrieved |
+| critical recall | the same, restricted to critical context: constraints, conflicts, observations, resources, API boundaries, event subjects |
+| verification recall | fraction of the checks that must be run, that were returned |
+| precision | `1 - irrelevance` |
+| irrelevance (noise) | fraction of retrieved facts outside the graph's one-hop closure of the touched entities |
+| retrieval reduction | `1 - catalog facts / baseline-full facts` |
+
+### Why noise is measured against reachability, not against the truth set
+
+The first definition of noise was the complement of truth recall. It is
+degenerate: a scenario lists the handful of facts that matter, so any
+*additional legitimate* fact — a sibling constraint, a second check, an adjacent
+contract — is scored as noise. The metric rewarded returning almost nothing.
+
+It was replaced, before any repository other than the golden corpus was
+measured, with a question that does not depend on the truth set at all:
+
+> did routing drag in anything the graph itself says is unrelated?
+
+Relevance is the **one-hop** graph closure of the touched entities, computed
+from the compiled graph. The metric therefore cannot be satisfied by inflating a
+truth set, and it is not satisfied by returning less either — that shows up as
+recall. Recall is still measured against the truth sets, unchanged.
+
+"One hop" is load-bearing and is enforced. An earlier implementation took the
+transitive closure, which on the golden corpus grows to nearly the whole
+repository; a router that returned *every fact in the repository* then scored as
+precise, because everything was technically adjacent to something. The metric
+was measuring its own closure, not the router.
+
+## Thresholds
+
+Pre-registered in `evals/thresholds.json`, dated, and not moved since:
+
+```json
+{ "minRecall": 0.9, "minCriticalRecall": 0.95, "minVerificationRecall": 0.8,
+  "minPrecision": 0.75, "maxNoise": 0.6, "minRetrievalReduction": 0.4,
+  "minRecallOverBaseline": 0.9 }
+```
+
+Two additional gates use the same numbers:
+
+- **recall must not be worse than baseline-targeted.** A catalog that retrieves
+  less than reading the component would be worthless.
+- **recall must be at least 0.9 × baseline-full recall.** Routing must not be
+  meaningfully worse than reading everything.
+
+### Where the gate is applied
+
+- **Aggregate** (`node evals/run-all.mjs`) applies every threshold,
+  **and fails if any single corpus fails or is missing**. A corpus that is
+  missing fails the gate outright: silently scoring three repositories and
+  reporting PASS would be a gate that shrinks when the system shrinks. The two
+  relative-baseline checks are evaluated per corpus and folded into the
+  aggregate's failure list; the aggregate does not re-derive them, so they are
+  lost if a corpus never runs — which is why a missing corpus is itself a
+  failure. A per-corpus failure is folded into the aggregate exit
+  code, so a broken fixture cannot print FAIL and still exit 0. This is the
+  system-level claim.
+- **Per repository** (`agentdoc eval routing`) applies every threshold except
+  retrieval reduction, which is reported as *not measured* when the whole-repo
+  baseline holds fewer than 100 facts. A one-component repository has almost
+  nothing to route away from; failing it would be punishing the fixture for
+  being small. The floor and the reasoning are in `evals/harness.mjs`.
+
+## Corpora and scenarios
+
+| corpus | scenarios | what it proves |
+|---|---|---|
+| `examples/koda` (golden corpus) | 9 | monorepo discovery, TypeScript + Go, `apps`/`packages`/`workers`, OpenAPI, OIDC, runtime calls, resources, events, verification, journeys, warning acceptances, authority conflicts |
+| `fixtures/a-node-ts-postgres` | 3 | single Node/TypeScript service, REST, Postgres, GitHub Actions |
+| `fixtures/b-go-multi-service` | 4 | multi-service Go, protobuf contracts, shared library, resources |
+| `fixtures/c-alt-monorepo` | 4 | a layout that is deliberately not `apps`+`packages`+`workers`, mixed runtimes, inconsistent pre-existing documentation |
+
+Scenarios are repository-scoped: a scenario only runs against the corpus it was
+written for, declared by a `repository` tag in a `.agentdoc-fixture.json` at the
+repository root. Expectations from one fixture can never leak into another's
+score.
+
+### Scenario families
+
+The golden corpus covers, in order: modify one API consumer; change a database
+schema; change authentication; add an event producer; modify a worker binding;
+change a cross-component endpoint; change a shared package; change a runtime
+cron; debug a production mismatch.
+
+The synthetic fixtures add: change an API contract; change a write path; change
+the only-writer rule; break an end-to-end journey; change an edge worker's flush
+schedule; follow stale documentation while debugging; change a subject vocabulary
+shared across runtimes; add an operation to a contract.
+
+## Running it
+
+```bash
+agentdoc eval routing --json      # one repository (from a corpus directory)
+node evals/run-all.mjs            # aggregate gate — source checkout only
+node evals/run-all.mjs --json     # machine-readable
+```
+
+`run-all.mjs` iterates the corpora in `fixtures/` and `examples/`, so it only
+exists in a source checkout; it is not part of the installed CLI's job.
+
+## What the evaluation changed
+
+The evaluation is a tool, not a ceremony. Running it against the golden corpus
+exposed real defects that no unit test had:
+
+- contract files did not route (changing an endpoint resolved to nothing);
+- a scheduled trigger did not resolve to the component serving the target route,
+  so "who does this cron call?" had no answer;
+- event subjects defined in a shared package were not credited to the component
+  that published them;
+- a documentation file linked by a descriptor did not route to its component, so
+  stale documentation was unreachable — precisely the migration case;
+- the noise metric was degenerate (above), and the `transitiveClosure` variant
+  that replaced it briefly was worse;
+- migration directories were matched by a fixed list of framework conventions
+  rather than by the stable fact that the directory is called `migrations`;
+- cross-unit resolution depended on adapter execution order;
+- a repository path was resolved against the process working directory, so the
+  same commit compiled differently depending on where the compiler ran.
+
+Each was fixed in the engine, not by relaxing a threshold.
+
+## Honest limitations
+
+- Ground truth is hand-authored per scenario. It is written from the
+  repository's own manifests and contracts, not from the system's output, but it
+  is still a human judgement about what matters.
+- The baseline is a model of unassisted exploration, not a record of a real
+  agent session. It is generous to the baseline by construction (baseline-full
+  is everything), which makes the reduction claim conservative.
+- `retrievalReduction` is a fact-count proxy, not a token count. It does not
+  weight a long constraint file against a short ref.
+- Scenarios touch one or two files. A change spanning twenty files is not
+  measured.

package/docs/MIGRATION.md

@@ -0,0 +1,261 @@
+# MIGRATION
+
+Two workflows. CREATE for a repository with no catalog; MIGRATE for an
+established one. Both start with the same audit; MIGRATE adds a source-of-truth
+mapping and a reviewed-ambiguity list that a human must work through.
+
+## Choosing
+
+| situation | mode |
+|---|---|
+| no `agentdoc/` directory, small or new repository | CREATE |
+| no `agentdoc/`, large or established repository | MIGRATE |
+| an existing catalog, documentation that has drifted | MIGRATE |
+| a catalog that is stale but structurally sound | MAINTAIN |
+
+The test is not size. It is whether the repository has documentation that
+*claims* things. Claims need verifying; that is migration work regardless of how
+few components there are.
+
+## The audit
+
+```bash
+agentdoc audit
+```
+
+`audit` runs even when the repository cannot compile, and reports compilation
+failures as part of the report. It inventories four sources of truth
+independently:
+
+1. **manifests and code** — what can be derived;
+2. **machine-readable contracts** — what boundaries are declared;
+3. **authored documentation** — what someone believed;
+4. **authoritative external systems** — what is deployed, via ObservationSets.
+
+## The report
+
+```json
+{
+  "inventory":  { "files", "units", "entities", "components", "apis", "relations", "contracts", "documentationFiles", "observations", "verification" },
+  "facts":      [ { "fact": "component:default/teacher placement.system", "classification": "DERIVED", "evidence": ["authored"] } ],
+  "counts":     { "CONFIRMED": 4, "DERIVED": 38 },
+  "contradictions": [ ... ],
+  "documentationContradictions": [ ... ],
+  "normative":  { "sections": [ { "path", "section", "heading", "sha256", "coveredBy" } ], "uncovered": [ ... ], "requirements": [ ... ] },
+  "reviewedAmbiguity": [ ... ],
+  "documentation": { "orphanDocs", "undocumentedComponents", "unverifiedComponents" },
+  "contracts":  { "inventory", "unclaimed" },
+  "observations": [ { "id", "capturedAt", "maxAgeDays", "environment" } ],
+  "scaffold":   { "files", "summary", "rule" }
+}
+```
+
+### Classification
+
+Every major fact lands in exactly one class:
+
+| class | meaning | may be authored? |
+|---|---|---|
+| `CONFIRMED` | two or more independent sources agree | yes |
+| `DERIVED` | deterministically derivable from the repository | yes, once reviewed |
+| `OBSERVED` | read from an authoritative external system | **no** — it belongs in an ObservationSet |
+| `CONFLICT` | sources disagree and no authority rule settles it | **no** — needs a human |
+| `UNRESOLVED` | neither the repository nor an external system answers it | **no** — needs a human |
+
+**Uncertainty is never promoted to authored truth.** This is the rule the whole
+report exists to enforce.
+
+### Documentation contradictions
+
+Two checks run against claims the graph actually owns, and only those. A document
+saying "we use Kafka" is not checkable until a Resource descriptor says what the
+Kafka thing is; a document saying "there is no edge component" is checkable
+immediately, and is `NEGATED_COMPONENT`. A port claim in a document linked to a
+component is `PORT_MISMATCH` when the source binds a different port, and
+`UNLINKED_PORT_CLAIM` when the document is linked to nothing and therefore cannot
+be checked at all.
+
+The rule for extending this: only check vocabulary the graph owns. A check that
+guesses is a check that cries wolf.
+
+**These are the only automatic documentation checks.** The audit does not read
+prose for meaning, so it does not find where an arbitrary sentence contradicts
+the code. A specification saying "the verifier runtime is installed on the
+production host" is invisible to it until someone turns that sentence into a
+requirement — which is what the next section makes mandatory.
+
+### Normative requirements
+
+A specification that is linked to a component is *routed*; that says nothing
+about whether anyone checked the implementation against it. List the documents
+that state obligations in `docs.normative`:
+
+```yaml
+docs:
+  normative:
+    - specs/**
+    - verification_layer/*_PROFILE.md
+```
+
+Every level-2 (`##`) section of those documents is then pinned by digest and
+must be accounted for in one of two ways:
+
+- **a requirement** on the component that must fulfil it, citing the section:
+
+  ```yaml
+  # provisioning/agentdoc.yaml
+  spec:
+    requirements:
+      - id: verification-runtime-on-host
+        key: requirement.provides.verification-runtime
+        value: provided
+        source:
+          path: specs/EXECUTION_LAYER_SPEC.md
+          section: 13-host-configuration
+          sha256: <digest from `agentdoc audit` → normative.sections>
+        satisfiedBy: [ansible]
+        expects:
+          installs: /var/lib/agent-platform-runtime/verification-python-v1
+  ```
+
+- **an attestation** in `normativeCoverage` that the section carries no
+  implementation obligation (`disposition: no-implementation-obligation` or
+  `informative`), with a reason and a review condition.
+
+A section that is neither raises `AGENTDOC_NORMATIVE_UNCOVERED`. A requirement
+is an `AUTHORED` fact that can only be *corroborated*: until implementation
+evidence on the same key agrees with it — a `DERIVED` fact from an adapter
+named in `satisfiedBy` that finds what `expects` names (see `ADAPTERS.md`,
+`ansible`), or a
+`requirement-satisfaction` reviewed override citing the implementation — it is
+`UNRESOLVED` and raises `AGENTDOC_REQUIREMENT_UNVERIFIED`, naming the adapter in
+`satisfiedBy` that is missing. Implementation evidence that disagrees is a
+conflict no authority rule may settle (see `AUTHORITY.md`). Editing a cited
+section invalidates its requirement or attestation
+(`AGENTDOC_NORMATIVE_SOURCE_STALE`): the review has to be redone.
+
+A missing adapter lowers coverage; it never makes an obligation disappear.
+
+### Reviewed ambiguity
+
+The list a human must work through:
+
+| kind | what a human decides |
+|---|---|
+| `EVENT_WITHOUT_PRODUCER` | is this subject reserved, or is a producer missing? |
+| `COMPONENT_OWNERSHIP` | which system or domain owns this? |
+| `UNCLAIMED_CONTRACT` | which API owns this contract, or is it dead? |
+| `REVIEWED_OVERRIDE` | is this interpretation still true? |
+| `NEGATED_COMPONENT`, `PORT_MISMATCH`, `UNLINKED_PORT_CLAIM` | which source is right? |
+| `NORMATIVE_UNCOVERED` | what does this section oblige, and which component must do it? |
+| `REQUIREMENT_UNVERIFIED` | where is the implementation — or is this a gap to fix before release? |
+
+## CREATE
+
+```bash
+agentdoc init          # configuration + doc skeleton
+$EDITOR agentdoc/agentdoc.config.yaml        # layout, namespace, adapters
+agentdoc audit         # what is provable
+agentdoc scaffold --write
+$EDITOR '**/agentdoc.yaml'                   # every generated description
+agentdoc validate
+agentdoc compile
+agentdoc check
+```
+
+`init` writes `agentdoc/agentdoc.config.yaml`, the four central descriptor
+files, `agentdoc/journeys.yaml`, the documentation hierarchy
+(`docs/PRODUCT.md`, `docs/ARCHITECTURE.md`, `docs/CONSTRAINTS.md`,
+`docs/adr/`), the observations README, and a CI workflow. It never overwrites an
+existing file.
+
+`scaffold` is a dry run unless `--write` is passed.
+
+### What the scaffolder will and will not do
+
+It writes a component descriptor only for a unit it can name from the
+repository — a package name, a module path, or a directory name. It writes a
+description that is a template sentence naming the artifact evidence. It does
+**not** choose a system or a domain, because placement is an ownership decision
+no extractor can make; the generated descriptor carries a commented
+`placementRationale` prompt instead.
+
+That is deliberate. A scaffolder that guesses ownership produces a catalog
+nobody trusts.
+
+## MIGRATE
+
+```bash
+agentdoc init
+# set componentDescriptors to the real layout; enable the right adapters
+agentdoc audit > migration-report.json
+```
+
+Then, in order:
+
+1. **Source-of-truth mapping.** Configure `docs.normative`, then work
+   `normative.uncovered`: every section of every normative document becomes a
+   component requirement or a reviewed attestation. Prove each requirement
+   with implementation evidence, or leave it `REQUIREMENT_UNVERIFIED` for a
+   human. For other major claims, record which of the four inventories
+   supports them; anything supported by none goes to `reviewedAmbiguity`.
+2. **Contradiction detection.** Work `contradictions` and
+   `documentationContradictions`. Each one needs a human decision: fix the
+   documentation, fix the code, or record an authority rule saying which class
+   wins for that key.
+3. **Orphan documentation.** `documentation.orphanDocs`. For each: link it to a
+   component via `context.docs`, or delete it. An orphan is a document nothing
+   will ever route to.
+4. **Missing documentation.** `documentation.undocumentedComponents` and
+   `documentation.unverifiedComponents`. These are the components an agent will
+   get wrong.
+5. **Stale contracts.** `contracts.unclaimed`, and any contract whose consumers
+   changed. Point the API descriptor at the current file; do not duplicate it.
+6. **Ownership reconstruction.** Placement comes from the code where the code is
+   unambiguous (a service that only ever talks to one identity provider), and
+   from a human everywhere else.
+7. **Catalog proposal.** `scaffold` lists what it would write.
+8. **Migration.** `scaffold --write`, then review every generated file.
+9. **Validation.** `validate` → `compile` → `check --strict`. The strict gate
+   fails on every warning not accepted with pinned evidence, so a migration
+   cannot finish with an unreviewed normative section or an unproven
+   requirement. A known gap is accepted explicitly (`warningAcceptances`) and
+   stays `UNRESOLVED` in the audit.
+
+### Do not trust the documentation
+
+The single most common migration failure is copying existing prose into
+descriptions. The audit exists to find out how much of that prose is still true.
+A description is a one-sentence present-tense statement of current purpose; if
+you cannot write one, the component's purpose is not understood yet, and that is
+worth discovering before documenting it.
+
+## Documentation hierarchy
+
+| kind | belongs there | must not |
+|---|---|---|
+| `PRODUCT.md` | what it is, for whom, what it does not do | describe components |
+| `ARCHITECTURE.md` | why the shape is what it is | enumerate components or contracts |
+| `CONSTRAINTS.md` | repository-wide rules, one rule one reason one check | restate a component's constraints |
+| `docs/adr/NNNN-*.md` | one decision, its consequences, superseded not edited | be a design document |
+| `<component>/CONSTRAINTS.md` | rules for that component only | duplicate a repository-wide rule |
+| `<component>/README.md` | how to run it, what it is for | restate the catalog |
+| runbooks | operational procedure | contain architectural decisions |
+| `agentdoc/observations/` | what a live system said, with evidence | contain credentials |
+| the graph | topology, contracts, events, checks, conflicts | contain prose |
+
+**The duplication rule.** A fact lives in exactly one place. Elsewhere, point at
+it. If `ARCHITECTURE.md` lists the components, it will be wrong within a month,
+and `agentdoc query` will be right. The architecture document explains *why*;
+the graph says *what*.
+
+## Finishing
+
+```bash
+agentdoc validate && agentdoc check
+agentdoc eval routing
+git add -A && git commit
+```
+
+Then install CI (`CI.md`) so the next drift is caught by a gate rather than by a
+review.

package/docs/PROVENANCE.md

@@ -0,0 +1,133 @@
+# PROVENANCE
+
+Every important fact in the compiled graph is traceable to evidence. This
+document defines what a provenance record is, how it is produced, and the rules
+that keep evidence free of secrets.
+
+## The record
+
+```json
+{
+  "id": "p-0075",
+  "class": "observed",
+  "path": "workers/koda-cron/wrangler.toml",
+  "extractor": { "name": "platform-manifest-extractor", "version": "1.0.0" },
+  "contentHash": "sha256:9f2c...",
+  "location": { "lineStart": 5, "lineEnd": 7, "jsonPointer": "/facts/2" },
+  "subjects": ["/entities/9/derived/bindings/0", "/conflicts/2"]
+}
+```
+
+| field | meaning |
+|---|---|
+| `id` | positional over a canonical sort, so ids are stable for a given input set |
+| `class` | how the evidence was obtained (below) |
+| `path` | repository-relative; never absolute, never outside the checkout |
+| `extractor` | which component read it, and at what version |
+| `contentHash` | `sha256:` of the file bytes as read |
+| `location` | optional: line range or JSON pointer, when the evidence is narrower than the file |
+| `subjects` | RFC 6901 pointers into this graph that the record proves |
+
+## Classes
+
+| class | meaning | typical extractor |
+|---|---|---|
+| `declared` | the file states the fact outright | `descriptor-reader`, `placement-extractor` |
+| `reviewed` | a human interpretation recorded in configuration, not derivable from a file | `override-extractor` |
+| `observed` | the file shows the fact | `manifest-extractor`, `import-extractor`, `ci-extractor` |
+| `validated` | two independent files agree | `api-contract-extractor`, `oidc-extractor` |
+| `runtime` | read from an external system | `observation:<collector>` |
+| `standard` | the authority is a published standard | `standard-extractor` |
+
+`runtime` records are the only ones whose path is not a source file of the
+repository under analysis in the ordinary sense: they point at the committed
+evidence bundle the observation cites.
+
+`reviewed` exists so that a human interpretation is not filed as `declared`.
+Filed that way it mapped to `AUTHORED` — the strongest class — so a reviewed
+override corroborated by a source reference was relabelled `AUTHORED` on the
+merged edge, and the graph understated its own highest-authority input. The class
+is not an election either: elections happen per fact in the authority model, and a
+provenance class must not decide one on its own.
+
+## Coverage rules
+
+- Every relation, external dependency, capability, verification entry, gate,
+  journey, contract, conflict and assertion carries at least one
+  `provenanceIds` entry.
+- Every entity carries provenance for its authored fields and, for components,
+  for its derived block.
+- A relation, index entry or fact with no provenance is a compiler defect. The
+  pipeline fails with `AGENTDOC_PROVENANCE` rather than emitting it.
+
+## Subject resolution
+
+Extractors emit symbolic subjects while they run (`rel:pending`,
+`ass:pending`, `evt:<pattern>`, …) and the compiler binds them to concrete
+pointers once the whole graph exists. This is why an extractor's output does not
+depend on how many other extractors ran first.
+
+A subject that never binds is an error: `AGENTDOC_PROVENANCE`. The single
+exception is evidence whose candidate edge or fact was *dropped* (an unresolved
+reviewed override, a rejected candidate). Those records are removed and the
+count is reported as `droppedProvenance` in the compile stats, so a regression
+here is visible rather than silent.
+
+## Input hashing
+
+`source.inputHash` covers:
+
+1. compiler and graph-schema versions, entity api version, authority-rule count,
+   every enabled adapter's name and version, and the content hash of every
+   shipped schema;
+2. every file the compiler read — path and content, length-prefixed;
+3. every existence probe — path and result;
+4. every directory listing — directory and sorted file list.
+
+Every byte the compiler reads flows through a single tracked read path, so the
+record is complete by construction rather than by remembering to register files.
+A change to any input changes the hash; `check` refuses a graph whose hash
+disagrees.
+
+## Secrets
+
+Two independent layers, because derivation can synthesise a secret-shaped string
+that no source scan would have caught.
+
+**Layer 1 — sources.** Every authored or observed document is scanned before it
+can influence any output. Refusal is `AGENTDOC_SECRET`, with the JSON pointer.
+
+**Layer 2 — outputs.** The serialized graph, every audit report, every query
+payload and every eval report is scanned again before it is written or printed.
+Refusal is `AGENTDOC_SECRET` and the command exits non-zero. Nothing is written
+partially.
+
+Detected shapes include private-key headers, cloud access-key ids, provider
+token prefixes, JWT-shaped triples, URLs with embedded credentials, `Bearer`
+tokens, and `secret: value` / `token = value` style assignments.
+
+**Commands are redacted, not dropped.** A verification command is a routing
+index, not a credential transport. An inline assignment whose name contains
+`SECRET`, `TOKEN`, `PASSWORD`, `PASSPHRASE`, `API_KEY`, `PRIVATE_KEY`,
+`CREDENTIAL` or `ACCESS_KEY` keeps its name and loses its value:
+
+```
+DATABASE_URL=postgres://u:p@host/db vitest run
+  ->  DATABASE_URL=[REDACTED] vitest run
+```
+
+The redaction marker itself is not treated as a secret, so a redacted command
+remains runnable and printable.
+
+**Observations.** An `evidenceBundle` must be a committed, repository-relative
+file. A temp path, a console URL with a session token, or a screenshot filename
+is not evidence. The environment identifier is constrained to a slug; nothing in
+an observation may be a credential.
+
+## What is deliberately not provenance
+
+- **Confidence.** A heuristic derivation is still traceable; `confidence:
+  candidate` says it may not be elected. Provenance and confidence answer
+  different questions.
+- **Ownership.** Placement is an authored fact with a `declared` record, not a
+  derived one. The extractor identity tells you who read it, not who decided it.