agent-doc-system 0.0.0-stage → 1.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/INSTALL_FOR_AGENTS.md +60 -0
- package/LICENSE +21 -0
- package/README.md +151 -2
- package/adapters/cache.mjs +27 -0
- package/adapters/contract-consumers.mjs +56 -0
- package/adapters/database.mjs +116 -0
- package/adapters/deploy-config.mjs +34 -0
- package/adapters/dockerfile.mjs +26 -0
- package/adapters/events.mjs +145 -0
- package/adapters/generic.mjs +86 -0
- package/adapters/github-actions.mjs +106 -0
- package/adapters/go.mjs +91 -0
- package/adapters/graphql.mjs +28 -0
- package/adapters/index.mjs +55 -0
- package/adapters/node.mjs +118 -0
- package/adapters/object-store.mjs +64 -0
- package/adapters/oidc.mjs +56 -0
- package/adapters/openapi.mjs +50 -0
- package/adapters/pnpm.mjs +42 -0
- package/adapters/protobuf.mjs +35 -0
- package/adapters/registry.mjs +27 -0
- package/adapters/resource-client.mjs +61 -0
- package/adapters/turborepo.mjs +135 -0
- package/adapters/typescript.mjs +85 -0
- package/adapters/wrangler.mjs +206 -0
- package/bin/agentdoc.mjs +478 -0
- package/bin/usage.txt +26 -0
- package/core/acceptances.mjs +69 -0
- package/core/audit.mjs +407 -0
- package/core/authority/classes.mjs +39 -0
- package/core/authority/engine.mjs +243 -0
- package/core/authority/facts.mjs +41 -0
- package/core/codes.mjs +119 -0
- package/core/compile.mjs +881 -0
- package/core/contracts.mjs +121 -0
- package/core/derive.mjs +102 -0
- package/core/descriptors.mjs +486 -0
- package/core/determinism.mjs +95 -0
- package/core/discover.mjs +147 -0
- package/core/facts.mjs +53 -0
- package/core/fsx.mjs +342 -0
- package/core/graph.mjs +260 -0
- package/core/impact.mjs +175 -0
- package/core/indexes.mjs +165 -0
- package/core/jsonschema.mjs +298 -0
- package/core/observations.mjs +129 -0
- package/core/provenance.mjs +83 -0
- package/core/query.mjs +384 -0
- package/core/relations.mjs +286 -0
- package/core/scaffold.mjs +322 -0
- package/core/secrets.mjs +92 -0
- package/core/sourcescan.mjs +88 -0
- package/core/toml.mjs +163 -0
- package/core/yaml.mjs +475 -0
- package/docs/ADAPTERS.md +172 -0
- package/docs/AUTHORITY.md +260 -0
- package/docs/CI.md +136 -0
- package/docs/COMPARISON-KODA.md +164 -0
- package/docs/EVALS.md +166 -0
- package/docs/MIGRATION.md +193 -0
- package/docs/PROVENANCE.md +133 -0
- package/docs/QUERY.md +180 -0
- package/docs/SPEC.md +339 -0
- package/docs/VERIFICATION.md +28 -0
- package/docs/ci/generic-ci.sh +28 -0
- package/docs/ci/github-actions.yml +35 -0
- package/evals/harness.mjs +426 -0
- package/evals/scenarios/fixture-a-01-change-api.json +9 -0
- package/evals/scenarios/fixture-a-02-change-db.json +9 -0
- package/evals/scenarios/fixture-a-03-enforce-constraint.json +9 -0
- package/evals/scenarios/fixture-b-01-change-proto.json +9 -0
- package/evals/scenarios/fixture-b-02-change-write-path.json +9 -0
- package/evals/scenarios/fixture-b-03-change-shared-lib.json +9 -0
- package/evals/scenarios/fixture-b-04-journey.json +9 -0
- package/evals/scenarios/fixture-c-01-mixed-runtime-edge.json +19 -0
- package/evals/scenarios/fixture-c-02-stale-doc-contradiction.json +9 -0
- package/evals/scenarios/fixture-c-03-shared-vocabulary.json +9 -0
- package/evals/scenarios/fixture-c-04-contract-change.json +9 -0
- package/evals/scenarios/koda-01-modify-api-consumer.json +26 -0
- package/evals/scenarios/koda-02-change-db-schema.json +20 -0
- package/evals/scenarios/koda-03-change-auth.json +21 -0
- package/evals/scenarios/koda-04-add-event-producer.json +19 -0
- package/evals/scenarios/koda-05-modify-worker-binding.json +25 -0
- package/evals/scenarios/koda-06-change-cross-component-endpoint.json +22 -0
- package/evals/scenarios/koda-07-change-shared-package.json +16 -0
- package/evals/scenarios/koda-08-change-runtime-cron.json +23 -0
- package/evals/scenarios/koda-09-debug-production-mismatch.json +26 -0
- package/evals/thresholds.json +11 -0
- package/package.json +46 -4
- package/schemas/api.schema.json +84 -0
- package/schemas/common.schema.json +107 -0
- package/schemas/component.schema.json +40 -0
- package/schemas/config.schema.json +685 -0
- package/schemas/domain.schema.json +14 -0
- package/schemas/graph.schema.json +1084 -0
- package/schemas/journeys.schema.json +47 -0
- package/schemas/observation.schema.json +143 -0
- package/schemas/resource.schema.json +28 -0
- package/schemas/system.schema.json +26 -0
- package/skills/agent-doc-system/SKILL.md +113 -0
- package/skills/agent-doc-system/references/authority-and-conflicts.md +70 -0
- package/skills/agent-doc-system/references/cli.md +89 -0
- package/skills/agent-doc-system/references/create-migrate.md +101 -0
- package/skills/agent-doc-system/references/maintenance.md +52 -0
- package/templates/ADR.md +26 -0
- package/templates/ARCHITECTURE.md +19 -0
- package/templates/CONSTRAINTS.md +17 -0
- package/templates/PRODUCT.md +17 -0
- package/templates/api.yaml +10 -0
- package/templates/ci.yml +31 -0
- package/templates/component.yaml +14 -0
- package/templates/domain.yaml +6 -0
- package/templates/journey.yaml +7 -0
- package/templates/observation.yaml +19 -0
- package/templates/resource.yaml +8 -0
- package/templates/system.yaml +10 -0
- package/templates/warning-acceptance.yaml +14 -0
package/docs/EVALS.md
ADDED
|
@@ -0,0 +1,166 @@
|
|
|
1
|
+
# EVALS
|
|
2
|
+
|
|
3
|
+
Does routing an agent through this system retrieve the *right* context, and less
|
|
4
|
+
noise than exploring the repository unaided?
|
|
5
|
+
|
|
6
|
+
## What is measured
|
|
7
|
+
|
|
8
|
+
The question is not "does it compile". A catalog can compile perfectly and still
|
|
9
|
+
route an agent into the wrong corner of the repository. The measurement is
|
|
10
|
+
retrieval quality, on the same scenarios, for two retrieval systems.
|
|
11
|
+
|
|
12
|
+
**baseline-targeted** — what an agent that has no catalog and looks only where
|
|
13
|
+
the change is: the touched component, its subtree, its relations, its documents
|
|
14
|
+
and its checks.
|
|
15
|
+
|
|
16
|
+
**baseline-full** — what an agent with no routing ends up holding: every entity,
|
|
17
|
+
every relation, every event subject, every check, every document in the
|
|
18
|
+
repository. This is the honest cost of "just read the repository", and it is the
|
|
19
|
+
size yardstick for reduction.
|
|
20
|
+
|
|
21
|
+
**catalog** — what `agentdoc query` returns for the touched paths, plus the
|
|
22
|
+
context of the components `impact` says are affected.
|
|
23
|
+
|
|
24
|
+
No model is involved. Retrieval sets are compared as sets, so the measurement
|
|
25
|
+
is deterministic and cannot be moved by phrasing.
|
|
26
|
+
|
|
27
|
+
## Metrics
|
|
28
|
+
|
|
29
|
+
| metric | definition |
|
|
30
|
+
|---|---|
|
|
31
|
+
| recall | fraction of the scenario's ground-truth facts the catalog retrieved |
|
|
32
|
+
| critical recall | the same, restricted to critical context: constraints, conflicts, observations, resources, API boundaries, event subjects |
|
|
33
|
+
| verification recall | fraction of the checks that must be run, that were returned |
|
|
34
|
+
| precision | `1 - irrelevance` |
|
|
35
|
+
| irrelevance (noise) | fraction of retrieved facts outside the graph's one-hop closure of the touched entities |
|
|
36
|
+
| retrieval reduction | `1 - catalog facts / baseline-full facts` |
|
|
37
|
+
|
|
38
|
+
### Why noise is measured against reachability, not against the truth set
|
|
39
|
+
|
|
40
|
+
The first definition of noise was the complement of truth recall. It is
|
|
41
|
+
degenerate: a scenario lists the handful of facts that matter, so any
|
|
42
|
+
*additional legitimate* fact — a sibling constraint, a second check, an adjacent
|
|
43
|
+
contract — is scored as noise. The metric rewarded returning almost nothing.
|
|
44
|
+
|
|
45
|
+
It was replaced, before any repository other than the golden corpus was
|
|
46
|
+
measured, with a question that does not depend on the truth set at all:
|
|
47
|
+
|
|
48
|
+
> did routing drag in anything the graph itself says is unrelated?
|
|
49
|
+
|
|
50
|
+
Relevance is the **one-hop** graph closure of the touched entities, computed
|
|
51
|
+
from the compiled graph. The metric therefore cannot be satisfied by inflating a
|
|
52
|
+
truth set, and it is not satisfied by returning less either — that shows up as
|
|
53
|
+
recall. Recall is still measured against the truth sets, unchanged.
|
|
54
|
+
|
|
55
|
+
"One hop" is load-bearing and is enforced. An earlier implementation took the
|
|
56
|
+
transitive closure, which on the golden corpus grows to nearly the whole
|
|
57
|
+
repository; a router that returned *every fact in the repository* then scored as
|
|
58
|
+
precise, because everything was technically adjacent to something. The metric
|
|
59
|
+
was measuring its own closure, not the router.
|
|
60
|
+
|
|
61
|
+
## Thresholds
|
|
62
|
+
|
|
63
|
+
Pre-registered in `evals/thresholds.json`, dated, and not moved since:
|
|
64
|
+
|
|
65
|
+
```json
|
|
66
|
+
{ "minRecall": 0.9, "minCriticalRecall": 0.95, "minVerificationRecall": 0.8,
|
|
67
|
+
"minPrecision": 0.75, "maxNoise": 0.6, "minRetrievalReduction": 0.4,
|
|
68
|
+
"minRecallOverBaseline": 0.9 }
|
|
69
|
+
```
|
|
70
|
+
|
|
71
|
+
Two additional gates use the same numbers:
|
|
72
|
+
|
|
73
|
+
- **recall must not be worse than baseline-targeted.** A catalog that retrieves
|
|
74
|
+
less than reading the component would be worthless.
|
|
75
|
+
- **recall must be at least 0.9 × baseline-full recall.** Routing must not be
|
|
76
|
+
meaningfully worse than reading everything.
|
|
77
|
+
|
|
78
|
+
### Where the gate is applied
|
|
79
|
+
|
|
80
|
+
- **Aggregate** (`node evals/run-all.mjs`) applies every threshold,
|
|
81
|
+
**and fails if any single corpus fails or is missing**. A corpus that is
|
|
82
|
+
missing fails the gate outright: silently scoring three repositories and
|
|
83
|
+
reporting PASS would be a gate that shrinks when the system shrinks. The two
|
|
84
|
+
relative-baseline checks are evaluated per corpus and folded into the
|
|
85
|
+
aggregate's failure list; the aggregate does not re-derive them, so they are
|
|
86
|
+
lost if a corpus never runs — which is why a missing corpus is itself a
|
|
87
|
+
failure. A per-corpus failure is folded into the aggregate exit
|
|
88
|
+
code, so a broken fixture cannot print FAIL and still exit 0. This is the
|
|
89
|
+
system-level claim.
|
|
90
|
+
- **Per repository** (`agentdoc eval routing`) applies every threshold except
|
|
91
|
+
retrieval reduction, which is reported as *not measured* when the whole-repo
|
|
92
|
+
baseline holds fewer than 100 facts. A one-component repository has almost
|
|
93
|
+
nothing to route away from; failing it would be punishing the fixture for
|
|
94
|
+
being small. The floor and the reasoning are in `evals/harness.mjs`.
|
|
95
|
+
|
|
96
|
+
## Corpora and scenarios
|
|
97
|
+
|
|
98
|
+
| corpus | scenarios | what it proves |
|
|
99
|
+
|---|---|---|
|
|
100
|
+
| `examples/koda` (golden corpus) | 9 | monorepo discovery, TypeScript + Go, `apps`/`packages`/`workers`, OpenAPI, OIDC, runtime calls, resources, events, verification, journeys, warning acceptances, authority conflicts |
|
|
101
|
+
| `fixtures/a-node-ts-postgres` | 3 | single Node/TypeScript service, REST, Postgres, GitHub Actions |
|
|
102
|
+
| `fixtures/b-go-multi-service` | 4 | multi-service Go, protobuf contracts, shared library, resources |
|
|
103
|
+
| `fixtures/c-alt-monorepo` | 4 | a layout that is deliberately not `apps`+`packages`+`workers`, mixed runtimes, inconsistent pre-existing documentation |
|
|
104
|
+
|
|
105
|
+
Scenarios are repository-scoped: a scenario only runs against the corpus it was
|
|
106
|
+
written for, declared by a `repository` tag in a `.agentdoc-fixture.json` at the
|
|
107
|
+
repository root. Expectations from one fixture can never leak into another's
|
|
108
|
+
score.
|
|
109
|
+
|
|
110
|
+
### Scenario families
|
|
111
|
+
|
|
112
|
+
The golden corpus covers, in order: modify one API consumer; change a database
|
|
113
|
+
schema; change authentication; add an event producer; modify a worker binding;
|
|
114
|
+
change a cross-component endpoint; change a shared package; change a runtime
|
|
115
|
+
cron; debug a production mismatch.
|
|
116
|
+
|
|
117
|
+
The synthetic fixtures add: change an API contract; change a write path; change
|
|
118
|
+
the only-writer rule; break an end-to-end journey; change an edge worker's flush
|
|
119
|
+
schedule; follow stale documentation while debugging; change a subject vocabulary
|
|
120
|
+
shared across runtimes; add an operation to a contract.
|
|
121
|
+
|
|
122
|
+
## Running it
|
|
123
|
+
|
|
124
|
+
```bash
|
|
125
|
+
agentdoc eval routing --json # one repository (from a corpus directory)
|
|
126
|
+
node evals/run-all.mjs # aggregate gate — source checkout only
|
|
127
|
+
node evals/run-all.mjs --json # machine-readable
|
|
128
|
+
```
|
|
129
|
+
|
|
130
|
+
`run-all.mjs` iterates the corpora in `fixtures/` and `examples/`, so it only
|
|
131
|
+
exists in a source checkout; it is not part of the installed CLI's job.
|
|
132
|
+
|
|
133
|
+
## What the evaluation changed
|
|
134
|
+
|
|
135
|
+
The evaluation is a tool, not a ceremony. Running it against the golden corpus
|
|
136
|
+
exposed real defects that no unit test had:
|
|
137
|
+
|
|
138
|
+
- contract files did not route (changing an endpoint resolved to nothing);
|
|
139
|
+
- a scheduled trigger did not resolve to the component serving the target route,
|
|
140
|
+
so "who does this cron call?" had no answer;
|
|
141
|
+
- event subjects defined in a shared package were not credited to the component
|
|
142
|
+
that published them;
|
|
143
|
+
- a documentation file linked by a descriptor did not route to its component, so
|
|
144
|
+
stale documentation was unreachable — precisely the migration case;
|
|
145
|
+
- the noise metric was degenerate (above), and the `transitiveClosure` variant
|
|
146
|
+
that replaced it briefly was worse;
|
|
147
|
+
- migration directories were matched by a fixed list of framework conventions
|
|
148
|
+
rather than by the stable fact that the directory is called `migrations`;
|
|
149
|
+
- cross-unit resolution depended on adapter execution order;
|
|
150
|
+
- a repository path was resolved against the process working directory, so the
|
|
151
|
+
same commit compiled differently depending on where the compiler ran.
|
|
152
|
+
|
|
153
|
+
Each was fixed in the engine, not by relaxing a threshold.
|
|
154
|
+
|
|
155
|
+
## Honest limitations
|
|
156
|
+
|
|
157
|
+
- Ground truth is hand-authored per scenario. It is written from the
|
|
158
|
+
repository's own manifests and contracts, not from the system's output, but it
|
|
159
|
+
is still a human judgement about what matters.
|
|
160
|
+
- The baseline is a model of unassisted exploration, not a record of a real
|
|
161
|
+
agent session. It is generous to the baseline by construction (baseline-full
|
|
162
|
+
is everything), which makes the reduction claim conservative.
|
|
163
|
+
- `retrievalReduction` is a fact-count proxy, not a token count. It does not
|
|
164
|
+
weight a long constraint file against a short ref.
|
|
165
|
+
- Scenarios touch one or two files. A change spanning twenty files is not
|
|
166
|
+
measured.
|
|
@@ -0,0 +1,193 @@
|
|
|
1
|
+
# MIGRATION
|
|
2
|
+
|
|
3
|
+
Two workflows. CREATE for a repository with no catalog; MIGRATE for an
|
|
4
|
+
established one. Both start with the same audit; MIGRATE adds a source-of-truth
|
|
5
|
+
mapping and a reviewed-ambiguity list that a human must work through.
|
|
6
|
+
|
|
7
|
+
## Choosing
|
|
8
|
+
|
|
9
|
+
| situation | mode |
|
|
10
|
+
|---|---|
|
|
11
|
+
| no `agentdoc/` directory, small or new repository | CREATE |
|
|
12
|
+
| no `agentdoc/`, large or established repository | MIGRATE |
|
|
13
|
+
| an existing catalog, documentation that has drifted | MIGRATE |
|
|
14
|
+
| a catalog that is stale but structurally sound | MAINTAIN |
|
|
15
|
+
|
|
16
|
+
The test is not size. It is whether the repository has documentation that
|
|
17
|
+
*claims* things. Claims need verifying; that is migration work regardless of how
|
|
18
|
+
few components there are.
|
|
19
|
+
|
|
20
|
+
## The audit
|
|
21
|
+
|
|
22
|
+
```bash
|
|
23
|
+
agentdoc audit
|
|
24
|
+
```
|
|
25
|
+
|
|
26
|
+
`audit` runs even when the repository cannot compile, and reports compilation
|
|
27
|
+
failures as part of the report. It inventories four sources of truth
|
|
28
|
+
independently:
|
|
29
|
+
|
|
30
|
+
1. **manifests and code** — what can be derived;
|
|
31
|
+
2. **machine-readable contracts** — what boundaries are declared;
|
|
32
|
+
3. **authored documentation** — what someone believed;
|
|
33
|
+
4. **authoritative external systems** — what is deployed, via ObservationSets.
|
|
34
|
+
|
|
35
|
+
## The report
|
|
36
|
+
|
|
37
|
+
```json
|
|
38
|
+
{
|
|
39
|
+
"inventory": { "files", "units", "entities", "components", "apis", "relations", "contracts", "documentationFiles", "observations", "verification" },
|
|
40
|
+
"facts": [ { "fact": "component:default/teacher placement.system", "classification": "DERIVED", "evidence": ["authored"] } ],
|
|
41
|
+
"counts": { "CONFIRMED": 4, "DERIVED": 38 },
|
|
42
|
+
"contradictions": [ ... ],
|
|
43
|
+
"documentationContradictions": [ ... ],
|
|
44
|
+
"reviewedAmbiguity": [ ... ],
|
|
45
|
+
"documentation": { "orphanDocs", "undocumentedComponents", "unverifiedComponents" },
|
|
46
|
+
"contracts": { "inventory", "unclaimed" },
|
|
47
|
+
"observations": [ { "id", "capturedAt", "maxAgeDays", "environment" } ],
|
|
48
|
+
"scaffold": { "files", "summary", "rule" }
|
|
49
|
+
}
|
|
50
|
+
```
|
|
51
|
+
|
|
52
|
+
### Classification
|
|
53
|
+
|
|
54
|
+
Every major fact lands in exactly one class:
|
|
55
|
+
|
|
56
|
+
| class | meaning | may be authored? |
|
|
57
|
+
|---|---|---|
|
|
58
|
+
| `CONFIRMED` | two or more independent sources agree | yes |
|
|
59
|
+
| `DERIVED` | deterministically derivable from the repository | yes, once reviewed |
|
|
60
|
+
| `OBSERVED` | read from an authoritative external system | **no** — it belongs in an ObservationSet |
|
|
61
|
+
| `CONFLICT` | sources disagree and no authority rule settles it | **no** — needs a human |
|
|
62
|
+
| `UNRESOLVED` | neither the repository nor an external system answers it | **no** — needs a human |
|
|
63
|
+
|
|
64
|
+
**Uncertainty is never promoted to authored truth.** This is the rule the whole
|
|
65
|
+
report exists to enforce.
|
|
66
|
+
|
|
67
|
+
### Documentation contradictions
|
|
68
|
+
|
|
69
|
+
Two checks run against claims the graph actually owns, and only those. A document
|
|
70
|
+
saying "we use Kafka" is not checkable until a Resource descriptor says what the
|
|
71
|
+
Kafka thing is; a document saying "there is no edge component" is checkable
|
|
72
|
+
immediately, and is `NEGATED_COMPONENT`. A port claim in a document linked to a
|
|
73
|
+
component is `PORT_MISMATCH` when the source binds a different port, and
|
|
74
|
+
`UNLINKED_PORT_CLAIM` when the document is linked to nothing and therefore cannot
|
|
75
|
+
be checked at all.
|
|
76
|
+
|
|
77
|
+
The rule for extending this: only check vocabulary the graph owns. A check that
|
|
78
|
+
guesses is a check that cries wolf.
|
|
79
|
+
|
|
80
|
+
### Reviewed ambiguity
|
|
81
|
+
|
|
82
|
+
The list a human must work through:
|
|
83
|
+
|
|
84
|
+
| kind | what a human decides |
|
|
85
|
+
|---|---|
|
|
86
|
+
| `EVENT_WITHOUT_PRODUCER` | is this subject reserved, or is a producer missing? |
|
|
87
|
+
| `COMPONENT_OWNERSHIP` | which system or domain owns this? |
|
|
88
|
+
| `UNCLAIMED_CONTRACT` | which API owns this contract, or is it dead? |
|
|
89
|
+
| `REVIEWED_OVERRIDE` | is this interpretation still true? |
|
|
90
|
+
| `NEGATED_COMPONENT`, `PORT_MISMATCH`, `UNLINKED_PORT_CLAIM` | which source is right? |
|
|
91
|
+
|
|
92
|
+
## CREATE
|
|
93
|
+
|
|
94
|
+
```bash
|
|
95
|
+
agentdoc init # configuration + doc skeleton
|
|
96
|
+
$EDITOR agentdoc/agentdoc.config.yaml # layout, namespace, adapters
|
|
97
|
+
agentdoc audit # what is provable
|
|
98
|
+
agentdoc scaffold --write
|
|
99
|
+
$EDITOR '**/agentdoc.yaml' # every generated description
|
|
100
|
+
agentdoc validate
|
|
101
|
+
agentdoc compile
|
|
102
|
+
agentdoc check
|
|
103
|
+
```
|
|
104
|
+
|
|
105
|
+
`init` writes `agentdoc/agentdoc.config.yaml`, the four central descriptor
|
|
106
|
+
files, `agentdoc/journeys.yaml`, the documentation hierarchy
|
|
107
|
+
(`docs/PRODUCT.md`, `docs/ARCHITECTURE.md`, `docs/CONSTRAINTS.md`,
|
|
108
|
+
`docs/adr/`), the observations README, and a CI workflow. It never overwrites an
|
|
109
|
+
existing file.
|
|
110
|
+
|
|
111
|
+
`scaffold` is a dry run unless `--write` is passed.
|
|
112
|
+
|
|
113
|
+
### What the scaffolder will and will not do
|
|
114
|
+
|
|
115
|
+
It writes a component descriptor only for a unit it can name from the
|
|
116
|
+
repository — a package name, a module path, or a directory name. It writes a
|
|
117
|
+
description that is a template sentence naming the artifact evidence. It does
|
|
118
|
+
**not** choose a system or a domain, because placement is an ownership decision
|
|
119
|
+
no extractor can make; the generated descriptor carries a commented
|
|
120
|
+
`placementRationale` prompt instead.
|
|
121
|
+
|
|
122
|
+
That is deliberate. A scaffolder that guesses ownership produces a catalog
|
|
123
|
+
nobody trusts.
|
|
124
|
+
|
|
125
|
+
## MIGRATE
|
|
126
|
+
|
|
127
|
+
```bash
|
|
128
|
+
agentdoc init
|
|
129
|
+
# set componentDescriptors to the real layout; enable the right adapters
|
|
130
|
+
agentdoc audit > migration-report.json
|
|
131
|
+
```
|
|
132
|
+
|
|
133
|
+
Then, in order:
|
|
134
|
+
|
|
135
|
+
1. **Source-of-truth mapping.** For each major claim, record which of the four
|
|
136
|
+
inventories supports it. Anything supported by none goes to
|
|
137
|
+
`reviewedAmbiguity`.
|
|
138
|
+
2. **Contradiction detection.** Work `contradictions` and
|
|
139
|
+
`documentationContradictions`. Each one needs a human decision: fix the
|
|
140
|
+
documentation, fix the code, or record an authority rule saying which class
|
|
141
|
+
wins for that key.
|
|
142
|
+
3. **Orphan documentation.** `documentation.orphanDocs`. For each: link it to a
|
|
143
|
+
component via `context.docs`, or delete it. An orphan is a document nothing
|
|
144
|
+
will ever route to.
|
|
145
|
+
4. **Missing documentation.** `documentation.undocumentedComponents` and
|
|
146
|
+
`documentation.unverifiedComponents`. These are the components an agent will
|
|
147
|
+
get wrong.
|
|
148
|
+
5. **Stale contracts.** `contracts.unclaimed`, and any contract whose consumers
|
|
149
|
+
changed. Point the API descriptor at the current file; do not duplicate it.
|
|
150
|
+
6. **Ownership reconstruction.** Placement comes from the code where the code is
|
|
151
|
+
unambiguous (a service that only ever talks to one identity provider), and
|
|
152
|
+
from a human everywhere else.
|
|
153
|
+
7. **Catalog proposal.** `scaffold` lists what it would write.
|
|
154
|
+
8. **Migration.** `scaffold --write`, then review every generated file.
|
|
155
|
+
9. **Validation.** `validate` → `compile` → `check`.
|
|
156
|
+
|
|
157
|
+
### Do not trust the documentation
|
|
158
|
+
|
|
159
|
+
The single most common migration failure is copying existing prose into
|
|
160
|
+
descriptions. The audit exists to find out how much of that prose is still true.
|
|
161
|
+
A description is a one-sentence present-tense statement of current purpose; if
|
|
162
|
+
you cannot write one, the component's purpose is not understood yet, and that is
|
|
163
|
+
worth discovering before documenting it.
|
|
164
|
+
|
|
165
|
+
## Documentation hierarchy
|
|
166
|
+
|
|
167
|
+
| kind | belongs there | must not |
|
|
168
|
+
|---|---|---|
|
|
169
|
+
| `PRODUCT.md` | what it is, for whom, what it does not do | describe components |
|
|
170
|
+
| `ARCHITECTURE.md` | why the shape is what it is | enumerate components or contracts |
|
|
171
|
+
| `CONSTRAINTS.md` | repository-wide rules, one rule one reason one check | restate a component's constraints |
|
|
172
|
+
| `docs/adr/NNNN-*.md` | one decision, its consequences, superseded not edited | be a design document |
|
|
173
|
+
| `<component>/CONSTRAINTS.md` | rules for that component only | duplicate a repository-wide rule |
|
|
174
|
+
| `<component>/README.md` | how to run it, what it is for | restate the catalog |
|
|
175
|
+
| runbooks | operational procedure | contain architectural decisions |
|
|
176
|
+
| `agentdoc/observations/` | what a live system said, with evidence | contain credentials |
|
|
177
|
+
| the graph | topology, contracts, events, checks, conflicts | contain prose |
|
|
178
|
+
|
|
179
|
+
**The duplication rule.** A fact lives in exactly one place. Elsewhere, point at
|
|
180
|
+
it. If `ARCHITECTURE.md` lists the components, it will be wrong within a month,
|
|
181
|
+
and `agentdoc query` will be right. The architecture document explains *why*;
|
|
182
|
+
the graph says *what*.
|
|
183
|
+
|
|
184
|
+
## Finishing
|
|
185
|
+
|
|
186
|
+
```bash
|
|
187
|
+
agentdoc validate && agentdoc check
|
|
188
|
+
agentdoc eval routing
|
|
189
|
+
git add -A && git commit
|
|
190
|
+
```
|
|
191
|
+
|
|
192
|
+
Then install CI (`CI.md`) so the next drift is caught by a gate rather than by a
|
|
193
|
+
review.
|
|
@@ -0,0 +1,133 @@
|
|
|
1
|
+
# PROVENANCE
|
|
2
|
+
|
|
3
|
+
Every important fact in the compiled graph is traceable to evidence. This
|
|
4
|
+
document defines what a provenance record is, how it is produced, and the rules
|
|
5
|
+
that keep evidence free of secrets.
|
|
6
|
+
|
|
7
|
+
## The record
|
|
8
|
+
|
|
9
|
+
```json
|
|
10
|
+
{
|
|
11
|
+
"id": "p-0075",
|
|
12
|
+
"class": "observed",
|
|
13
|
+
"path": "workers/koda-cron/wrangler.toml",
|
|
14
|
+
"extractor": { "name": "platform-manifest-extractor", "version": "1.0.0" },
|
|
15
|
+
"contentHash": "sha256:9f2c...",
|
|
16
|
+
"location": { "lineStart": 5, "lineEnd": 7, "jsonPointer": "/facts/2" },
|
|
17
|
+
"subjects": ["/entities/9/derived/bindings/0", "/conflicts/2"]
|
|
18
|
+
}
|
|
19
|
+
```
|
|
20
|
+
|
|
21
|
+
| field | meaning |
|
|
22
|
+
|---|---|
|
|
23
|
+
| `id` | positional over a canonical sort, so ids are stable for a given input set |
|
|
24
|
+
| `class` | how the evidence was obtained (below) |
|
|
25
|
+
| `path` | repository-relative; never absolute, never outside the checkout |
|
|
26
|
+
| `extractor` | which component read it, and at what version |
|
|
27
|
+
| `contentHash` | `sha256:` of the file bytes as read |
|
|
28
|
+
| `location` | optional: line range or JSON pointer, when the evidence is narrower than the file |
|
|
29
|
+
| `subjects` | RFC 6901 pointers into this graph that the record proves |
|
|
30
|
+
|
|
31
|
+
## Classes
|
|
32
|
+
|
|
33
|
+
| class | meaning | typical extractor |
|
|
34
|
+
|---|---|---|
|
|
35
|
+
| `declared` | the file states the fact outright | `descriptor-reader`, `placement-extractor` |
|
|
36
|
+
| `reviewed` | a human interpretation recorded in configuration, not derivable from a file | `override-extractor` |
|
|
37
|
+
| `observed` | the file shows the fact | `manifest-extractor`, `import-extractor`, `ci-extractor` |
|
|
38
|
+
| `validated` | two independent files agree | `api-contract-extractor`, `oidc-extractor` |
|
|
39
|
+
| `runtime` | read from an external system | `observation:<collector>` |
|
|
40
|
+
| `standard` | the authority is a published standard | `standard-extractor` |
|
|
41
|
+
|
|
42
|
+
`runtime` records are the only ones whose path is not a source file of the
|
|
43
|
+
repository under analysis in the ordinary sense: they point at the committed
|
|
44
|
+
evidence bundle the observation cites.
|
|
45
|
+
|
|
46
|
+
`reviewed` exists so that a human interpretation is not filed as `declared`.
|
|
47
|
+
Filed that way it mapped to `AUTHORED` — the strongest class — so a reviewed
|
|
48
|
+
override corroborated by a source reference was relabelled `AUTHORED` on the
|
|
49
|
+
merged edge, and the graph understated its own highest-authority input. The class
|
|
50
|
+
is not an election either: elections happen per fact in the authority model, and a
|
|
51
|
+
provenance class must not decide one on its own.
|
|
52
|
+
|
|
53
|
+
## Coverage rules
|
|
54
|
+
|
|
55
|
+
- Every relation, external dependency, capability, verification entry, gate,
|
|
56
|
+
journey, contract, conflict and assertion carries at least one
|
|
57
|
+
`provenanceIds` entry.
|
|
58
|
+
- Every entity carries provenance for its authored fields and, for components,
|
|
59
|
+
for its derived block.
|
|
60
|
+
- A relation, index entry or fact with no provenance is a compiler defect. The
|
|
61
|
+
pipeline fails with `AGENTDOC_PROVENANCE` rather than emitting it.
|
|
62
|
+
|
|
63
|
+
## Subject resolution
|
|
64
|
+
|
|
65
|
+
Extractors emit symbolic subjects while they run (`rel:pending`,
|
|
66
|
+
`ass:pending`, `evt:<pattern>`, …) and the compiler binds them to concrete
|
|
67
|
+
pointers once the whole graph exists. This is why an extractor's output does not
|
|
68
|
+
depend on how many other extractors ran first.
|
|
69
|
+
|
|
70
|
+
A subject that never binds is an error: `AGENTDOC_PROVENANCE`. The single
|
|
71
|
+
exception is evidence whose candidate edge or fact was *dropped* (an unresolved
|
|
72
|
+
reviewed override, a rejected candidate). Those records are removed and the
|
|
73
|
+
count is reported as `droppedProvenance` in the compile stats, so a regression
|
|
74
|
+
here is visible rather than silent.
|
|
75
|
+
|
|
76
|
+
## Input hashing
|
|
77
|
+
|
|
78
|
+
`source.inputHash` covers:
|
|
79
|
+
|
|
80
|
+
1. compiler and graph-schema versions, entity api version, authority-rule count,
|
|
81
|
+
every enabled adapter's name and version, and the content hash of every
|
|
82
|
+
shipped schema;
|
|
83
|
+
2. every file the compiler read — path and content, length-prefixed;
|
|
84
|
+
3. every existence probe — path and result;
|
|
85
|
+
4. every directory listing — directory and sorted file list.
|
|
86
|
+
|
|
87
|
+
Every byte the compiler reads flows through a single tracked read path, so the
|
|
88
|
+
record is complete by construction rather than by remembering to register files.
|
|
89
|
+
A change to any input changes the hash; `check` refuses a graph whose hash
|
|
90
|
+
disagrees.
|
|
91
|
+
|
|
92
|
+
## Secrets
|
|
93
|
+
|
|
94
|
+
Two independent layers, because derivation can synthesise a secret-shaped string
|
|
95
|
+
that no source scan would have caught.
|
|
96
|
+
|
|
97
|
+
**Layer 1 — sources.** Every authored or observed document is scanned before it
|
|
98
|
+
can influence any output. Refusal is `AGENTDOC_SECRET`, with the JSON pointer.
|
|
99
|
+
|
|
100
|
+
**Layer 2 — outputs.** The serialized graph, every audit report, every query
|
|
101
|
+
payload and every eval report is scanned again before it is written or printed.
|
|
102
|
+
Refusal is `AGENTDOC_SECRET` and the command exits non-zero. Nothing is written
|
|
103
|
+
partially.
|
|
104
|
+
|
|
105
|
+
Detected shapes include private-key headers, cloud access-key ids, provider
|
|
106
|
+
token prefixes, JWT-shaped triples, URLs with embedded credentials, `Bearer`
|
|
107
|
+
tokens, and `secret: value` / `token = value` style assignments.
|
|
108
|
+
|
|
109
|
+
**Commands are redacted, not dropped.** A verification command is a routing
|
|
110
|
+
index, not a credential transport. An inline assignment whose name contains
|
|
111
|
+
`SECRET`, `TOKEN`, `PASSWORD`, `PASSPHRASE`, `API_KEY`, `PRIVATE_KEY`,
|
|
112
|
+
`CREDENTIAL` or `ACCESS_KEY` keeps its name and loses its value:
|
|
113
|
+
|
|
114
|
+
```
|
|
115
|
+
DATABASE_URL=postgres://u:p@host/db vitest run
|
|
116
|
+
-> DATABASE_URL=[REDACTED] vitest run
|
|
117
|
+
```
|
|
118
|
+
|
|
119
|
+
The redaction marker itself is not treated as a secret, so a redacted command
|
|
120
|
+
remains runnable and printable.
|
|
121
|
+
|
|
122
|
+
**Observations.** An `evidenceBundle` must be a committed, repository-relative
|
|
123
|
+
file. A temp path, a console URL with a session token, or a screenshot filename
|
|
124
|
+
is not evidence. The environment identifier is constrained to a slug; nothing in
|
|
125
|
+
an observation may be a credential.
|
|
126
|
+
|
|
127
|
+
## What is deliberately not provenance
|
|
128
|
+
|
|
129
|
+
- **Confidence.** A heuristic derivation is still traceable; `confidence:
|
|
130
|
+
candidate` says it may not be elected. Provenance and confidence answer
|
|
131
|
+
different questions.
|
|
132
|
+
- **Ownership.** Placement is an authored fact with a `declared` record, not a
|
|
133
|
+
derived one. The extractor identity tells you who read it, not who decided it.
|
package/docs/QUERY.md
ADDED
|
@@ -0,0 +1,180 @@
|
|
|
1
|
+
# QUERY
|
|
2
|
+
|
|
3
|
+
The context router. Its job is **routing**, not documentation concatenation: for
|
|
4
|
+
a path or an entity, return the smallest set of facts an agent needs to work
|
|
5
|
+
safely on that subject, and refuse to answer from a stale graph.
|
|
6
|
+
|
|
7
|
+
## Contract
|
|
8
|
+
|
|
9
|
+
```bash
|
|
10
|
+
agentdoc query <repo-path|entity-ref> [--json] [--md]
|
|
11
|
+
```
|
|
12
|
+
|
|
13
|
+
Resolution order:
|
|
14
|
+
|
|
15
|
+
1. an exact entity ref;
|
|
16
|
+
2. a documentation or constraint file a descriptor links, even outside the
|
|
17
|
+
component's source root;
|
|
18
|
+
3. a directory that is exactly a unit root;
|
|
19
|
+
4. the longest unit root that is a prefix of the path;
|
|
20
|
+
5. a canonical contract file, which routes to the API entity that claims it.
|
|
21
|
+
|
|
22
|
+
If nothing resolves, the command fails with `AGENTDOC_REF_UNRESOLVED` and exits
|
|
23
|
+
non-zero. It never falls back to "the nearest component" or to a global dump.
|
|
24
|
+
|
|
25
|
+
## Freshness refusal
|
|
26
|
+
|
|
27
|
+
`query` re-compiles in memory, reads the committed graph, and refuses it unless
|
|
28
|
+
`inputHash` and `dirty` agree. (The recorded commit is provenance, not a gate —
|
|
29
|
+
see `CI.md`.) Time-dependent gates run first: a
|
|
30
|
+
stale observation is reported as `AGENTDOC_OBSERVATION_STALE` before anything
|
|
31
|
+
else, because "your observation is 5 years old" and "rebuild the graph" send the
|
|
32
|
+
reader to completely different places.
|
|
33
|
+
|
|
34
|
+
There is no silent fallback. `--allow-stale-observations` exists as an explicit
|
|
35
|
+
override and is visible in the command line.
|
|
36
|
+
|
|
37
|
+
## What comes back
|
|
38
|
+
|
|
39
|
+
```json
|
|
40
|
+
{
|
|
41
|
+
"query": { "needle": "...", "resolved": "component:default/teacher" },
|
|
42
|
+
"entity": { "ref", "kind", "name", "description", "spec", "sourcePaths", "derived" },
|
|
43
|
+
"placement": { "systems": [], "domains": [] },
|
|
44
|
+
"relations": { "items": [ { "type", "direction", "other", "evidenceClass", "attributes" } ], "total", "truncated" },
|
|
45
|
+
"neighbours": [ { "ref", "kind", "name", "description" } ],
|
|
46
|
+
"apis": { "provided": [], "consumed": [], "provider", "consumers", "contract" },
|
|
47
|
+
"events": { "items": [], "total", "truncated" },
|
|
48
|
+
"resources": { "used": [], "usedBy": [] },
|
|
49
|
+
"externalDependencies": { "items": [], "total", "truncated" },
|
|
50
|
+
"capabilities": { "items": [], "total", "truncated" },
|
|
51
|
+
"constraints": { "docs": [], "constraints": [], "runbooks": [] },
|
|
52
|
+
"verification": { "items": [ { "id", "tier", "command", "configPaths" } ], "total", "truncated" },
|
|
53
|
+
"journeys": { "items": [], "total", "truncated" },
|
|
54
|
+
"journeyPeers": [ "component:default/..." ],
|
|
55
|
+
"contracts": { "items": [], "total", "truncated" },
|
|
56
|
+
"authority": {
|
|
57
|
+
"conflicts": { "items": [ { "key", "kind", "status", "election", "assertions" } ] },
|
|
58
|
+
"assertions": { "items": [] },
|
|
59
|
+
"observations":[ { "id", "environment", "sourceSystem", "collector", "capturedAt", "facts" } ]
|
|
60
|
+
},
|
|
61
|
+
"provenance": { "items": [] },
|
|
62
|
+
"globalGates": [ { "id", "tier", "command" } ],
|
|
63
|
+
"truncated": [ "relations", "events" ]
|
|
64
|
+
}
|
|
65
|
+
```
|
|
66
|
+
|
|
67
|
+
## Rules that matter
|
|
68
|
+
|
|
69
|
+
**Contracts are pointers, not copies.** `contracts[].ref` is the canonical
|
|
70
|
+
contract; `operations` is the inventory an agent needs to find a handler without
|
|
71
|
+
reading the whole specification.
|
|
72
|
+
|
|
73
|
+
**Critical context is separated from the rest.** `authority.conflicts`,
|
|
74
|
+
`authority.observations` and `constraints.constraints` are the things that cause
|
|
75
|
+
the most damage when missed, and the markdown rendering puts them under explicit
|
|
76
|
+
headings: `CONFLICTS (do not assume one is true)`, `RUNTIME OBSERVATIONS`,
|
|
77
|
+
`CONSTRAINTS`.
|
|
78
|
+
|
|
79
|
+
**Both sides of a conflict are printed.** A conflicted assertion shows its
|
|
80
|
+
evidence class, its confidence and its value. The contradicted value is never
|
|
81
|
+
withheld.
|
|
82
|
+
|
|
83
|
+
**A contract subject carries the provider's and consumers' checks.** Changing an
|
|
84
|
+
endpoint is proved by running their suites, so `verification` for an API subject
|
|
85
|
+
is the union over the provider and its consumers.
|
|
86
|
+
|
|
87
|
+
**Journey peers are named.** If a component appears in a journey, the other
|
|
88
|
+
components in that journey are in scope and are listed explicitly rather than
|
|
89
|
+
left for the agent to rediscover.
|
|
90
|
+
|
|
91
|
+
**Budgets are reported, never silent.** Every capped section returns
|
|
92
|
+
`{ items, total, truncated }`, and the top-level `truncated` lists every section
|
|
93
|
+
that was cut — including the nested `authority.conflicts` and
|
|
94
|
+
`authority.assertions`, reported as `authority.conflicts` and so on. A section is
|
|
95
|
+
never quietly shortened.
|
|
96
|
+
|
|
97
|
+
**Review commitments are surfaced.** `reviewCommitments` collects every conflict
|
|
98
|
+
election, accepted warning and reviewed override with the condition under which
|
|
99
|
+
it must be re-examined. A review condition nobody can see is a review condition
|
|
100
|
+
nobody performs. Where no condition was recorded — a conflict settled with no
|
|
101
|
+
governing authority rule — the entry says so explicitly rather than pointing at a
|
|
102
|
+
record that does not exist.
|
|
103
|
+
|
|
104
|
+
**Budgets are adjustable.** The defaults are deliberately generous, because a
|
|
105
|
+
one-hop route returns a handful of relations and a tight cap would truncate
|
|
106
|
+
everything. When an agent is working in a constrained context it can say so:
|
|
107
|
+
|
|
108
|
+
```bash
|
|
109
|
+
agentdoc query component:default/api --budget relations=10,verification=4
|
|
110
|
+
```
|
|
111
|
+
|
|
112
|
+
Limits are positive integers. Every capped section then reports
|
|
113
|
+
`{ items, total, truncated }` and the top-level `truncated` names each one, so
|
|
114
|
+
the slice is never mistaken for the whole.
|
|
115
|
+
|
|
116
|
+
## Default budgets
|
|
117
|
+
|
|
118
|
+
| section | cap |
|
|
119
|
+
|---|---|
|
|
120
|
+
| relations | 40 |
|
|
121
|
+
| events | 20 |
|
|
122
|
+
| externalDependencies | 15 |
|
|
123
|
+
| capabilities | 10 |
|
|
124
|
+
| verification | 12 |
|
|
125
|
+
| journeys | 10 |
|
|
126
|
+
| conflicts | 20 |
|
|
127
|
+
| assertions | 25 |
|
|
128
|
+
| provenance | 20 |
|
|
129
|
+
| contracts | 10 |
|
|
130
|
+
|
|
131
|
+
## Markdown rendering
|
|
132
|
+
|
|
133
|
+
`--md` (the default) produces a compact block suitable for a prompt or a human:
|
|
134
|
+
|
|
135
|
+
```
|
|
136
|
+
# component:default/koda-cron (Component)
|
|
137
|
+
Triggers governed endpoints on a platform schedule and owns no business logic.
|
|
138
|
+
- source: workers/koda-cron
|
|
139
|
+
- system: system:default/scheduler
|
|
140
|
+
- CONSTRAINTS: workers/koda-cron/CONSTRAINTS.md
|
|
141
|
+
- verify:
|
|
142
|
+
- [unit] vitest run
|
|
143
|
+
- CONFLICTS (do not assume one is true):
|
|
144
|
+
- schedule.cron [contradiction/resolved]
|
|
145
|
+
- DERIVED/deterministic: "0 3 * * *"
|
|
146
|
+
- OBSERVED_RUNTIME/direct: "0 * * * *,3 0 * * *"
|
|
147
|
+
- RUNTIME OBSERVATIONS:
|
|
148
|
+
- production via platform-scheduler at 2026-09-27T18:40:00Z (evidence …)
|
|
149
|
+
```
|
|
150
|
+
|
|
151
|
+
## Change impact
|
|
152
|
+
|
|
153
|
+
```bash
|
|
154
|
+
agentdoc impact <paths...>
|
|
155
|
+
agentdoc impact --diff <git-ref>
|
|
156
|
+
```
|
|
157
|
+
|
|
158
|
+
Impact answers a question about the working tree, so it compiles in memory and
|
|
159
|
+
does not require a fresh committed graph. Using it before rebuilding is the
|
|
160
|
+
point.
|
|
161
|
+
|
|
162
|
+
It reports: affected components with the reason each was reached, affected
|
|
163
|
+
contracts, documentation, resources, verification and catalog sources; whether
|
|
164
|
+
the graph will be invalidated; the recommended verification commands; any open
|
|
165
|
+
conflicts on the affected subjects; and related journeys.
|
|
166
|
+
|
|
167
|
+
Reachability rules: a contract change reaches its provider and every consumer; a
|
|
168
|
+
component change reaches its transitive dependents two hops out, and everything
|
|
169
|
+
connected to it by a runtime call or a schedule. Two hops is deliberate — a
|
|
170
|
+
third hop is where the graph stops being a routing aid and becomes a dump.
|
|
171
|
+
|
|
172
|
+
## Using it in a prompt
|
|
173
|
+
|
|
174
|
+
Ask for `--md` and give the output to the agent as-is. Two rules keep it
|
|
175
|
+
honest:
|
|
176
|
+
|
|
177
|
+
1. Do not summarise a conflict away. If the context contains a `CONFLICTS`
|
|
178
|
+
block, the agent must be told about it.
|
|
179
|
+
2. If the context contains `UNRESOLVED` or a conflict with `basis: none`, the
|
|
180
|
+
correct agent behaviour is to say so and stop, not to choose a side.
|