agent-doc-system 0.0.0-stage → 1.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/INSTALL_FOR_AGENTS.md +60 -0
- package/LICENSE +21 -0
- package/README.md +151 -2
- package/adapters/cache.mjs +27 -0
- package/adapters/contract-consumers.mjs +56 -0
- package/adapters/database.mjs +116 -0
- package/adapters/deploy-config.mjs +34 -0
- package/adapters/dockerfile.mjs +26 -0
- package/adapters/events.mjs +145 -0
- package/adapters/generic.mjs +86 -0
- package/adapters/github-actions.mjs +106 -0
- package/adapters/go.mjs +91 -0
- package/adapters/graphql.mjs +28 -0
- package/adapters/index.mjs +55 -0
- package/adapters/node.mjs +118 -0
- package/adapters/object-store.mjs +64 -0
- package/adapters/oidc.mjs +56 -0
- package/adapters/openapi.mjs +50 -0
- package/adapters/pnpm.mjs +42 -0
- package/adapters/protobuf.mjs +35 -0
- package/adapters/registry.mjs +27 -0
- package/adapters/resource-client.mjs +61 -0
- package/adapters/turborepo.mjs +135 -0
- package/adapters/typescript.mjs +85 -0
- package/adapters/wrangler.mjs +206 -0
- package/bin/agentdoc.mjs +478 -0
- package/bin/usage.txt +26 -0
- package/core/acceptances.mjs +69 -0
- package/core/audit.mjs +407 -0
- package/core/authority/classes.mjs +39 -0
- package/core/authority/engine.mjs +243 -0
- package/core/authority/facts.mjs +41 -0
- package/core/codes.mjs +119 -0
- package/core/compile.mjs +881 -0
- package/core/contracts.mjs +121 -0
- package/core/derive.mjs +102 -0
- package/core/descriptors.mjs +486 -0
- package/core/determinism.mjs +95 -0
- package/core/discover.mjs +147 -0
- package/core/facts.mjs +53 -0
- package/core/fsx.mjs +342 -0
- package/core/graph.mjs +260 -0
- package/core/impact.mjs +175 -0
- package/core/indexes.mjs +165 -0
- package/core/jsonschema.mjs +298 -0
- package/core/observations.mjs +129 -0
- package/core/provenance.mjs +83 -0
- package/core/query.mjs +384 -0
- package/core/relations.mjs +286 -0
- package/core/scaffold.mjs +322 -0
- package/core/secrets.mjs +92 -0
- package/core/sourcescan.mjs +88 -0
- package/core/toml.mjs +163 -0
- package/core/yaml.mjs +475 -0
- package/docs/ADAPTERS.md +172 -0
- package/docs/AUTHORITY.md +260 -0
- package/docs/CI.md +136 -0
- package/docs/COMPARISON-KODA.md +164 -0
- package/docs/EVALS.md +166 -0
- package/docs/MIGRATION.md +193 -0
- package/docs/PROVENANCE.md +133 -0
- package/docs/QUERY.md +180 -0
- package/docs/SPEC.md +339 -0
- package/docs/VERIFICATION.md +28 -0
- package/docs/ci/generic-ci.sh +28 -0
- package/docs/ci/github-actions.yml +35 -0
- package/evals/harness.mjs +426 -0
- package/evals/scenarios/fixture-a-01-change-api.json +9 -0
- package/evals/scenarios/fixture-a-02-change-db.json +9 -0
- package/evals/scenarios/fixture-a-03-enforce-constraint.json +9 -0
- package/evals/scenarios/fixture-b-01-change-proto.json +9 -0
- package/evals/scenarios/fixture-b-02-change-write-path.json +9 -0
- package/evals/scenarios/fixture-b-03-change-shared-lib.json +9 -0
- package/evals/scenarios/fixture-b-04-journey.json +9 -0
- package/evals/scenarios/fixture-c-01-mixed-runtime-edge.json +19 -0
- package/evals/scenarios/fixture-c-02-stale-doc-contradiction.json +9 -0
- package/evals/scenarios/fixture-c-03-shared-vocabulary.json +9 -0
- package/evals/scenarios/fixture-c-04-contract-change.json +9 -0
- package/evals/scenarios/koda-01-modify-api-consumer.json +26 -0
- package/evals/scenarios/koda-02-change-db-schema.json +20 -0
- package/evals/scenarios/koda-03-change-auth.json +21 -0
- package/evals/scenarios/koda-04-add-event-producer.json +19 -0
- package/evals/scenarios/koda-05-modify-worker-binding.json +25 -0
- package/evals/scenarios/koda-06-change-cross-component-endpoint.json +22 -0
- package/evals/scenarios/koda-07-change-shared-package.json +16 -0
- package/evals/scenarios/koda-08-change-runtime-cron.json +23 -0
- package/evals/scenarios/koda-09-debug-production-mismatch.json +26 -0
- package/evals/thresholds.json +11 -0
- package/package.json +46 -4
- package/schemas/api.schema.json +84 -0
- package/schemas/common.schema.json +107 -0
- package/schemas/component.schema.json +40 -0
- package/schemas/config.schema.json +685 -0
- package/schemas/domain.schema.json +14 -0
- package/schemas/graph.schema.json +1084 -0
- package/schemas/journeys.schema.json +47 -0
- package/schemas/observation.schema.json +143 -0
- package/schemas/resource.schema.json +28 -0
- package/schemas/system.schema.json +26 -0
- package/skills/agent-doc-system/SKILL.md +113 -0
- package/skills/agent-doc-system/references/authority-and-conflicts.md +70 -0
- package/skills/agent-doc-system/references/cli.md +89 -0
- package/skills/agent-doc-system/references/create-migrate.md +101 -0
- package/skills/agent-doc-system/references/maintenance.md +52 -0
- package/templates/ADR.md +26 -0
- package/templates/ARCHITECTURE.md +19 -0
- package/templates/CONSTRAINTS.md +17 -0
- package/templates/PRODUCT.md +17 -0
- package/templates/api.yaml +10 -0
- package/templates/ci.yml +31 -0
- package/templates/component.yaml +14 -0
- package/templates/domain.yaml +6 -0
- package/templates/journey.yaml +7 -0
- package/templates/observation.yaml +19 -0
- package/templates/resource.yaml +8 -0
- package/templates/system.yaml +10 -0
- package/templates/warning-acceptance.yaml +14 -0
|
@@ -0,0 +1,260 @@
|
|
|
1
|
+
# AUTHORITY
|
|
2
|
+
|
|
3
|
+
The model that distinguishes what the repository claims, what can be derived
|
|
4
|
+
from it, and what was actually observed running.
|
|
5
|
+
|
|
6
|
+
## Why this exists
|
|
7
|
+
|
|
8
|
+
A catalog that merges these three produces confident, confident-looking wrong
|
|
9
|
+
answers. Every one of the following is a real failure mode this model is built
|
|
10
|
+
to make impossible to hide:
|
|
11
|
+
|
|
12
|
+
- the presumed production binding source is not the real authority;
|
|
13
|
+
- a binding assumed to be missing is actually four;
|
|
14
|
+
- a schedule described as daily is an hourly trigger with hour gating;
|
|
15
|
+
- an external upload API is assumed to accept a field the deployed API rejects;
|
|
16
|
+
- recovered runtime state intentionally differs from the governed target state.
|
|
17
|
+
|
|
18
|
+
Each of these is, mechanically, the same event: **two evidence classes made
|
|
19
|
+
claims about the same fact, and the catalog quietly picked one.**
|
|
20
|
+
|
|
21
|
+
## Evidence classes
|
|
22
|
+
|
|
23
|
+
### `AUTHORED`
|
|
24
|
+
|
|
25
|
+
Explicitly declared and intentionally maintained in this repository: catalog
|
|
26
|
+
descriptors, architecture documents, ADRs, canonical contracts, configuration
|
|
27
|
+
manifests, declared ownership.
|
|
28
|
+
|
|
29
|
+
- **Strengths:** states intent; reviewed; versioned with the decision.
|
|
30
|
+
- **Failure modes:** drifts; states what *should* be, not what *is*; a config
|
|
31
|
+
file is a claim about deployment, not a deployment.
|
|
32
|
+
- **Never sufficient** for a claim about what is currently running.
|
|
33
|
+
|
|
34
|
+
### `DERIVED`
|
|
35
|
+
|
|
36
|
+
Facts deterministically extracted from committed inputs: package dependencies,
|
|
37
|
+
imports, routes, runtime bindings, CI checks, service calls, events, database
|
|
38
|
+
usage, artifact shape.
|
|
39
|
+
|
|
40
|
+
- **Strengths:** reproducible; anyone can re-derive it; cannot silently rot.
|
|
41
|
+
- **Failure modes:** heuristic; incomplete; a consumer of a contract that goes
|
|
42
|
+
through a generated client may not name the contract file; absence of evidence
|
|
43
|
+
is not evidence of absence.
|
|
44
|
+
- **A derivation whose result is ambiguous must degrade to a diagnostic, not to
|
|
45
|
+
a fact.** An ambiguous bucket literal produces `AGENTDOC_BINDING_AMBIGUOUS`,
|
|
46
|
+
not a guessed resource edge.
|
|
47
|
+
|
|
48
|
+
### `OBSERVED_RUNTIME`
|
|
49
|
+
|
|
50
|
+
Facts read from an authoritative external or running system: deployed
|
|
51
|
+
configuration, real scheduler state, live bindings, cloud provider capabilities,
|
|
52
|
+
the actual production version, external API behaviour.
|
|
53
|
+
|
|
54
|
+
Every observation carries:
|
|
55
|
+
|
|
56
|
+
```
|
|
57
|
+
source the authority system, e.g. platform-scheduler
|
|
58
|
+
timestamp when it was read, ISO-8601 UTC
|
|
59
|
+
environment which environment it describes
|
|
60
|
+
retrieval method the command or API used
|
|
61
|
+
evidence a durable, committed, secret-free reference
|
|
62
|
+
freshness maxAgeDays, a promise the observation must keep
|
|
63
|
+
supersedes the observation set it replaces, if any
|
|
64
|
+
```
|
|
65
|
+
|
|
66
|
+
Observations are committed to the repository. That is deliberate: an observation
|
|
67
|
+
nobody can reproduce is an anecdote, and an anecdote does not belong in a graph
|
|
68
|
+
that claims to describe reality.
|
|
69
|
+
|
|
70
|
+
**An observation can never overwrite an authored fact.** It is added to the same
|
|
71
|
+
`(subject, key)` group, and the authority engine decides what happens. Without a
|
|
72
|
+
rule, the disagreement is an error.
|
|
73
|
+
|
|
74
|
+
### `EXTERNAL_STANDARD`
|
|
75
|
+
|
|
76
|
+
Standards whose canonical authority is outside the repository: OIDC discovery,
|
|
77
|
+
OpenAPI, protocol specifications, provider API schemas.
|
|
78
|
+
|
|
79
|
+
- The desired wire surface is `AUTHORED` (the contract file) or
|
|
80
|
+
`EXTERNAL_STANDARD` (the specification).
|
|
81
|
+
- What a live deployment accepts is `OBSERVED_RUNTIME`.
|
|
82
|
+
- These routinely disagree, and that disagreement is the point.
|
|
83
|
+
|
|
84
|
+
### `REVIEWED_OVERRIDE`
|
|
85
|
+
|
|
86
|
+
A human-approved interpretation, used when deterministic discovery cannot
|
|
87
|
+
resolve a fact. It is bounded and must be justified:
|
|
88
|
+
|
|
89
|
+
```
|
|
90
|
+
fact which kind of fact is being asserted
|
|
91
|
+
subject the entity ref
|
|
92
|
+
target the entity ref, where applicable
|
|
93
|
+
reason why this is true, in one sentence
|
|
94
|
+
evidence file paths under a mapped component's source root, or the contract
|
|
95
|
+
```
|
|
96
|
+
|
|
97
|
+
Evidence must live under a mapped component's source root or be the declared
|
|
98
|
+
contract. A reviewed override therefore cannot assert something no file in the
|
|
99
|
+
repository can corroborate. Overrides additionally produce a `REVIEWED_OVERRIDE`
|
|
100
|
+
fact, so the interpretation is a first-class citizen of the conflict machinery
|
|
101
|
+
rather than a bypass.
|
|
102
|
+
|
|
103
|
+
### `UNRESOLVED`
|
|
104
|
+
|
|
105
|
+
Known ambiguity. It must not become truth, must not be silently dropped, and
|
|
106
|
+
must not be resolved by picking a plausible answer.
|
|
107
|
+
|
|
108
|
+
Two sources of unresolved facts:
|
|
109
|
+
|
|
110
|
+
1. **Contradiction with no governing rule.** An error.
|
|
111
|
+
2. **Only heuristic candidates.** A warning; the candidates never become elected
|
|
112
|
+
facts. Migration authority alone does not establish database ownership; a
|
|
113
|
+
library that ships migrations but owns no pool gets a warning and a candidate,
|
|
114
|
+
not an invented `usesResource` edge.
|
|
115
|
+
|
|
116
|
+
## Confidence
|
|
117
|
+
|
|
118
|
+
A separate axis from evidence class, recording *how* the claim was obtained:
|
|
119
|
+
|
|
120
|
+
| confidence | meaning |
|
|
121
|
+
|---|---|
|
|
122
|
+
| `direct` | read straight from the named authority system |
|
|
123
|
+
| `deterministic` | a reproducible function of committed inputs |
|
|
124
|
+
| `declared` | asserted by a human-maintained document |
|
|
125
|
+
| `reviewed` | a reviewer signed off |
|
|
126
|
+
| `candidate` | heuristic; never elected |
|
|
127
|
+
|
|
128
|
+
`candidate` is enforced mechanically, not by convention: an assertion whose only
|
|
129
|
+
evidence is a candidate stays `status: candidate`, and the group is reported as
|
|
130
|
+
an ambiguity.
|
|
131
|
+
|
|
132
|
+
## Authority rules
|
|
133
|
+
|
|
134
|
+
A rule says which evidence class is authoritative **for one key**, for a
|
|
135
|
+
subject matching a pattern.
|
|
136
|
+
|
|
137
|
+
```yaml
|
|
138
|
+
authority:
|
|
139
|
+
rules:
|
|
140
|
+
- id: deployed-schedule-is-runtime-truth
|
|
141
|
+
keyPattern: " schedule\\.(cron|enabled|target)$"
|
|
142
|
+
elect: [OBSERVED_RUNTIME]
|
|
143
|
+
rationale: "The committed manifest states intent; only the live scheduler states what fires."
|
|
144
|
+
reviewWhen: "Review when the scheduler configuration changes."
|
|
145
|
+
degradeOnStale: conflict
|
|
146
|
+
```
|
|
147
|
+
|
|
148
|
+
- `keyPattern` is matched against `"<subject> <key>"`; the most specific match
|
|
149
|
+
(longest pattern) wins.
|
|
150
|
+
- `elect` may list one to three evidence classes. Naming `UNRESOLVED` is
|
|
151
|
+
rejected at load: electing nothing elects nothing.
|
|
152
|
+
- `rationale` is mandatory. A rule nobody can justify is a rule nobody can
|
|
153
|
+
review.
|
|
154
|
+
- `reviewWhen` is mandatory.
|
|
155
|
+
|
|
156
|
+
**There is no default global precedence.** A fact with disagreeing evidence and
|
|
157
|
+
no matching rule is an error. This is the single most important property of the
|
|
158
|
+
model: silence is not a resolution.
|
|
159
|
+
|
|
160
|
+
## Election
|
|
161
|
+
|
|
162
|
+
For a group of assertions on one `(subject, key)`:
|
|
163
|
+
|
|
164
|
+
1. **Agreement** — all values equal. Every assertion is `elected`. Agreement
|
|
165
|
+
across evidence classes is corroboration, and it is visible as such. If the
|
|
166
|
+
only evidence is a heuristic candidate, the group stays unresolved.
|
|
167
|
+
2. **Contradiction** — values differ. The basis is attempted in order:
|
|
168
|
+
- `reviewed-override` — a rule elects `REVIEWED_OVERRIDE` and exactly one
|
|
169
|
+
exists. Status `accepted`. The divergence is still recorded.
|
|
170
|
+
- `authority-rule` — the rule's elect list intersects the present classes in
|
|
171
|
+
exactly one class with exactly one assertion.
|
|
172
|
+
- `superset` — one candidate's value strictly contains every other (for
|
|
173
|
+
example an observed set that is a strict subset of a declared set). The
|
|
174
|
+
superset is elected; the containment is stated.
|
|
175
|
+
- `recency` — only when the rule names `OBSERVED_RUNTIME` and more than one
|
|
176
|
+
dated observation exists. The newest wins, and the rule is cited.
|
|
177
|
+
- otherwise `basis: none`, status `unresolved`, and an **error**.
|
|
178
|
+
3. Every non-elected assertion is kept with `status: contradicted` (or
|
|
179
|
+
`candidate`). **The contradicted value stays in the graph.** A system that
|
|
180
|
+
discards the repository's own claim cannot explain the divergence to a human.
|
|
181
|
+
|
|
182
|
+
## Conflict kinds
|
|
183
|
+
|
|
184
|
+
| kind | meaning |
|
|
185
|
+
|---|---|
|
|
186
|
+
| `contradiction` | two evidence classes disagree about the same value |
|
|
187
|
+
| `staleness` | two runtime observations disagree; one is out of date |
|
|
188
|
+
| `supersession` | a newer observation explicitly replaces an older one |
|
|
189
|
+
| `ambiguity` | only heuristic candidates, or an ungoverned disagreement |
|
|
190
|
+
| `reviewed-divergence` | a reviewer approved the divergence and it is recorded |
|
|
191
|
+
|
|
192
|
+
Status is `unresolved`, `resolved` or `accepted`.
|
|
193
|
+
|
|
194
|
+
## Staleness
|
|
195
|
+
|
|
196
|
+
An observation is stale when `now - capturedAt > maxAgeDays`. Staleness is
|
|
197
|
+
evaluated at gate time, never at compile time, because a compile must stay a
|
|
198
|
+
pure function of the checkout.
|
|
199
|
+
|
|
200
|
+
- `check` and `query` fail closed on a stale observation and name it.
|
|
201
|
+
- `--allow-stale-observations` exists as an explicit, visible override.
|
|
202
|
+
- `degradeOnStale` lets a rule say what a stale election means: keep the
|
|
203
|
+
election but raise a staleness conflict (`conflict`), or drop to unresolved
|
|
204
|
+
(`unresolved`).
|
|
205
|
+
|
|
206
|
+
## Worked example
|
|
207
|
+
|
|
208
|
+
The committed manifest declares:
|
|
209
|
+
|
|
210
|
+
```toml
|
|
211
|
+
[triggers]
|
|
212
|
+
crons = ["0 3 * * *"]
|
|
213
|
+
```
|
|
214
|
+
|
|
215
|
+
The platform reports:
|
|
216
|
+
|
|
217
|
+
```json
|
|
218
|
+
{ "triggers": [ { "cron": "0 * * * *", "gate": { "hourEquals": 3 } },
|
|
219
|
+
{ "cron": "3 0 * * *", "enabled": true } ] }
|
|
220
|
+
```
|
|
221
|
+
|
|
222
|
+
The observation says, of `schedule.cron`:
|
|
223
|
+
|
|
224
|
+
> The scheduler evaluates an hourly trigger plus a daily trigger; the hourly
|
|
225
|
+
> trigger is gated to hour 3, so the effective daily behaviour is not the
|
|
226
|
+
> manifest's single daily expression.
|
|
227
|
+
|
|
228
|
+
The graph then contains:
|
|
229
|
+
|
|
230
|
+
```
|
|
231
|
+
conflict component:default/worker schedule.cron contradiction resolved
|
|
232
|
+
assertion AUTHORED/DERIVED "0 3 * * *" contradicted
|
|
233
|
+
assertion OBSERVED_RUNTIME "0 * * * *,3 0 * * *" elected
|
|
234
|
+
election authority-rule / deployed-schedule-is-runtime-truth
|
|
235
|
+
```
|
|
236
|
+
|
|
237
|
+
A query for that component prints both values, marks the election, and points at
|
|
238
|
+
the evidence. An agent debugging "the nightly job did not run" sees immediately
|
|
239
|
+
that the repository's description of the schedule is not what the scheduler
|
|
240
|
+
does — which is the finding. Silently preferring either side would have hidden
|
|
241
|
+
it.
|
|
242
|
+
|
|
243
|
+
## The one place a label is chosen without an election is a *relation's*
|
|
244
|
+
`evidenceClass`, and it elects nothing: when its provenance records agree there is
|
|
245
|
+
one class and it is reported; when they disagree the edge is reported as
|
|
246
|
+
`REVIEWED_OVERRIDE` if a human interpretation is among them, and `DERIVED`
|
|
247
|
+
otherwise. Picking by strength would be a global precedence order in a helper, and
|
|
248
|
+
it was actively wrong — a reviewed override corroborated by a source reference was
|
|
249
|
+
relabelled `AUTHORED`, so the graph understated its own highest-authority input
|
|
250
|
+
while the corresponding assertion still read `REVIEWED_OVERRIDE`.
|
|
251
|
+
|
|
252
|
+
Anti-patterns this model forbids
|
|
253
|
+
|
|
254
|
+
- A global precedence order (`AUTHORED` always beats `DERIVED`).
|
|
255
|
+
- Overwriting an authored value with an observed one because the observed one is
|
|
256
|
+
newer, with no rule.
|
|
257
|
+
- Treating a configuration file as evidence of deployment.
|
|
258
|
+
- Accepting a warning to silence a conflict.
|
|
259
|
+
- Deleting the contradicted assertion after an election.
|
|
260
|
+
- Promoting a heuristic candidate because it is the only candidate.
|
package/docs/CI.md
ADDED
|
@@ -0,0 +1,136 @@
|
|
|
1
|
+
# CI
|
|
2
|
+
|
|
3
|
+
The graph is a build artifact. The gate proves the committed one is identical to
|
|
4
|
+
a fresh compile, so drift fails in CI rather than in a review three weeks later.
|
|
5
|
+
|
|
6
|
+
## The gate
|
|
7
|
+
|
|
8
|
+
```yaml
|
|
9
|
+
name: agentdoc
|
|
10
|
+
on:
|
|
11
|
+
pull_request:
|
|
12
|
+
push:
|
|
13
|
+
branches: [main]
|
|
14
|
+
jobs:
|
|
15
|
+
agentdoc:
|
|
16
|
+
runs-on: ubuntu-latest
|
|
17
|
+
timeout-minutes: 5
|
|
18
|
+
steps:
|
|
19
|
+
- uses: actions/checkout@v4
|
|
20
|
+
with: { fetch-depth: 0 }
|
|
21
|
+
- uses: actions/setup-node@v4
|
|
22
|
+
with: { node-version: '24' }
|
|
23
|
+
# Pin a tag for reproducibility, e.g. github:lbframe/agent-doc-system#v1.0.0 —
|
|
24
|
+
# a compiler upgrade intentionally invalidates every committed graph.
|
|
25
|
+
- run: npm install -g github:lbframe/agent-doc-system
|
|
26
|
+
- run: agentdoc validate
|
|
27
|
+
- run: agentdoc compile
|
|
28
|
+
- run: agentdoc check
|
|
29
|
+
- run: agentdoc check --require-clean
|
|
30
|
+
- run: agentdoc eval routing
|
|
31
|
+
```
|
|
32
|
+
|
|
33
|
+
`agentdoc init` writes this workflow to `.github/workflows/agentdoc.yml` (from
|
|
34
|
+
`templates/ci.yml` in the CLI package). The same file is kept at
|
|
35
|
+
`docs/ci/github-actions.yml` for reference.
|
|
36
|
+
|
|
37
|
+
`fetch-depth: 0` is recommended: `--diff <ref>` needs history, and `dirty` is
|
|
38
|
+
more meaningful against a real `HEAD`.
|
|
39
|
+
|
|
40
|
+
**What freshness actually checks.** The input hash (every byte the compiler
|
|
41
|
+
read, plus every existence probe, directory listing and walk) and the dirty flag
|
|
42
|
+
(whether a compiler input differs from `HEAD`). It deliberately does **not**
|
|
43
|
+
compare the graph's recorded commit with `HEAD`: committing the graph changes
|
|
44
|
+
`HEAD`, so a commit-equality gate could never be satisfied by a committed graph.
|
|
45
|
+
The commit is recorded as provenance — which checkout produced this graph — not
|
|
46
|
+
as a gate.
|
|
47
|
+
|
|
48
|
+
Two things to decide per repository:
|
|
49
|
+
|
|
50
|
+
- **Commit the graph or ignore it.** Both work. Committing it makes the
|
|
51
|
+
architecture reviewable in the pull request, which is valuable in a mature
|
|
52
|
+
repository. Ignoring it keeps diffs small. The gate works either way, because
|
|
53
|
+
the workflow compiles before checking.
|
|
54
|
+
- **Run the routing evaluation or not.** `agentdoc eval routing` skips cleanly
|
|
55
|
+
on a repository with no registered scenarios, so it is safe to run
|
|
56
|
+
everywhere. The aggregate gate over every corpus (`node evals/run-all.mjs`)
|
|
57
|
+
is development tooling — it belongs to this repository's own CI, not to a
|
|
58
|
+
project that merely uses the catalog.
|
|
59
|
+
|
|
60
|
+
## Failure modes worth knowing
|
|
61
|
+
|
|
62
|
+
The full list is `core/codes.mjs`; these are the ones an operator meets. A
|
|
63
|
+
YAML or schema error names its file and line and needs no table.
|
|
64
|
+
|
|
65
|
+
|
|
66
|
+
| code | what happened | what to do |
|
|
67
|
+
|---|---|---|
|
|
68
|
+
| `AGENTDOC_STALE_GRAPH` | committed bytes differ from a fresh compile | `compile`, review the graph diff, commit |
|
|
69
|
+
| `AGENTDOC_GRAPH_FRESHNESS` | the input hash or the dirty flag disagrees | rebuild; if the input changed, review the change first |
|
|
70
|
+
| `AGENTDOC_COMPILER_MISMATCH` | graph built by another compiler version | `compile` and commit; the diff is the upgrade's real effect |
|
|
71
|
+
| `AGENTDOC_GRAPH_SCHEMA` | graph schema changed | `compile` and commit |
|
|
72
|
+
| `AGENTDOC_GRAPH_MISSING` | no graph at the configured path | `compile` |
|
|
73
|
+
| `AGENTDOC_DIRTY_SOURCE` | a catalog input differs from `HEAD` | commit the input, then recompile |
|
|
74
|
+
| `AGENTDOC_OBSERVATION_STALE` | an observation passed `maxAgeDays` | re-observe, or restate the fact as authored intent if the divergence is intended |
|
|
75
|
+
| `AGENTDOC_CONFLICT_UNRESOLVED` | two classes disagree and no rule governs the key | add an authority rule, or ask a human |
|
|
76
|
+
| `AGENTDOC_WARNING_ACCEPTANCE` | an acceptance no longer matches, or its evidence moved | review the disposition, then update the digests |
|
|
77
|
+
| `AGENTDOC_SECRET` | secret-shaped input or output | remove the credential; never redact-and-continue |
|
|
78
|
+
| `AGENTDOC_NONDETERMINISTIC` | a host path or unsourced timestamp reached the graph | file a defect; this is a compiler bug |
|
|
79
|
+
| `AGENTDOC_CONTRACT_UNCLAIMED` | a contract file no API claims | point an API at it, or delete it |
|
|
80
|
+
| `AGENTDOC_COMPONENT_UNCORROBORATED` | a descriptor no adapter can corroborate | fix the location, or enable the right adapter |
|
|
81
|
+
| `AGENTDOC_BINDING_AMBIGUOUS` | a client matches several Resources of one type | declare which, or review and add an override |
|
|
82
|
+
| `AGENTDOC_CONTRACT_SYNTAX` | a contract file is not valid for its declared type | fix the contract; do not change its type |
|
|
83
|
+
| `AGENTDOC_OIDC_DISCOVERY` | the provider does not expose its discovery path | fix the provider or the descriptor |
|
|
84
|
+
| `AGENTDOC_DUPLICATE_IDENTITY` | two entities claim the same kind and name | rename one |
|
|
85
|
+
| `AGENTDOC_COMPILER_MISMATCH` | the compiler was upgraded | `compile` and commit |
|
|
86
|
+
| `AGENTDOC_OVERRIDE_STALE` | a reviewed override's cited evidence changed | review the interpretation, then `pin-evidence` and update the digest |
|
|
87
|
+
| `AGENTDOC_VERIFICATION_UNTIERED` | a check task has no recognisable tier | name the task, or extend the adapter |
|
|
88
|
+
|
|
89
|
+
## Ordering inside `check`
|
|
90
|
+
|
|
91
|
+
1. compilation errors;
|
|
92
|
+
2. time-dependent gates (observation staleness, election degradation);
|
|
93
|
+
3. graph freshness (`--require-clean`, input hash, compiler/schema versions);
|
|
94
|
+
4. byte-for-byte comparison.
|
|
95
|
+
|
|
96
|
+
Step 2 precedes step 3 on purpose. "Your observation is five years old" and
|
|
97
|
+
"rebuild the graph" send the reader to completely different places, and when
|
|
98
|
+
both are true the first is the one that matters.
|
|
99
|
+
|
|
100
|
+
## Merge discipline
|
|
101
|
+
|
|
102
|
+
- A PR that changes a compiler input must include the recompiled graph. `impact`
|
|
103
|
+
tells you which files are inputs.
|
|
104
|
+
- A PR that changes an authority rule must explain why that class is
|
|
105
|
+
authoritative for that key. `rationale` and `reviewWhen` are mandatory, and a
|
|
106
|
+
reviewer should read them.
|
|
107
|
+
- A PR that adds a warning acceptance is a reviewed decision. The evidence
|
|
108
|
+
digests are pinned; if the evidence later moves, the acceptance stops
|
|
109
|
+
matching and forces a new review. That is the mechanism working, not failing.
|
|
110
|
+
- Never hand-edit `.agentdoc/graph.json`. The gate rejects it, and it would be
|
|
111
|
+
wrong anyway.
|
|
112
|
+
|
|
113
|
+
## Local parity
|
|
114
|
+
|
|
115
|
+
```bash
|
|
116
|
+
agentdoc validate
|
|
117
|
+
agentdoc compile
|
|
118
|
+
agentdoc check --require-clean
|
|
119
|
+
```
|
|
120
|
+
|
|
121
|
+
The same commands, the same codes, the same exit status. There is no CI-only
|
|
122
|
+
behaviour.
|
|
123
|
+
|
|
124
|
+
## Other CI systems
|
|
125
|
+
|
|
126
|
+
The gate is four commands and an exit code. Any runner works:
|
|
127
|
+
|
|
128
|
+
```sh
|
|
129
|
+
set -e
|
|
130
|
+
agentdoc validate
|
|
131
|
+
agentdoc compile
|
|
132
|
+
agentdoc check --require-clean
|
|
133
|
+
```
|
|
134
|
+
|
|
135
|
+
`ci/generic-ci.sh` (next to this document) is exactly that, plus a clear
|
|
136
|
+
failure when the `agentdoc` CLI is not installed on the runner.
|
|
@@ -0,0 +1,164 @@
|
|
|
1
|
+
# Koda comparison
|
|
2
|
+
|
|
3
|
+
Reference: the Koda production repository (private) at `origin/main` =
|
|
4
|
+
`19b2227f17d0116ef5e8d8a6dd1e86ff1fdb913b`.
|
|
5
|
+
The portable system was run against a working copy of that commit. No Koda source,
|
|
6
|
+
configuration or production behaviour was modified.
|
|
7
|
+
|
|
8
|
+
This file is generated by `scripts/compare-koda.py`; every number in the tables
|
|
9
|
+
below is computed from the two graphs. Prose figures that describe a fixed past
|
|
10
|
+
event — how many duplicates an earlier run emitted, how many rows of noise a
|
|
11
|
+
misconfiguration produced — are historical, and are marked as such.
|
|
12
|
+
|
|
13
|
+
**Two bases are used, and they are not interchangeable.** *Triples* are
|
|
14
|
+
`(type, source, target)` — the architecture-level fact. *Relations* are individual
|
|
15
|
+
emitted edges, which outnumber triples wherever one fact is deliberately several
|
|
16
|
+
edges: a scheduler pointing at four routes of one API is four relations and one
|
|
17
|
+
triple. The comparison table is on triples; the relation totals are given
|
|
18
|
+
separately and are stated as relations.
|
|
19
|
+
|
|
20
|
+
**Relation names are normalised before comparison.** The reference prefixes every
|
|
21
|
+
relation with the product namespace (`koda-build-depends-on`); the portable system
|
|
22
|
+
uses neutral names (`buildDependsOn`). Without that normalisation the shared count
|
|
23
|
+
is 44% rather than 84%.
|
|
24
|
+
|
|
25
|
+
## Entities — preserved exactly
|
|
26
|
+
|
|
27
|
+
| kind | reference | portable | identical |
|
|
28
|
+
|---|---|---|---|
|
|
29
|
+
| Domain | 3 | 3 | yes |
|
|
30
|
+
| System | 5 | 5 | yes |
|
|
31
|
+
| Component | 24 | 24 | yes |
|
|
32
|
+
| API | 10 | 10 | yes |
|
|
33
|
+
| Resource | 11 | 11 | yes |
|
|
34
|
+
| **total** | **53** | **53** | **yes** |
|
|
35
|
+
|
|
36
|
+
Every entity ref, name, kind and placement is reproduced. No entity was lost.
|
|
37
|
+
|
|
38
|
+
## Relations — compared as triples
|
|
39
|
+
|
|
40
|
+
| type | reference | portable | shared | reference-only | portable-only |
|
|
41
|
+
|---|---|---|---|---|---|
|
|
42
|
+
| `apiConsumedBy` | 25 | 21 | 19 | 6 | 2 |
|
|
43
|
+
| `apiProvidedBy` | 10 | 10 | 10 | 0 | 0 |
|
|
44
|
+
| `buildDependencyOf` | 24 | 27 | 23 | 1 | 4 |
|
|
45
|
+
| `buildDependsOn` | 24 | 27 | 23 | 1 | 4 |
|
|
46
|
+
| `consumesApi` | 25 | 21 | 19 | 6 | 2 |
|
|
47
|
+
| `hasPart` | 23 | 23 | 23 | 0 | 0 |
|
|
48
|
+
| `partOf` | 23 | 23 | 23 | 0 | 0 |
|
|
49
|
+
| `providesApi` | 10 | 10 | 10 | 0 | 0 |
|
|
50
|
+
| `resourceUsedBy` | 26 | 16 | 16 | 10 | 0 |
|
|
51
|
+
| `runtimeCalledBy` | 3 | 2 | 2 | 1 | 0 |
|
|
52
|
+
| `runtimeCalls` | 3 | 2 | 2 | 1 | 0 |
|
|
53
|
+
| `scheduledBy` | 3 | 2 | 2 | 1 | 0 |
|
|
54
|
+
| `schedules` | 3 | 2 | 2 | 1 | 0 |
|
|
55
|
+
| `testDependencyOf` | 3 | 4 | 3 | 0 | 1 |
|
|
56
|
+
| `testDependsOn` | 3 | 4 | 3 | 0 | 1 |
|
|
57
|
+
| `usesResource` | 26 | 16 | 16 | 10 | 0 |
|
|
58
|
+
| **triples** | **234** | **210** | **196 (84%)** | **38** | **14** |
|
|
59
|
+
|
|
60
|
+
Emitted relation count: reference 234, portable 216. The portable graph emits more
|
|
61
|
+
relations than triples because it does not collapse distinct instances — the
|
|
62
|
+
reference emits the same number as triples, collapsing them.
|
|
63
|
+
|
|
64
|
+
## Other surfaces
|
|
65
|
+
|
|
66
|
+
| surface | reference | portable | note |
|
|
67
|
+
|---|---|---|---|
|
|
68
|
+
| event subjects | 42 | 44 | 20/42 reference subjects covered; 22 need Koda-specific extraction |
|
|
69
|
+
| external dependencies | 24 | 31 | classification declared in configuration |
|
|
70
|
+
| capabilities | 1 | 1 | preserved, via a declared detector |
|
|
71
|
+
| verification entries | 78 | 95 | plus 2 repository-scoped gates; no unrunnable command is emitted |
|
|
72
|
+
| journeys | 1 | 1 | identical |
|
|
73
|
+
| provenance records | 961 | 709 | every relation, fact, contract, conflict and index entry corroborated in both |
|
|
74
|
+
| diagnostics | 3 | 22 | the portable system surfaces more, because it refuses to guess |
|
|
75
|
+
|
|
76
|
+
## New surfaces with no reference equivalent
|
|
77
|
+
|
|
78
|
+
| surface | count |
|
|
79
|
+
|---|---|
|
|
80
|
+
| fact assertions | 110 |
|
|
81
|
+
| conflicts | 7 |
|
|
82
|
+
| unresolved conflicts | 7 |
|
|
83
|
+
| contract pointers | 10 |
|
|
84
|
+
| repository gates | 2 |
|
|
85
|
+
|
|
86
|
+
## What was not reproduced, and why
|
|
87
|
+
|
|
88
|
+
The 38 reference triples absent from the portable graph, grouped by cause:
|
|
89
|
+
|
|
90
|
+
1. **10 resource edges** (`usesResource`/`resourceUsedBy` and their inverses). The
|
|
91
|
+
reference resolves a component to a logical dataset from a bucket literal plus a
|
|
92
|
+
name token. The portable system reports `AGENTDOC_BINDING_AMBIGUOUS` instead,
|
|
93
|
+
because several authored `logical-dataset` Resources match one object-store client
|
|
94
|
+
and no deterministic rule separates them. The information is surfaced for review
|
|
95
|
+
rather than guessed. This is a deliberate difference, not a regression: the
|
|
96
|
+
reference's heuristic is exactly the kind of inference the authority model forbids
|
|
97
|
+
from becoming a fact, and the ambiguity is what produces the 7 unresolved
|
|
98
|
+
`resource.ownership` conflicts above.
|
|
99
|
+
2. **6 API-consumer edges** (`consumesApi`/`apiConsumedBy`). Some are declared as
|
|
100
|
+
`reviewedOverrides` in the reference configuration that were not ported for this
|
|
101
|
+
comparison; the rest were derived by a reference-specific rule. The split is not
|
|
102
|
+
stated as a count because the reference graph records no marker distinguishing a
|
|
103
|
+
reviewed override from a derived edge, so the number is not recoverable from the
|
|
104
|
+
artefact and would be a guess presented as a measurement.
|
|
105
|
+
3. **1 schedule edge and 1 runtime call**: one unreviewed override, and one
|
|
106
|
+
route-owner tie that ranked resolution declines to break.
|
|
107
|
+
4. **1 build edge** derived by a reference-specific devDependency rule.
|
|
108
|
+
|
|
109
|
+
The portable graph additionally contains 14 triples the reference does not,
|
|
110
|
+
principally TypeScript workspace edges derived from `package.json` dependencies,
|
|
111
|
+
which the reference does not record.
|
|
112
|
+
|
|
113
|
+
## Defects found by running against the real repository
|
|
114
|
+
|
|
115
|
+
Each of these was found by running against Koda rather than against a fixture, and
|
|
116
|
+
fixed in the engine rather than in a threshold.
|
|
117
|
+
|
|
118
|
+
1. **Duplicate relations.** The first run emitted 14 relations twice — the same fact
|
|
119
|
+
once per adapter that noticed it — because the derivation tag `via` was part of a
|
|
120
|
+
relation's identity. A `consumesApi` edge asserted by both a reviewed override and
|
|
121
|
+
a source reference appeared twice, with the override's provenance hidden behind a
|
|
122
|
+
derived edge with identical endpoints. `via` is now excluded from identity and the
|
|
123
|
+
derivation routes merge into a sorted list.
|
|
124
|
+
Two follow-on defects of the same kind were found while fixing this: the merge
|
|
125
|
+
*relabelled* a corroborated override as `AUTHORED`, so the graph understated its own
|
|
126
|
+
highest-authority input; and a mechanism route and a declared-client route to one
|
|
127
|
+
Resource still shipped as two edges. Both are fixed.
|
|
128
|
+
2. A flow-mapping scalar containing a comma — `{ description: Body is bad, invalid, or wrong. }` —
|
|
129
|
+
was rejected. Real OpenAPI documents contain these and the ecosystem parses them.
|
|
130
|
+
The flow parser now matches that behaviour deterministically.
|
|
131
|
+
3. A Go workspace member depended on only by import, with no `require` line, was
|
|
132
|
+
invisible: the import pattern required three dots and Go module paths have two.
|
|
133
|
+
4. A repository path was resolved against the process working directory, so the same
|
|
134
|
+
commit compiled differently depending on where the compiler ran.
|
|
135
|
+
5. A task-graph command was emitted for a component that does not declare the task,
|
|
136
|
+
putting an unrunnable command in front of an agent as if it were the answer.
|
|
137
|
+
6. Every third-party package was recorded as an external dependency — 602 rows of
|
|
138
|
+
noise. External-service classification is now declared, and an unclassified count
|
|
139
|
+
is reported for a human to triage.
|
|
140
|
+
7. A command carrying a credential as a separate argument (`curl -u user:pass`)
|
|
141
|
+
reached the graph verbatim. Redaction and the output scanner now cover it.
|
|
142
|
+
|
|
143
|
+
## Requirements the portable system adds
|
|
144
|
+
|
|
145
|
+
- Every `Resource` must state its placement (`system` or `placementRationale`). The
|
|
146
|
+
reference allowed a resource with no owner statement; the portable system treats
|
|
147
|
+
that as the ambiguity it is. Cost: one line per resource.
|
|
148
|
+
- Capability detectors and external-service classification are declared in
|
|
149
|
+
configuration rather than hard-coded in an extractor.
|
|
150
|
+
- Reviewed overrides must state the condition under which they are re-examined, and
|
|
151
|
+
the digests of the evidence they cite are re-verified on every compile.
|
|
152
|
+
- Observations, facts, conflicts and contract pointers are new surfaces with no
|
|
153
|
+
reference equivalent.
|
|
154
|
+
|
|
155
|
+
## Verdict
|
|
156
|
+
|
|
157
|
+
All 53 entities, all 46 placement edges, all 10 API provider edges and the journey
|
|
158
|
+
are preserved exactly. 196 of 234 relation triples (84%) are identical. Every missing
|
|
159
|
+
edge is accounted for above, and in each case the portable system declines to guess
|
|
160
|
+
rather than assert something the repository does not state. No information was
|
|
161
|
+
silently lost: unresolved bindings surface as diagnostics carrying the evidence that
|
|
162
|
+
would settle them.
|
|
163
|
+
|
|
164
|
+
**No blocking regression.** The seven defects above were fixed in the engine.
|