agent-doc-system 1.1.0 → 1.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +402 -0
- package/README.md +38 -0
- package/adapters/ansible.mjs +1 -1
- package/adapters/cache.mjs +9 -2
- package/adapters/contract-consumers.mjs +1 -1
- package/adapters/database.mjs +19 -4
- package/adapters/deploy-config.mjs +3 -2
- package/adapters/dockerfile.mjs +3 -2
- package/adapters/events.mjs +1 -1
- package/adapters/generic.mjs +2 -1
- package/adapters/github-actions.mjs +245 -85
- package/adapters/go.mjs +5 -4
- package/adapters/index.mjs +3 -1
- package/adapters/node.mjs +3 -2
- package/adapters/object-store.mjs +2 -2
- package/adapters/python.mjs +1142 -0
- package/adapters/resource-client.mjs +1 -1
- package/adapters/rust.mjs +990 -0
- package/adapters/turborepo.mjs +3 -2
- package/adapters/typescript.mjs +7 -5
- package/adapters/wrangler.mjs +3 -2
- package/bin/agentdoc.mjs +11 -4
- package/bin/usage.txt +2 -0
- package/core/audit.mjs +12 -8
- package/core/codes.mjs +4 -0
- package/core/compile.mjs +131 -16
- package/core/dependents.mjs +42 -0
- package/core/derive.mjs +4 -3
- package/core/descriptors.mjs +3 -2
- package/core/discover.mjs +61 -9
- package/core/fsx.mjs +185 -36
- package/core/graph.mjs +1 -1
- package/core/impact.mjs +66 -48
- package/core/paths.mjs +71 -0
- package/core/query.mjs +76 -31
- package/core/scaffold.mjs +216 -44
- package/core/sourcescan.mjs +14 -2
- package/core/toml.mjs +463 -140
- package/docs/ADAPTERS.md +196 -6
- package/docs/BENCHMARK.md +130 -0
- package/docs/CI.md +7 -1
- package/docs/EVALS.md +239 -16
- package/docs/MIGRATION.md +26 -6
- package/docs/PROVENANCE.md +6 -0
- package/docs/QUERY.md +78 -7
- package/docs/SPEC.md +69 -2
- package/docs/ci/github-actions.yml +1 -1
- package/evals/baseline-search.mjs +272 -0
- package/evals/facts.mjs +28 -0
- package/evals/harness.mjs +149 -119
- package/evals/scenarios/fixture-b-03-change-shared-lib.json +8 -1
- package/evals/scenarios/fixture-c-03-shared-vocabulary.json +7 -1
- package/evals/scenarios/fixture-e-01-change-shared-lib.json +14 -0
- package/evals/scenarios/fixture-e-02-change-manifest.json +9 -0
- package/evals/scenarios/fixture-e-03-local-module.json +9 -0
- package/evals/scenarios/fixture-e-04-ambiguous-crate.json +14 -0
- package/evals/scenarios/fixture-f-01-change-shared-lib.json +9 -0
- package/evals/scenarios/fixture-f-02-change-manifest.json +9 -0
- package/evals/scenarios/fixture-f-03-shadowed-import.json +9 -0
- package/evals/scenarios/fixture-f-04-nested-workspace.json +9 -0
- package/evals/scenarios/fixture-f-05-alembic-database.json +9 -0
- package/evals/scenarios/fixture-f-06-unittest-job.json +9 -0
- package/package.json +5 -2
- package/skills/agent-doc-system/SKILL.md +2 -1
- package/skills/agent-doc-system/references/cli.md +6 -1
- package/skills/agent-doc-system/references/create-migrate.md +21 -5
- package/templates/ci.yml +1 -1
package/docs/ADAPTERS.md
CHANGED
|
@@ -44,9 +44,17 @@ Available on `ctx`:
|
|
|
44
44
|
| `productionDependencies(ref)` | production dependency names, for client-of-resource rules |
|
|
45
45
|
| `deferSecondPass(fn)` | run after every unit is known |
|
|
46
46
|
| `requirements` | declared normative requirements (`subject`, `key`, `value`, `source`, `satisfiedBy`, `expects`) an adapter may derive the implementation side of |
|
|
47
|
-
| `unitByPkgName`, `unitByGoModule` | cross-unit resolution built before extraction |
|
|
47
|
+
| `unitByPkgName`, `unitByGoModule`, `unitByCrateName` | cross-unit resolution built before extraction |
|
|
48
|
+
| `unitByPythonDist`, `unitByPythonImport` | Python distribution names (PEP 503) and import names, built before extraction |
|
|
48
49
|
| `errors.push({ code, message })` | a hard error, collected rather than thrown |
|
|
49
50
|
|
|
51
|
+
For per-unit source inventories, use `sourceFiles(repo, unit.root)` or
|
|
52
|
+
`repo.walkUnit(unit.root, directory = unit.root)`. Discovery computes ownership
|
|
53
|
+
once from eligible units: these scans prune nested unit roots, so a file belongs
|
|
54
|
+
only to its deepest enclosing unit. `repo.ownsUnitPath(unit.root, path)` applies
|
|
55
|
+
the same boundary to custom traversals. Keep `repo.walk` for repository-wide
|
|
56
|
+
discovery and authored inputs; explicit cross-unit references can still be read.
|
|
57
|
+
|
|
50
58
|
## The three rules
|
|
51
59
|
|
|
52
60
|
### 1. An adapter may not know the repository
|
|
@@ -90,6 +98,8 @@ cannot name the file a fact came from, you cannot add the fact.
|
|
|
90
98
|
| `turborepo` | `turbo.json` (nearest ancestor) | per-package task commands |
|
|
91
99
|
| `typescript` | `tsconfig.json` | language, project references, file-router routes |
|
|
92
100
|
| `go` | `go.mod`, `go.work` | module roots, module graph, `go test`/`go vet` |
|
|
101
|
+
| `rust` | `Cargo.toml` (`[package]`), workspace `members` (including a workspace below the repository root) | crate roots, path, workspace, and `[patch]`/`[replace]` dependency edges, `cargo test`/`cargo clippy` |
|
|
102
|
+
| `python` | `pyproject.toml`, `setup.py`, `setup.cfg`, `requirements*.txt` | distribution graph, import edges, evidence-backed test and static commands |
|
|
93
103
|
| `openapi` | `*.openapi.y[a]ml` | operation inventory, deterministic consumers |
|
|
94
104
|
| `protobuf` / `graphql` | contract files | method/type inventory, consumers |
|
|
95
105
|
| `oidc` | discovery-path literals | OIDC client consumers |
|
|
@@ -106,6 +116,53 @@ cannot name the file a fact came from, you cannot add the fact.
|
|
|
106
116
|
`generic` is always on. Everything else is opt-in through `adapters: [...]`, and
|
|
107
117
|
an unknown name is `AGENTDOC_CONFIG`, not a silent no-op.
|
|
108
118
|
|
|
119
|
+
### `github-actions`
|
|
120
|
+
|
|
121
|
+
Recognizable test, lint and build invocations in `run` steps contribute
|
|
122
|
+
checks. The executable and task determine the tier; a shell `test`, a step
|
|
123
|
+
name or text printed by `echo` does not. Turbo (including package-manager
|
|
124
|
+
wrappers), Nx (`run`, `run-many`, `affected`), Make and Just tasks are resolved
|
|
125
|
+
by task name. Multi-task commands contribute a check for each distinct tier.
|
|
126
|
+
Repository scripts invoked directly (`./scripts/test.sh`) or through `bash`,
|
|
127
|
+
`sh`, `node`, `python` or `python3` also contribute checks when their basename,
|
|
128
|
+
without its extension, matches the task vocabulary (`test`, `lint`, `check`,
|
|
129
|
+
`typecheck`, `build`, `e2e`, `integration`, `contract`, etc.). The script must
|
|
130
|
+
resolve to a literal file inside the repository, relative to the current
|
|
131
|
+
step directory. Its path and arguments use the same scope rules as other
|
|
132
|
+
invocations; without component scope it is a repository gate. Scripts with
|
|
133
|
+
other names, such as `deploy.sh`, are ignored. Script contents are not executed
|
|
134
|
+
or interpreted.
|
|
135
|
+
Simple scripts are split into individual commands, never copied wholesale or
|
|
136
|
+
truncated. A literal `cd` updates the directory for following commands in the
|
|
137
|
+
same step, including across lines. Shell control flow, pipelines,
|
|
138
|
+
substitutions and heredocs are not interpreted; a non-literal `cd`, directory
|
|
139
|
+
stack operation, `source`, `eval` or `export` stops extraction for that step.
|
|
140
|
+
|
|
141
|
+
`uses` actions whose names contain lint, test, check, scan or audit also
|
|
142
|
+
contribute checks. Setup, checkout, cache, upload, download and artifact
|
|
143
|
+
actions are excluded. Action scope comes from `with.working-directory`,
|
|
144
|
+
`with.working_directory` or `with.path`; run-step defaults do not apply. The
|
|
145
|
+
action reference is recorded as its command, with the workflow as its evidence.
|
|
146
|
+
|
|
147
|
+
Scope comes from the step's `working-directory` (including job and workflow
|
|
148
|
+
defaults), a component path argument, an explicit package selector, or a
|
|
149
|
+
workflow trigger's `paths` filter with a literal directory prefix. Paths match
|
|
150
|
+
whole segments; a mention in another step cannot scope a check. Turbo filters
|
|
151
|
+
match literal package names or directory paths; Nx project selectors match a
|
|
152
|
+
package name, `project.json` name or component root; Make's `-C` resolves a
|
|
153
|
+
directory relative to the step's working directory. Unscoped checks are
|
|
154
|
+
repository gates. A working directory is retained in the recorded
|
|
155
|
+
command so the check can be run from the repository root.
|
|
156
|
+
|
|
157
|
+
`impact` on a workflow follows its recorded check configuration paths to the
|
|
158
|
+
covered components, which seed the same reverse build/test dependency closure
|
|
159
|
+
as directly changed components. Their own checks precede their dependents'
|
|
160
|
+
checks, with test-only routes contributing only unit checks. Repository gates
|
|
161
|
+
from that file appear in `affectedGates` and are appended after component
|
|
162
|
+
checks in `recommendedVerification`. The combined list is capped at 25;
|
|
163
|
+
`recommendedVerificationTotal` and `recommendedVerificationTruncated` include
|
|
164
|
+
gates. A workflow matched by a check or gate is removed from `unmatchedFiles`.
|
|
165
|
+
|
|
109
166
|
### `ansible`
|
|
110
167
|
|
|
111
168
|
Reads playbooks under a unit root and follows what they reach — plays,
|
|
@@ -139,6 +196,122 @@ candidate, which is never elected — the requirement stays
|
|
|
139
196
|
`AGENTDOC_REQUIREMENT_UNVERIFIED` and the diagnostic names what was opaque.
|
|
140
197
|
Inventory files, `group_vars`/`host_vars` and variables are not read.
|
|
141
198
|
|
|
199
|
+
### `rust`
|
|
200
|
+
|
|
201
|
+
Reads `Cargo.toml`. A `[package]` is a unit. A virtual workspace manifest (only
|
|
202
|
+
`[workspace]`) is not. Nothing is executed, and `Cargo.lock` is not read.
|
|
203
|
+
|
|
204
|
+
| it derives | from |
|
|
205
|
+
|---|---|
|
|
206
|
+
| `languages: rust`, `runtimes: rust` | the package manifest |
|
|
207
|
+
| deployable | `[[bin]]`, `src/main.rs`, or a `src/bin/` directory |
|
|
208
|
+
| importable | `[lib]` or `src/lib.rs`; both shapes when both are present |
|
|
209
|
+
| `buildDependsOn` | `[dependencies]`, `[build-dependencies]`, and `[target.<cfg>.dependencies]` / `[target.<cfg>.build-dependencies]`, when the entry is a `path`, `workspace = true` resolved through the nearest `[workspace.dependencies]`, or a `[patch.<registry>]` / `[replace]` path redirect to a sibling inside the repository. A production `use`, `extern crate`, or `<crate>::` path (`-` and `_` are the same ident) adds provenance to that edge; it never creates one |
|
|
210
|
+
| `testDependsOn` | `[dev-dependencies]` and target-specific dev-dependencies, resolved the same way |
|
|
211
|
+
| `hasTests` | a `tests/` directory, or `#[test]` / `#[cfg(test)]` in a `.rs` file |
|
|
212
|
+
| verification | `cargo test -p <name>` (unit) and `cargo clippy -p <name>` (static) |
|
|
213
|
+
|
|
214
|
+
`-p <name>` is the whole scope of the recorded command. Cargo, run at a virtual
|
|
215
|
+
workspace root, has no current package; `-p` is what makes the command mean
|
|
216
|
+
this crate rather than every member. `configPaths` is that crate's
|
|
217
|
+
`Cargo.toml`.
|
|
218
|
+
|
|
219
|
+
Workspace roots come from every `Cargo.toml` that has `[workspace]`, not only
|
|
220
|
+
the one at the repository root. The search is bounded to five directories
|
|
221
|
+
down, stays inside the repository, and does not enter `discovery.notUnitRoots`
|
|
222
|
+
or their descendants (the authored-descriptor exception is described below),
|
|
223
|
+
`discovery.generatedDirs`, git-ignored directories, dot-directories, or `test` / `tests` / `examples` /
|
|
224
|
+
`example`. `members` minus `exclude` are still relative to the manifest that
|
|
225
|
+
declares them; `*` matches one path segment, and a member that leaves the
|
|
226
|
+
repository is ignored. `workspace = true` resolves against the nearest
|
|
227
|
+
enclosing workspace. A virtual manifest (no `[package]`) is not a unit and is
|
|
228
|
+
not scaffolded. A manifest that has both `[package]` and `[workspace]` is a
|
|
229
|
+
unit, the same as any other package. Scaffold types a crate from its shape: deployable-only
|
|
230
|
+
(`[[bin]]`, `src/main.rs`, `src/bin/`) is `service`, importable-only is
|
|
231
|
+
`library`, and a crate that is both follows the usual rule (deployable wins,
|
|
232
|
+
so `service`).
|
|
233
|
+
|
|
234
|
+
The manifest route and a corroborating import share the target crate's
|
|
235
|
+
package name as their edge key, so one dependency is one edge. A rename
|
|
236
|
+
(`package = "…"`) is the crates.io name; the `use` path is the table key.
|
|
237
|
+
`[patch.<registry>]` and `[replace]` contribute only when the value is a
|
|
238
|
+
`path` that stays inside the repository and the package is already a
|
|
239
|
+
dependency. A path and a patch that name two different siblings is
|
|
240
|
+
`AGENTDOC_BINDING_AMBIGUOUS` and no edge is created.
|
|
241
|
+
|
|
242
|
+
**What it cannot see, and therefore never invents:** a dependency that exists
|
|
243
|
+
only as a `use`, `extern crate`, or `<crate>::` path. Rust will not link a
|
|
244
|
+
crate the manifest does not declare, and a local module (`mod engine;
|
|
245
|
+
engine::run()`) is the same spelling as a crate path, so an import that the
|
|
246
|
+
manifest did not already resolve creates no edge. An import whose ident
|
|
247
|
+
matches two sibling crates (`foo-bar` and `foo_bar` are both `foo_bar`) is
|
|
248
|
+
`AGENTDOC_BINDING_AMBIGUOUS` and still creates no edge. Also invisible: a
|
|
249
|
+
`Cargo.lock` resolution or any version requirement that merely shares a name
|
|
250
|
+
with a sibling; a workspace member outside the repository; a workspace
|
|
251
|
+
manifest deeper than the search bound; a `#[cfg(test)]` or `#[test]` item,
|
|
252
|
+
which is not a production import; a duplicate package name, which is not
|
|
253
|
+
elected either. A `Cargo.toml` this TOML reader rejects yields
|
|
254
|
+
`AGENTDOC_YAML_PARSE` and no dependency edges. The unit is still recognised when a `[package]` header is present, so
|
|
255
|
+
a parse failure is not an empty unit and not a crash. Char literals and
|
|
256
|
+
lifetimes are not strings: a lifetime does not hide the imports after it, and
|
|
257
|
+
a `"` string that spans lines does not leak the idents inside it.
|
|
258
|
+
|
|
259
|
+
### `python`
|
|
260
|
+
|
|
261
|
+
Reads a unit's Python manifest and its `import` / `from` statements. It does
|
|
262
|
+
not run Python, resolve a lockfile, or inspect a virtualenv.
|
|
263
|
+
|
|
264
|
+
| it derives | from |
|
|
265
|
+
|---|---|
|
|
266
|
+
| deployable | `[project.scripts]`, `[project.gui-scripts]`, `[tool.poetry.scripts]`, a `console_scripts` entry point, `__main__.py`, `manage.py`, or an ASGI/WSGI app object (`app` / `application` assigned from `FastAPI`, `Flask`, `Starlette`, `Litestar`, `Sanic` or `Quart`, or a call to `get_asgi_application` / `get_wsgi_application`) |
|
|
267
|
+
| importable | a build backend (`[build-system] build-backend`, or `setup.py` / `setup.cfg`) and a package |
|
|
268
|
+
| the import name | an explicit package list (`[tool.setuptools] packages`, `[tool.poetry] packages`, `[tool.hatch.build.targets.wheel] packages`); otherwise the distribution name with `-` turned into `_` when that directory exists; otherwise the only package directory |
|
|
269
|
+
| `buildDependsOn` | `[project] dependencies`, `[tool.uv.sources]` (`path`, or `workspace = true` resolved against the nearest enclosing uv workspace), Poetry `path`, `-e <dir>` in a production `requirements*.txt`, or a production `import` / `from` of a sibling's import name |
|
|
270
|
+
| `testDependsOn` | `[dependency-groups]`, `[project.optional-dependencies]` groups named dev/test/lint/type, `[tool.poetry.group.*]`, or a requirements file whose name contains `dev` or `test` |
|
|
271
|
+
| `languages` / `runtimes` | the manifest. `requiresPython` only when `requires-python`, `python_requires`, or Poetry's `python` dependency is written down |
|
|
272
|
+
| `pytest` | a pytest dependency or group, `[tool.pytest.ini_options]`, `pytest.ini`, or `conftest.py` |
|
|
273
|
+
| `python -m unittest discover` | test modules that import `unittest`, and only when nothing above proves pytest |
|
|
274
|
+
| `ruff check` / `mypy` | the tool declared as a dependency or configured (`[tool.ruff]`, `ruff.toml`, `[tool.mypy]`, `mypy.ini`) |
|
|
275
|
+
|
|
276
|
+
`uv run` or `poetry run` is prefixed only when that lockfile contains a
|
|
277
|
+
`name = "<tool>"` entry. Both lockfiles proving the same tool is ambiguous:
|
|
278
|
+
the command is left unprefixed. PEP 508 extras, versions and markers are
|
|
279
|
+
parsed and not evaluated. Two import packages and no explicit list, or two
|
|
280
|
+
units publishing the same distribution or import name, produce
|
|
281
|
+
`AGENTDOC_BINDING_AMBIGUOUS` and no cross-unit link.
|
|
282
|
+
|
|
283
|
+
An import under `if TYPE_CHECKING:` is still an import. The adapter does not
|
|
284
|
+
evaluate control flow, so the name is classified like any other import:
|
|
285
|
+
`buildDependsOn` when it resolves to a sibling, or `testDependsOn` when the
|
|
286
|
+
manifest puts that distribution in a dev group. It is not a separate edge.
|
|
287
|
+
|
|
288
|
+
A uv workspace manifest is read at the repository root and below it, to a
|
|
289
|
+
fixed depth, in path order, staying inside the repository. `notUnitRoots`, their
|
|
290
|
+
descendants, and generated directories are not searched; the authored-descriptor
|
|
291
|
+
exception is described below. `workspace = true` names the member of
|
|
292
|
+
the nearest enclosing workspace whose distribution name matches; another unit
|
|
293
|
+
with the same name is not substituted. Every declared member that has a Python
|
|
294
|
+
manifest is a unit, including one with no build backend and no entrypoint.
|
|
295
|
+
Its shape is still only the evidence, so scaffold writes `other` when the
|
|
296
|
+
manifest is neither deployable nor importable. A manifest that only declares
|
|
297
|
+
the workspace, with no distribution of its own, is not a unit.
|
|
298
|
+
|
|
299
|
+
A `.py` file uses the shared source inventory and is indexed for the deepest
|
|
300
|
+
eligible unit that contains it. Package, entrypoint and test discovery use the
|
|
301
|
+
same ownership boundaries; workspace-member discovery remains repository-wide. A
|
|
302
|
+
parent unit does not gain an import edge from a file that lives in a nested
|
|
303
|
+
member.
|
|
304
|
+
|
|
305
|
+
An import is not evidence of a sibling when the importing unit itself provides
|
|
306
|
+
that top-level module or package — at the unit root, under `src/`, or in the
|
|
307
|
+
importing file's own directory. If a sibling publishes the same import name,
|
|
308
|
+
the result is `AGENTDOC_BINDING_AMBIGUOUS` and no edge.
|
|
309
|
+
|
|
310
|
+
**What it cannot see:** a namespace package with no `__init__.py`; a
|
|
311
|
+
`setup.py` dependency that is not a list of string literals; a `requirements`
|
|
312
|
+
file pulled in with `-r` / `-c`; a lockfile's resolved versions; a PDM
|
|
313
|
+
workspace; a workspace manifest deeper than the search bound.
|
|
314
|
+
|
|
142
315
|
`asyncapi` is a contract **type** (declared on an API descriptor, validated by
|
|
143
316
|
the contract layer), not an adapter — there is no `asyncapi` entry in
|
|
144
317
|
`adapters: [...]`.
|
|
@@ -186,16 +359,33 @@ vocabulary. It does, however, know the shape of a manifest filename in three
|
|
|
186
359
|
places, and those are the honest exceptions worth naming:
|
|
187
360
|
|
|
188
361
|
- `core/compile.mjs` builds a manifest pre-pass over `package.json`, `go.mod`
|
|
189
|
-
and `go.work`,
|
|
362
|
+
and `go.work`, and, when the `rust` adapter is enabled, over `Cargo.toml`.
|
|
363
|
+
`unitByPkgName`, `unitByGoModule`, `unitByCrateName` and
|
|
190
364
|
`transitiveDependencies` are available to every adapter regardless of
|
|
191
|
-
execution order.
|
|
365
|
+
execution order. Cargo is probed only when `rust` is enabled: an existence
|
|
366
|
+
probe is part of the input hash, and probing `Cargo.toml` in every unit
|
|
367
|
+
would change every graph that does not ask for it.
|
|
368
|
+
- The same pre-pass reads a Python manifest (`pyproject.toml`, otherwise
|
|
369
|
+
`setup.cfg` / `setup.py` / `requirements*.txt`) into `unitByPythonDist` and
|
|
370
|
+
`unitByPythonImport`, and feeds those siblings into `dependencyMap` and
|
|
371
|
+
`productionDependencies`, only when the `python` adapter is enabled. Each
|
|
372
|
+
unit root is then listed so `requirements*.txt` is visible; that listing is
|
|
373
|
+
part of the input hash. A repository that does not enable `python` is not
|
|
374
|
+
probed, and its graph does not change.
|
|
192
375
|
- `core/impact.mjs` classifies a changed file by consulting the compiled graph —
|
|
193
376
|
its contracts, its provenance paths, its extractor identities — rather than a
|
|
194
377
|
hand-written list of technology filenames. A list would silently under-report
|
|
195
378
|
for every repository that is not Node-and-Go.
|
|
196
|
-
- `core/discover.mjs`
|
|
197
|
-
|
|
198
|
-
|
|
379
|
+
- `core/discover.mjs` excludes `agentdoc`, `docs`, `scripts`, `node_modules`, and
|
|
380
|
+
`.github` by default. `discovery.notUnitRoots` replaces that list (`[]`
|
|
381
|
+
disables it). These roots and their descendants cannot be inferred from
|
|
382
|
+
adapter markers, adapter `roots()`, or supplemental roots, and scaffold never
|
|
383
|
+
proposes or writes descriptors there in dry run or `--write`. An existing
|
|
384
|
+
authored descriptor strictly below an excluded root still claims a unit and
|
|
385
|
+
is preserved; the adapter must corroborate its artifact type at that unit's
|
|
386
|
+
own root as usual. A descriptor exactly at an excluded root is refused with
|
|
387
|
+
`AGENTDOC_COMPONENT_UNCORROBORATED` naming `notUnitRoots` and asking the author
|
|
388
|
+
to remove the exclusion or move the descriptor.
|
|
199
389
|
|
|
200
390
|
Each is a place a future change should push outward, not a licence to add more.
|
|
201
391
|
|
|
@@ -0,0 +1,130 @@
|
|
|
1
|
+
# BENCHMARK
|
|
2
|
+
|
|
3
|
+
Two measurements are reproducible from a checkout. Neither involves a model;
|
|
4
|
+
both depend on the measurement rules described here.
|
|
5
|
+
|
|
6
|
+
## Routing quality: `node evals/run-all.mjs`
|
|
7
|
+
|
|
8
|
+
The routing evaluation is described in [EVALS.md](EVALS.md). It has 30 scenarios
|
|
9
|
+
on six repositories (Node, TypeScript, Go, Rust and Python), with thresholds
|
|
10
|
+
registered on 2026-09-28 and not moved since. For each scenario, it compares
|
|
11
|
+
the context `agentdoc query` + `agentdoc impact` returns with what an agent
|
|
12
|
+
gets without routing.
|
|
13
|
+
|
|
14
|
+
The measured systems are catalog routing, a corrected graph-assisted targeted
|
|
15
|
+
reference, and working-tree search. Their current results appear below. The
|
|
16
|
+
pre-registered thresholds and scenario truth are unchanged.
|
|
17
|
+
|
|
18
|
+
### Correction: the fact-recall comparison is withdrawn
|
|
19
|
+
|
|
20
|
+
Until 2026-10-27 this page, the README and the 1.2.0 release notes reported
|
|
21
|
+
that AgentDoc retrieves 100% of the facts a change needs, against 58% for a
|
|
22
|
+
baseline that reads only the touched component, with noise under 9% against
|
|
23
|
+
40%. That comparison does not hold, and it is withdrawn.
|
|
24
|
+
|
|
25
|
+
- The targeted baseline already holds almost every expected fact, under a
|
|
26
|
+
different label from the ground truth. It records `usesResource:<ref>` where
|
|
27
|
+
the truth says `uses:<ref>`, and `buildDependencyOf:<ref>` where the truth
|
|
28
|
+
names the component `<ref>`. Relabelled, it retrieves about 99% of the
|
|
29
|
+
expected facts.
|
|
30
|
+
- The baseline also takes those relations from the compiled graph, which an
|
|
31
|
+
agent without AgentDoc does not have. A fair baseline has to find them with
|
|
32
|
+
search over the repository's files.
|
|
33
|
+
|
|
34
|
+
The search comparison is now measured. The targeted reference's corrected
|
|
35
|
+
micro-recall is 100/101 (99.01%); its scenario-mean recall is 99.17%. The
|
|
36
|
+
100%-versus-58% comparison remains withdrawn.
|
|
37
|
+
|
|
38
|
+
### Search baseline
|
|
39
|
+
|
|
40
|
+
`baseline-search` starts with touched files, searches source/manifests for
|
|
41
|
+
import names and local paths in both directions to a fixed point, and reads
|
|
42
|
+
the reached components' docs and checks. Contracts, endpoint strings, event
|
|
43
|
+
subjects and resource evidence are found in working-tree files. Catalog
|
|
44
|
+
descriptors supply only component identity and path ownership. Retrieval never
|
|
45
|
+
reads the compiled graph or compiler relations. All naming guesses, exclusions
|
|
46
|
+
and check conventions are listed in [EVALS.md](EVALS.md#search-baseline-heuristics).
|
|
47
|
+
|
|
48
|
+
| Corpus (scenarios) | System | Mean recall | Noise | Check selectivity |
|
|
49
|
+
|---|---|---:|---:|---:|
|
|
50
|
+
| Koda (9) | baseline-targeted (corrected) | 100.00% | 41.23% | 100.00% |
|
|
51
|
+
| Koda (9) | baseline-search | 71.90% | 71.55% | 100.00% |
|
|
52
|
+
| Koda (9) | catalog | 100.00% | 4.02% | 28.79% |
|
|
53
|
+
| Node/TS (3) | baseline-targeted (corrected) | 100.00% | 17.86% | 100.00% |
|
|
54
|
+
| Node/TS (3) | baseline-search | 91.67% | 30.64% | 83.33% |
|
|
55
|
+
| Node/TS (3) | catalog | 100.00% | 0.00% | 100.00% |
|
|
56
|
+
| Go (4) | baseline-targeted (corrected) | 93.75% | 28.94% | 100.00% |
|
|
57
|
+
| Go (4) | baseline-search | 83.33% | 26.01% | 100.00% |
|
|
58
|
+
| Go (4) | catalog | 100.00% | 5.88% | 66.67% |
|
|
59
|
+
| Mixed monorepo (4) | baseline-targeted (corrected) | 100.00% | 37.82% | 100.00% |
|
|
60
|
+
| Mixed monorepo (4) | baseline-search | 100.00% | 41.25% | 100.00% |
|
|
61
|
+
| Mixed monorepo (4) | catalog | 100.00% | 5.88% | 54.17% |
|
|
62
|
+
| Rust (4) | baseline-targeted (corrected) | 100.00% | 48.25% | 100.00% |
|
|
63
|
+
| Rust (4) | baseline-search | 100.00% | 32.81% | 83.33% |
|
|
64
|
+
| Rust (4) | catalog | 100.00% | 0.00% | 25.00% |
|
|
65
|
+
| Python (6) | baseline-targeted (corrected) | 100.00% | 46.70% | 100.00% |
|
|
66
|
+
| Python (6) | baseline-search | 100.00% | 23.68% | 41.67% |
|
|
67
|
+
| Python (6) | catalog | 100.00% | 0.00% | 41.67% |
|
|
68
|
+
| Aggregate (30) | baseline-targeted (corrected) | 99.17% | 38.83% | 100.00% |
|
|
69
|
+
| Aggregate (30) | baseline-search | 88.51% | 42.61% | 84.44% |
|
|
70
|
+
| Aggregate (30) | catalog | 100.00% | 2.77% | 46.42% |
|
|
71
|
+
|
|
72
|
+
These are scenario means, aggregated by scenario count. Search micro-recall is
|
|
73
|
+
85/101 (84.16%), compared with 100/101 (99.01%) for corrected targeted and
|
|
74
|
+
101/101 for catalog routing. All three systems retrieve every scenario-required
|
|
75
|
+
verification expectation. Selectivity counts distinct registered check IDs,
|
|
76
|
+
including repository gates, after matching search's discovered scoped commands.
|
|
77
|
+
|
|
78
|
+
Search matches catalog fact recall in 21 of 30 scenarios, including all Rust,
|
|
79
|
+
Python and mixed-monorepo scenarios. It beats catalog recall in none. The
|
|
80
|
+
remaining misses include authored placement, authority conflicts/runtime
|
|
81
|
+
observations, and API identities/roles not recoverable by these lexical rules.
|
|
82
|
+
Its broad closure can also overreach through shared endpoint strings. This is
|
|
83
|
+
a comparison of deterministic retrieval rules on authored corpora, not an agent
|
|
84
|
+
session, a limit on search quality, or evidence of time/token savings. Noise is
|
|
85
|
+
scored against the catalog's one-hop graph closure and can penalize useful
|
|
86
|
+
source facts absent from that graph.
|
|
87
|
+
|
|
88
|
+
The unchanged relative recall gate passes: corrected targeted ties catalog in
|
|
89
|
+
five corpora and trails it in Go. All applicable gates pass. Normalization also
|
|
90
|
+
changes noise and deduplicates fact counts: the full Koda inventory now has 86
|
|
91
|
+
facts, below the unchanged 100-fact size floor. Aggregate retrieval reduction
|
|
92
|
+
is **not measured**, so the former 70% size figure is not a current result.
|
|
93
|
+
|
|
94
|
+
## Fresh setup on public repositories: `node scripts/bench/run.mjs`
|
|
95
|
+
|
|
96
|
+
[`scripts/bench/run.mjs`](../scripts/bench/run.mjs) clones each repository in
|
|
97
|
+
[`repos.json`](../scripts/bench/repos.json) at a pinned commit, then does the
|
|
98
|
+
following:
|
|
99
|
+
|
|
100
|
+
1. Runs `agentdoc init` and `agentdoc scaffold --write`.
|
|
101
|
+
2. Fills in the one line AgentDoc refuses to decide: each component's placement.
|
|
102
|
+
3. Counts every other edit a human would need before the catalog validates. That
|
|
103
|
+
number is the "setup ok" column, and the target is zero.
|
|
104
|
+
4. Compiles three times, and checks that the graph is byte-identical.
|
|
105
|
+
5. For each component with a matching file, samples the first tracked source
|
|
106
|
+
file whose path does not contain `test` (benchmarks and examples may be
|
|
107
|
+
included). It measures the `agentdoc query` answer, at about 4 characters per
|
|
108
|
+
token, and the `agentdoc impact` result.
|
|
109
|
+
|
|
110
|
+
Measured with AgentDoc 1.2.0 on Node 24, on a 2019 Intel Core i9 laptop (load average about 4):
|
|
111
|
+
|
|
112
|
+
| repository | files | setup ok | components | dependency edges | checks | init + scaffold + median compile | median compile | deterministic | `query` answer, median / max |
|
|
113
|
+
|---|---|---|---|---|---|---|---|---|---|
|
|
114
|
+
| [tokio-rs/axum](https://github.com/tokio-rs/axum) @ 86aa615 (Rust) | 505 | yes | 4 | 11 | 9 | 1.0 s | 0.5 s | yes | ~151 / 153 tokens |
|
|
115
|
+
| [pydantic/pydantic-ai](https://github.com/pydantic/pydantic-ai) @ 5a4d576 (Python, TS) | 5,142 | yes | 8 | 9 | 5 | 4.6 s | 1.9 s | yes | ~112 / 144 tokens |
|
|
116
|
+
| [fastapi/full-stack-fastapi-template](https://github.com/fastapi/full-stack-fastapi-template) @ f4bcfab (Python, TS) | 252 | yes | 3 | 0 | 9 | 1.0 s | 0.4 s | yes | ~100 / 126 tokens |
|
|
117
|
+
| [t3-oss/create-t3-turbo](https://github.com/t3-oss/create-t3-turbo) @ 8f945b7 (TypeScript) | 141 | yes | 13 | 47 | 25 | 1.0 s | 0.4 s | yes | ~158 / 263 tokens |
|
|
118
|
+
|
|
119
|
+
Timings depend on the machine and its load, so run the script yourself. The
|
|
120
|
+
other columns are deterministic for a given AgentDoc version and commit.
|
|
121
|
+
|
|
122
|
+
### What these numbers do not say
|
|
123
|
+
|
|
124
|
+
- A `query` answer is a map of what to read, what the change reaches, and which
|
|
125
|
+
checks to run. It does not replace the code, so its size is not a "tokens
|
|
126
|
+
saved" figure.
|
|
127
|
+
- "Checks" counts the commands AgentDoc can prove from the repository. For
|
|
128
|
+
example, pydantic-ai's tests live outside the package roots, so AgentDoc
|
|
129
|
+
records none for those packages rather than guess one.
|
|
130
|
+
- Four repositories are a smoke test of the setup path, not a survey.
|
package/docs/CI.md
CHANGED
|
@@ -20,7 +20,7 @@ jobs:
|
|
|
20
20
|
with: { fetch-depth: 0 }
|
|
21
21
|
- uses: actions/setup-node@v4
|
|
22
22
|
with: { node-version: '24' }
|
|
23
|
-
# Pin a tag for reproducibility, e.g. github:lbframe/agent-doc-system#v1.
|
|
23
|
+
# Pin a tag for reproducibility, e.g. github:lbframe/agent-doc-system#v1.3.0 —
|
|
24
24
|
# a compiler upgrade intentionally invalidates every committed graph.
|
|
25
25
|
- run: npm install -g github:lbframe/agent-doc-system
|
|
26
26
|
- run: agentdoc validate
|
|
@@ -47,6 +47,12 @@ compare the graph's recorded commit with `HEAD`: committing the graph changes
|
|
|
47
47
|
The commit is recorded as provenance — which checkout produced this graph — not
|
|
48
48
|
as a gate.
|
|
49
49
|
|
|
50
|
+
Files git ignores are not inputs, so a graph compiled after a local build
|
|
51
|
+
(`astro build`, `next build`, anything writing into an ignored directory) is the
|
|
52
|
+
graph CI reproduces from a clean checkout. No `git clean -fdX` is needed before
|
|
53
|
+
`compile`. An untracked file that is *not* ignored still counts, and marks the
|
|
54
|
+
graph dirty: commit it or ignore it.
|
|
55
|
+
|
|
50
56
|
Two things to decide per repository:
|
|
51
57
|
|
|
52
58
|
- **Commit the graph or ignore it.** Both work. Committing it makes the
|