@erclx/aitk 0.109.0 → 0.111.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +5 -1
- package/claude/.claude-plugin/plugin.json +1 -1
- package/docs/agents/audits.md +88 -0
- package/docs/agents/commands.md +20 -2
- package/docs/agents/index.md +1 -0
- package/docs/agents/scripting.md +13 -9
- package/docs/target-projects.md +6 -0
- package/package.json +1 -1
- package/scripts/core/verify.sh +74 -6
- package/src/audits/baseline.ts +194 -0
- package/src/audits/catalog.ts +464 -0
- package/src/audits/run.ts +202 -0
- package/src/cli.ts +10 -20
- package/src/commands/audits.ts +342 -0
- package/src/commands/claude.ts +34 -4
- package/src/commands/sync.ts +21 -0
- package/src/commands/upgrade.ts +231 -0
- package/src/sync/check.ts +18 -0
- package/src/version/compare.ts +53 -0
- package/src/version/installed.ts +40 -0
- package/src/version/manager.ts +67 -0
- package/src/version/skew.ts +192 -0
package/README.md
CHANGED
|
@@ -38,11 +38,15 @@ Nothing refreshes on its own. Claude Code ships auto-update off for third-party
|
|
|
38
38
|
```bash
|
|
39
39
|
claude plugin marketplace update aitk
|
|
40
40
|
claude plugin update aitk@aitk
|
|
41
|
-
|
|
41
|
+
aitk upgrade
|
|
42
42
|
```
|
|
43
43
|
|
|
44
44
|
The first two update the skills, the third updates the CLI, and they move independently. Restart Claude Code, or run `/reload-plugins`, to pick the skills up.
|
|
45
45
|
|
|
46
|
+
`aitk upgrade` reads the package manager off its own install path and reinstalls with that one, so you don't have to remember which put it there. It names what it detected before it runs anything, and it refuses a source checkout rather than reinstalling over your clone.
|
|
47
|
+
|
|
48
|
+
You don't have to wait until something breaks to find out you're behind. `aitk sync --check` and `aitk claude skills drift` both report the installed version against the newest published one, and neither changes its exit code over it, so an offline machine reads unknown rather than red.
|
|
49
|
+
|
|
46
50
|
To stop doing this by hand, turn auto-update on once under `/plugin` in the Marketplaces tab. Confirm what you are running with `aitk --version` and `claude plugin list`.
|
|
47
51
|
|
|
48
52
|
## Why
|
|
@@ -0,0 +1,88 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: Audits
|
|
3
|
+
description: Running every health check as one set, what the single verdict means, the exit code each outcome takes, the retained baseline and the delta it reports, and which corpora are kept out of the record
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Audits
|
|
7
|
+
|
|
8
|
+
`aitk audits run` runs every audit this repository owns, reports each one under a single verdict, and compares each count to the floor the last recorded run left behind. Nothing here is a new measure. Every number it prints comes from a verb that already published it, which is what makes the command cheap and what separates it from writing another check.
|
|
9
|
+
|
|
10
|
+
```bash
|
|
11
|
+
aitk audits run
|
|
12
|
+
aitk audits run --json
|
|
13
|
+
aitk audits run --record
|
|
14
|
+
aitk audits list --json
|
|
15
|
+
```
|
|
16
|
+
|
|
17
|
+
| Option | Behavior |
|
|
18
|
+
| --------------- | ----------------------------------------------------------------- |
|
|
19
|
+
| `--json` | Add a machine-readable record on stdout, keeping the frame |
|
|
20
|
+
| `--root <path>` | Measure this tree instead of the current worktree |
|
|
21
|
+
| `--record` | Write this run's tracked counts to `.claude/audits/baseline.json` |
|
|
22
|
+
|
|
23
|
+
## What it runs
|
|
24
|
+
|
|
25
|
+
Twelve verbs, listed by `aitk audits list`. Each runs once in its fullest form, and the aggregate reads that verb's own record rather than imposing a shared envelope on it. Every one of those records already has consumers naming its keys, so a common shape would be a breaking change bought for tidiness.
|
|
26
|
+
|
|
27
|
+
The verbs walk separate trees and share no state, so they run together. Measured on the authoring machine, twelve runs finish in 0.8 seconds of wall clock against 4.4 seconds of processor, which is under every other stage in `bun run check`.
|
|
28
|
+
|
|
29
|
+
Each is invoked as the CLI the caller is running rather than as a global `aitk`. A globally installed binary resolves to the main checkout no matter which worktree is executing, so the aggregate would measure a tree the branch never touched and report a pass over it.
|
|
30
|
+
|
|
31
|
+
## What gates and what reports
|
|
32
|
+
|
|
33
|
+
Three findings fail the run: an unresolved context citation, a banned character, word, or spelling, and a skill folder carrying no `REQUIREMENT.md`. Those are the same three `scripts/core/verify.sh` already fails a push on, and the aggregate adds nothing to the set. Each is a fact with no false-positive class behind it.
|
|
34
|
+
|
|
35
|
+
Everything else reports. A heavy bullet, a long entry, a board row nothing resolves, a degradation term in a comment, and an implementation reaching history ahead of its test are judgments a reader settles. A push failing on one of those teaches contributors to route around the stage, which is the split recorded across every audit here and the one this command inherits rather than moves.
|
|
36
|
+
|
|
37
|
+
Growth against the baseline reports too. The standards behind the largest measures set no hard cap, so a count that rose is a fact about the corpus and a judgment about whether it matters.
|
|
38
|
+
|
|
39
|
+
## Exit codes
|
|
40
|
+
|
|
41
|
+
| Code | Meaning |
|
|
42
|
+
| ---- | -------------------------------------------------------------------- |
|
|
43
|
+
| `0` | Every audit reported and none carried a finding that is a fact |
|
|
44
|
+
| `1` | Refused, with the reason on stderr |
|
|
45
|
+
| `2` | An audit carries a finding that is a fact |
|
|
46
|
+
| `3` | An audit did not report, so the run measured less than the whole set |
|
|
47
|
+
|
|
48
|
+
`3` is a defect in the run rather than in the tree, which is why it takes a code of its own. An aggregate reporting a pass over a set it never finished measuring is the failure the command exists against, so an unparseable record, a verb that could not be started, and an exit no verb documents all land there instead of quietly counting as clean. The `markdown audit` empty-ban-set exit is the precedent: a corpus walked with nothing looked for is a broken check, not a clean tree.
|
|
49
|
+
|
|
50
|
+
The verdict is published as a field as well as an exit code. A reader branching on `summary.verdict` gets `clean`, `reported`, `findings`, or `incomplete` without inferring the difference between a quiet corpus and one nobody finished measuring.
|
|
51
|
+
|
|
52
|
+
### An absent corpus is not an unmeasured one
|
|
53
|
+
|
|
54
|
+
A per-machine folder that is not on this disk reports as `absent` and moves neither the verdict nor the exit code. Every one of those folders is gitignored, so a fresh clone and every CI run carry none of them, and counting six expected absences as failures pins the verdict at `incomplete` forever. A signal that never changes is one nobody reads after the second time they see it.
|
|
55
|
+
|
|
56
|
+
The allowance is narrow. It covers a per-machine corpus refusing because its folder is missing, and nothing else. A tracked tree that cannot be found is a broken checkout, and a per-machine verb refusing for any other reason is a broken verb. Both stay `unmeasured`.
|
|
57
|
+
|
|
58
|
+
Every run states how many corpora it measured against how many it skipped, including a run with no findings at all. A count of what passed reads as a verdict on the whole set unless the run also says what it never reached.
|
|
59
|
+
|
|
60
|
+
### What a shallow checkout changes
|
|
61
|
+
|
|
62
|
+
`aitk gov test-order` scopes its range against the trunk and falls back to the root commit when no trunk ref resolves. A depth-1 checkout has neither, so the range is empty and the verb reports zero of everything rather than refusing. The numbers are real for the history present, which means a shallow run under-reports against a baseline taken from a full clone and shows as shrinkage. That is the safe direction, and it is the reason a `test-order` delta is worth reading only from a clone carrying the history.
|
|
63
|
+
|
|
64
|
+
## The retained baseline
|
|
65
|
+
|
|
66
|
+
`.claude/audits/baseline.json` holds the counts from the last run recorded with `--record`, alongside the day it was taken and the commit it was read at. Each later run reports which counts moved and by how much.
|
|
67
|
+
|
|
68
|
+
The file is committed. A per-machine record cannot answer the question this half exists for, which is whether a number grew since anyone last looked, and a fresh checkout has to inherit that answer rather than start over. The cost is that a branch moving a count either fixes it or re-records it and says why, which is the shape the consumed-copy assertion already carries.
|
|
69
|
+
|
|
70
|
+
It sits under the project root rather than beside the aggregator in `src/`. The numbers describe one repository's corpus, and `src/` ships to every project that installs the CLI, so a baseline in the package would hand a target these counts to measure its own tree against.
|
|
71
|
+
|
|
72
|
+
### What is kept out of it
|
|
73
|
+
|
|
74
|
+
A gitignored record folder holds one machine's session scratch. Its counts describe somebody else's disk, so committing them writes a floor no other clone can reproduce and every contributor reads a regression against a number that was never about their tree. The board and five of the six record kinds are therefore reported on every run and recorded in none. `records-standards` is the exception, because that corpus is tracked.
|
|
75
|
+
|
|
76
|
+
An audit that did not report is left out rather than written as zero. Zero there records a clean corpus nobody measured, and the next run reads its real numbers as a regression against a floor that was never taken.
|
|
77
|
+
|
|
78
|
+
### The first run
|
|
79
|
+
|
|
80
|
+
A run with no baseline behind it says so rather than showing a delta of zero. Those two states say the same thing to a reader and mean opposite things, which is a defect this repository has already had to fix twice elsewhere. The same reasoning covers a check the baseline never recorded, a key the run produced that the record does not carry, and a key the record carries that this run did not produce. Each is named for what it is.
|
|
81
|
+
|
|
82
|
+
A hand-edited baseline that does not parse refuses the whole run. Reading a broken record as an absent one would reset the floor the file exists to hold, silently.
|
|
83
|
+
|
|
84
|
+
## In the verify pipeline
|
|
85
|
+
|
|
86
|
+
`bun run check` runs the set as a reporting stage after the three gating stages, and never fails on it. Those three stages keep their own specific remedies, so the aggregate reports the rest and the growth, and a fact still fails the push at the stage that names what to do about it.
|
|
87
|
+
|
|
88
|
+
The stage reads `summary`, a flat object of scalars published beside the nested arrays. Every key in it is unique across the whole record, so a shell stage greps one out without a JSON parser. The three verbs the gating stages already ran walk their trees a second time here, which is the 0.8 seconds measured above and the reason the whole set runs rather than only the part those stages skip: one verdict over every audit is the value, and a stage measuring a subset would report a health nobody took.
|
package/docs/agents/commands.md
CHANGED
|
@@ -13,7 +13,7 @@ Full help: `aitk <command> --help`. Behavior notes for the install and sync verb
|
|
|
13
13
|
| -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
|
|
14
14
|
| `aitk init [path]` | Bootstrap a project with selected toolkit domains |
|
|
15
15
|
| `aitk sync [path]` | Sync all installed domains in a target project |
|
|
16
|
-
| `aitk sync --check` | Report toolkit drift
|
|
16
|
+
| `aitk sync --check` | Report toolkit drift and the installed version against the newest published (`--json`, `--exit-code`) |
|
|
17
17
|
| `aitk sandbox [cat:cmd]` | Run sandbox scenarios (interactive or routed), toolkit-only like the tree it reads |
|
|
18
18
|
| `aitk sandbox reset` | Reset sandbox to baseline |
|
|
19
19
|
| `aitk sandbox clean` | Wipe the sandbox |
|
|
@@ -44,9 +44,12 @@ Full help: `aitk <command> --help`. Behavior notes for the install and sync verb
|
|
|
44
44
|
| `aitk context audit` | Report required sections, length, cited paths, reference form, catalog tables, provenance, superseded-decision narration, and index drift |
|
|
45
45
|
| `aitk markdown audit` | Fail any markdown path on a banned character, word, or spelling, and report the structural checkpoints |
|
|
46
46
|
| `aitk claude skills audit` | Report both skill corpora against the mechanical rules in `standards/skill.md` |
|
|
47
|
-
| `aitk claude skills drift` | Name the shipped skill bodies rewritten between a given ref and `HEAD
|
|
47
|
+
| `aitk claude skills drift` | Name the shipped skill bodies rewritten between a given ref and `HEAD`, and the installed version against the newest published (`--json`) |
|
|
48
48
|
| `aitk gov test-order` | Report where an implementation reached history ahead of the test covering it (`--json`) |
|
|
49
|
+
| `aitk audits run` | Run every audit as one set, report per check under one verdict, and compare each count to the recorded baseline (`--json`, `--record`) |
|
|
50
|
+
| `aitk audits list` | List every audit the set runs, with the corpus each reads and whether it gates (`--json`) |
|
|
49
51
|
| `aitk capture [source]` | Render HTML capture sources to PNG, toolkit-only and absent from an installed package |
|
|
52
|
+
| `aitk upgrade` | Reinstall the CLI globally with the package manager the install path names (`--json`) |
|
|
50
53
|
|
|
51
54
|
## Domain commands
|
|
52
55
|
|
|
@@ -68,6 +71,7 @@ Each domain exposes a consistent shape where applicable: `list`, `install`, `syn
|
|
|
68
71
|
| `comments` | `scan` |
|
|
69
72
|
| `context` | `audit` |
|
|
70
73
|
| `markdown` | `audit` |
|
|
74
|
+
| `audits` | `run`, `list` |
|
|
71
75
|
|
|
72
76
|
Common patterns:
|
|
73
77
|
|
|
@@ -75,3 +79,17 @@ Common patterns:
|
|
|
75
79
|
- `install <name> <path>` → install a specific entry into a target project.
|
|
76
80
|
- `sync <path>` → reapply all installed entries in a target project.
|
|
77
81
|
- `create [name]` → scaffold a new authoring entry in this repo.
|
|
82
|
+
|
|
83
|
+
## Version skew
|
|
84
|
+
|
|
85
|
+
`aitk sync --check` and `aitk claude skills drift` are the two moments a target
|
|
86
|
+
already stops to reconcile with the toolkit, so each reports the installed
|
|
87
|
+
version against the newest published one. No other command performs the lookup,
|
|
88
|
+
which keeps a registry round trip out of the catalog reads an agent runs in a
|
|
89
|
+
loop.
|
|
90
|
+
|
|
91
|
+
The report carries three states and never changes an exit code. `behind` names
|
|
92
|
+
`aitk upgrade` as the remedy, `current` says so, and `unknown` carries the
|
|
93
|
+
reason the registry could not be reached. Branch on the `skew.state` field in
|
|
94
|
+
the JSON record rather than on the exit, since an offline machine has to read as
|
|
95
|
+
unmeasured rather than as a failing check.
|
package/docs/agents/index.md
CHANGED
|
@@ -8,6 +8,7 @@ category: Agent surface
|
|
|
8
8
|
|
|
9
9
|
CLI catalog and invocation rules for agents, split by command domain. Start with overview.
|
|
10
10
|
|
|
11
|
+
- [Audits](audits.md): Running every health check as one set, what the single verdict means, the exit code each outcome takes, the retained baseline and the delta it reports, and which corpora are kept out of the record
|
|
11
12
|
- [Capture](capture.md): Rendering committed HTML sources to PNG, what the command asserts about fonts, and why it is toolkit-only
|
|
12
13
|
- [Command catalog](commands.md): Every project-level command and every domain subcommand, plus the shape each domain exposes
|
|
13
14
|
- [Comments](comments.md): Comment density by language and kind, the two structural exclusions, and how the degradation sweep finds its vocabulary
|
package/docs/agents/scripting.md
CHANGED
|
@@ -11,15 +11,16 @@ What a skill or script reads to discover names at runtime, and how each domain i
|
|
|
11
11
|
|
|
12
12
|
Use these to discover what's available instead of hardcoding names.
|
|
13
13
|
|
|
14
|
-
| Command | Returns
|
|
15
|
-
| -------------------------------- |
|
|
16
|
-
| `aitk tooling list --json` | Stacks, extends chain, dep and script counts
|
|
17
|
-
| `aitk snippets list --json` | Presets and categories with their slugs
|
|
18
|
-
| `aitk standards list --json` | Standards docs and the paths each governs
|
|
19
|
-
| `aitk gov list --json` | Governance stacks, rule sets, and unreferenced rules
|
|
20
|
-
| `aitk claude seeds list --json` | Seed doc sources with content
|
|
21
|
-
| `aitk claude skills list --json` | Plugin skills, descriptions, requirement flag
|
|
22
|
-
| `aitk docs list --json` | Consumer docs plus per-domain context
|
|
14
|
+
| Command | Returns |
|
|
15
|
+
| -------------------------------- | ---------------------------------------------------------------- |
|
|
16
|
+
| `aitk tooling list --json` | Stacks, extends chain, dep and script counts |
|
|
17
|
+
| `aitk snippets list --json` | Presets and categories with their slugs |
|
|
18
|
+
| `aitk standards list --json` | Standards docs and the paths each governs |
|
|
19
|
+
| `aitk gov list --json` | Governance stacks, rule sets, and unreferenced rules |
|
|
20
|
+
| `aitk claude seeds list --json` | Seed doc sources with content |
|
|
21
|
+
| `aitk claude skills list --json` | Plugin skills, descriptions, requirement flag |
|
|
22
|
+
| `aitk docs list --json` | Consumer docs plus per-domain context |
|
|
23
|
+
| `aitk audits list --json` | Audits the set runs, the corpus each reads, and whether it gates |
|
|
23
24
|
|
|
24
25
|
### Catalog fields
|
|
25
26
|
|
|
@@ -129,4 +130,7 @@ AITK_NON_INTERACTIVE=1 aitk wiki init /path/to/project
|
|
|
129
130
|
|
|
130
131
|
# Run a sandbox scenario non-interactively
|
|
131
132
|
SANDBOX_SCENARIO=sync aitk sandbox infra:tooling
|
|
133
|
+
|
|
134
|
+
# Read every audit as one record. Exit 2 is a fact, 3 an audit that did not report
|
|
135
|
+
aitk audits run --json
|
|
132
136
|
```
|
package/docs/target-projects.md
CHANGED
|
@@ -114,6 +114,12 @@ When the toolkit updates, target projects pull changes per domain. There is one
|
|
|
114
114
|
|
|
115
115
|
`aitk sync --check <path>` reports what has drifted without writing anything. It splits each difference by cause, which is the question that decides what to do next.
|
|
116
116
|
|
|
117
|
+
#### The binary first
|
|
118
|
+
|
|
119
|
+
The report opens by naming the binary running it. The installed version reads against the newest published one, and a version behind that points you at `aitk upgrade`, since every section under it is a reading taken by whichever toolkit you happen to have. An unreachable registry reports unknown with its reason rather than failing, so `--exit-code` still gates on the drift the check measured locally and an offline machine stays green.
|
|
120
|
+
|
|
121
|
+
#### Then the causes
|
|
122
|
+
|
|
117
123
|
A `stale` file still matches what the toolkit installed, so the update is mechanical. A `customized` file carries local edits, so taking the upstream version is a decision and `aitk:claude-seed-sync` is the tool for it. A `stranded` file sits where an older toolkit installed it and the toolkit has since moved, which is what `aitk:migration-standards` handles.
|
|
118
124
|
|
|
119
125
|
That attribution comes from `.claude/aitk.json`, a stamp every install and sync writes. Standards, snippets, and governance record a hash per installed file. Tooling records the stack chain it resolved instead, since its install runs no per-file walk to attribute.
|
package/package.json
CHANGED
package/scripts/core/verify.sh
CHANGED
|
@@ -25,6 +25,11 @@ SANDBOX_UNDECLARED_CEILING=47
|
|
|
25
25
|
# a new arrival against, and a config file would absorb the arrival silently.
|
|
26
26
|
GOV_EXPECTED_UNREFERENCED="260-shadcn 320-tanstack-query"
|
|
27
27
|
|
|
28
|
+
# The retained counts the audit stage compares each run against. Spelled here
|
|
29
|
+
# rather than derived, because this script only ever names the file in a remedy
|
|
30
|
+
# a reader has to be able to open, and `aitk audits run` owns writing it.
|
|
31
|
+
AUDITS_BASELINE=".claude/audits/baseline.json"
|
|
32
|
+
|
|
28
33
|
check_dependencies() {
|
|
29
34
|
command -v bun >/dev/null 2>&1 || log_error "bun is not installed"
|
|
30
35
|
}
|
|
@@ -209,10 +214,11 @@ seed_entry_count() {
|
|
|
209
214
|
awk '{ total += $1 } END { print total + 0 }'
|
|
210
215
|
}
|
|
211
216
|
|
|
212
|
-
# One numeric summary key out of a
|
|
213
|
-
#
|
|
214
|
-
# the caller does not depend on the order the keys are
|
|
215
|
-
|
|
217
|
+
# One numeric summary key out of a command's JSON record. Every caller passes a
|
|
218
|
+
# name the nested objects in that record do not carry, so the match reaches the
|
|
219
|
+
# top level alone and the caller does not depend on the order the keys are
|
|
220
|
+
# emitted in.
|
|
221
|
+
json_summary_field() {
|
|
216
222
|
printf '%s' "$2" | grep -o "\"$1\":[0-9]\+" | grep -o '[0-9]\+'
|
|
217
223
|
}
|
|
218
224
|
|
|
@@ -466,8 +472,8 @@ main() {
|
|
|
466
472
|
# `|| x=""` on both, because a grep that matches nothing exits non-zero and
|
|
467
473
|
# errexit would take the script down at the assignment, before the guard
|
|
468
474
|
# below could name what went missing.
|
|
469
|
-
total=$(
|
|
470
|
-
armed=$(
|
|
475
|
+
total=$(json_summary_field totalScenarios "$coverage_output") || total=""
|
|
476
|
+
armed=$(json_summary_field armedScenarios "$coverage_output") || armed=""
|
|
471
477
|
if [ -z "$total" ] || [ -z "$armed" ]; then
|
|
472
478
|
log_error "The coverage report carried no scenario totals, so the stage measured nothing. Run bun src/cli.ts sandbox coverage --json."
|
|
473
479
|
fi
|
|
@@ -478,6 +484,68 @@ main() {
|
|
|
478
484
|
log_info "$armed of $total scenarios declare expectations, $undeclared undeclared against a ceiling of $SANDBOX_UNDECLARED_CEILING"
|
|
479
485
|
fi
|
|
480
486
|
|
|
487
|
+
# The three stages above gate on the three findings here that are facts, and
|
|
488
|
+
# this stage reports the rest. It runs the whole set anyway rather than only
|
|
489
|
+
# what those stages skip, because the aggregate's own value is one verdict
|
|
490
|
+
# over every audit, and a stage measuring a subset would report a health this
|
|
491
|
+
# repository never took.
|
|
492
|
+
#
|
|
493
|
+
# The duplicate walk costs 0.8s wall against roughly 4.4s of processor,
|
|
494
|
+
# measured on the authoring machine at 12 verbs run together. That is under
|
|
495
|
+
# every other stage in this script, which is what settles the open question
|
|
496
|
+
# about whether the pipeline can afford it.
|
|
497
|
+
#
|
|
498
|
+
# This reports and never fails. Growth in a judgment count is the thing the
|
|
499
|
+
# baseline exists to make visible, and failing a push on one would teach a
|
|
500
|
+
# contributor to route around the stage, which is the split every audit stage
|
|
501
|
+
# here already keeps. A fact still fails the push, at the specific stage above
|
|
502
|
+
# that names its own remedy.
|
|
503
|
+
log_step "Audit set"
|
|
504
|
+
local audits_output audits_status=0 audits_grown audits_shrunk audits_facts audits_unmeasured audits_absent audits_unrecorded
|
|
505
|
+
audits_output=$(cd "$PROJECT_ROOT" && bun src/cli.ts audits run --json 2>/dev/null) || audits_status=$?
|
|
506
|
+
if [ -z "$audits_output" ]; then
|
|
507
|
+
log_warn "Skipped, the audit set did not report (exit $audits_status)"
|
|
508
|
+
else
|
|
509
|
+
audits_grown=$(json_summary_field grown "$audits_output") || audits_grown=""
|
|
510
|
+
audits_shrunk=$(json_summary_field shrunk "$audits_output") || audits_shrunk=""
|
|
511
|
+
audits_facts=$(json_summary_field facts "$audits_output") || audits_facts=""
|
|
512
|
+
audits_unmeasured=$(json_summary_field unmeasured "$audits_output") || audits_unmeasured=""
|
|
513
|
+
audits_absent=$(json_summary_field absent "$audits_output") || audits_absent=""
|
|
514
|
+
audits_unrecorded=$(json_summary_field unrecorded "$audits_output") || audits_unrecorded=""
|
|
515
|
+
|
|
516
|
+
# An absent field is a record this stage cannot read, which is not the same
|
|
517
|
+
# as a run with nothing to report. Reading it as zero would print a clean
|
|
518
|
+
# line over a summary nobody parsed.
|
|
519
|
+
if [ -z "$audits_grown" ] || [ -z "$audits_facts" ] || [ -z "$audits_unmeasured" ]; then
|
|
520
|
+
log_warn "The audit record carried no summary, so this stage measured nothing. Run bun src/cli.ts audits run."
|
|
521
|
+
else
|
|
522
|
+
# An absent per-machine folder is the ordinary state here rather than a
|
|
523
|
+
# finding, since every one of them is gitignored and CI carries none. It
|
|
524
|
+
# is still stated, because a stage naming only what it measured claims a
|
|
525
|
+
# coverage it does not have.
|
|
526
|
+
if [ -n "$audits_absent" ] && [ "$audits_absent" -gt 0 ]; then
|
|
527
|
+
log_info "$audits_absent per-machine corpus/corpora absent, so unmeasured here by design"
|
|
528
|
+
fi
|
|
529
|
+
if [ "$audits_unmeasured" -gt 0 ]; then
|
|
530
|
+
log_warn "$audits_unmeasured audit(s) did not report, so the set is incomplete. Run bun src/cli.ts audits run."
|
|
531
|
+
fi
|
|
532
|
+
if [ "$audits_facts" -gt 0 ]; then
|
|
533
|
+
log_warn "$audits_facts audit(s) carry a finding that is a fact. The stage above names the remedy."
|
|
534
|
+
fi
|
|
535
|
+
if [ -n "$audits_unrecorded" ] && [ "$audits_unrecorded" -gt 0 ]; then
|
|
536
|
+
log_warn "$audits_unrecorded tracked audit(s) have no recorded floor. Take one with bun src/cli.ts audits run --record."
|
|
537
|
+
fi
|
|
538
|
+
if [ "$audits_grown" -gt 0 ]; then
|
|
539
|
+
log_warn "$audits_grown measure(s) grew against $AUDITS_BASELINE. Run bun src/cli.ts audits run to see which, then fix them or re-record and say why."
|
|
540
|
+
else
|
|
541
|
+
log_info "No measure grew against $AUDITS_BASELINE"
|
|
542
|
+
fi
|
|
543
|
+
if [ -n "$audits_shrunk" ] && [ "$audits_shrunk" -gt 0 ]; then
|
|
544
|
+
log_info "$audits_shrunk measure(s) fell against $AUDITS_BASELINE"
|
|
545
|
+
fi
|
|
546
|
+
fi
|
|
547
|
+
fi
|
|
548
|
+
|
|
481
549
|
# The plugin is the second delivery path and this is the only stage gating it,
|
|
482
550
|
# so the skip below is for a contributor's machine rather than for the merge
|
|
483
551
|
# gate. A runner installs the CLI as a workflow step, which makes an absent
|
|
@@ -0,0 +1,194 @@
|
|
|
1
|
+
import { mkdir, readFile, writeFile } from 'node:fs/promises'
|
|
2
|
+
import { dirname, join } from 'node:path'
|
|
3
|
+
import type { AuditResult } from '@/audits/catalog'
|
|
4
|
+
|
|
5
|
+
/**
|
|
6
|
+
* Where the retained counts live, relative to the project root.
|
|
7
|
+
*
|
|
8
|
+
* Committed rather than per-machine, because the question this half answers is
|
|
9
|
+
* whether a number grew since anyone last looked, and a fresh checkout has to
|
|
10
|
+
* inherit that answer. A per-machine record makes every contributor's first run
|
|
11
|
+
* a first run.
|
|
12
|
+
*
|
|
13
|
+
* Under the project root rather than beside the aggregator in `src/`, because
|
|
14
|
+
* the numbers describe one repository's corpus and `src/` ships to every target
|
|
15
|
+
* that installs the CLI. A baseline in the package would hand a target this
|
|
16
|
+
* repository's counts to measure its own tree against.
|
|
17
|
+
*/
|
|
18
|
+
export const BASELINE_REL = join('.claude', 'audits', 'baseline.json')
|
|
19
|
+
|
|
20
|
+
export interface Baseline {
|
|
21
|
+
/** The day the record was taken, as `YYYY-MM-DD`. */
|
|
22
|
+
readonly recordedAt: string
|
|
23
|
+
/** The commit the counts were read at, so a reader can reproduce them. */
|
|
24
|
+
readonly commit: string
|
|
25
|
+
readonly checks: Readonly<Record<string, Readonly<Record<string, number>>>>
|
|
26
|
+
}
|
|
27
|
+
|
|
28
|
+
export interface Stamp {
|
|
29
|
+
readonly recordedAt: string
|
|
30
|
+
readonly commit: string
|
|
31
|
+
}
|
|
32
|
+
|
|
33
|
+
export interface MovedCount {
|
|
34
|
+
readonly key: string
|
|
35
|
+
readonly from: number
|
|
36
|
+
readonly to: number
|
|
37
|
+
readonly delta: number
|
|
38
|
+
}
|
|
39
|
+
|
|
40
|
+
export type Delta =
|
|
41
|
+
| {
|
|
42
|
+
readonly id: string
|
|
43
|
+
readonly kind: 'compared'
|
|
44
|
+
readonly moved: readonly MovedCount[]
|
|
45
|
+
/** Keys the run reproduced exactly. */
|
|
46
|
+
readonly steady: readonly string[]
|
|
47
|
+
/** Keys the run produced that the baseline never recorded. */
|
|
48
|
+
readonly added: readonly { key: string; to: number }[]
|
|
49
|
+
/** Keys the baseline holds that this run did not produce. */
|
|
50
|
+
readonly dropped: readonly { key: string; from: number }[]
|
|
51
|
+
}
|
|
52
|
+
/** No recorded floor, so a zero delta would be indistinguishable from quiet. */
|
|
53
|
+
| { readonly id: string; readonly kind: 'unrecorded' }
|
|
54
|
+
/** Gitignored scratch, whose counts are one machine's and answer nobody else. */
|
|
55
|
+
| { readonly id: string; readonly kind: 'per-machine' }
|
|
56
|
+
/** The audit did not report, so there is nothing to compare. */
|
|
57
|
+
| { readonly id: string; readonly kind: 'unmeasured' }
|
|
58
|
+
|
|
59
|
+
/**
|
|
60
|
+
* Builds the record a run leaves behind.
|
|
61
|
+
*
|
|
62
|
+
* Only a tracked corpus is retained. A gitignored record folder holds one
|
|
63
|
+
* machine's session scratch, so committing its counts writes a floor no other
|
|
64
|
+
* clone can reproduce, and every contributor would read a regression against a
|
|
65
|
+
* number that describes somebody else's disk.
|
|
66
|
+
*
|
|
67
|
+
* An audit that did not report is left out rather than written as zero. Zeros
|
|
68
|
+
* there record a clean corpus nobody measured, and the next run reads its real
|
|
69
|
+
* numbers as a regression against a floor that was never taken.
|
|
70
|
+
*/
|
|
71
|
+
export function baselineFrom(
|
|
72
|
+
results: readonly AuditResult[],
|
|
73
|
+
stamp: Stamp,
|
|
74
|
+
): Baseline {
|
|
75
|
+
const checks: Record<string, Record<string, number>> = {}
|
|
76
|
+
|
|
77
|
+
for (const result of results) {
|
|
78
|
+
if (!result.tracked || result.counts === undefined) continue
|
|
79
|
+
checks[result.id] = { ...result.counts }
|
|
80
|
+
}
|
|
81
|
+
|
|
82
|
+
return { recordedAt: stamp.recordedAt, commit: stamp.commit, checks }
|
|
83
|
+
}
|
|
84
|
+
|
|
85
|
+
/**
|
|
86
|
+
* Compares this run against the recorded floor, one audit at a time.
|
|
87
|
+
*
|
|
88
|
+
* A first run reports `unrecorded` rather than a delta of zero. A zero delta
|
|
89
|
+
* against an absent baseline says the same thing as a corpus that did not move,
|
|
90
|
+
* and those are the two states this repository has already had to separate
|
|
91
|
+
* twice elsewhere.
|
|
92
|
+
*/
|
|
93
|
+
export function compareBaseline(
|
|
94
|
+
baseline: Baseline | undefined,
|
|
95
|
+
results: readonly AuditResult[],
|
|
96
|
+
): Delta[] {
|
|
97
|
+
return results.map((result) => {
|
|
98
|
+
if (!result.tracked) return { id: result.id, kind: 'per-machine' as const }
|
|
99
|
+
if (result.counts === undefined) {
|
|
100
|
+
return { id: result.id, kind: 'unmeasured' as const }
|
|
101
|
+
}
|
|
102
|
+
|
|
103
|
+
const recorded = baseline?.checks[result.id]
|
|
104
|
+
if (recorded === undefined) {
|
|
105
|
+
return { id: result.id, kind: 'unrecorded' as const }
|
|
106
|
+
}
|
|
107
|
+
|
|
108
|
+
const moved: MovedCount[] = []
|
|
109
|
+
const steady: string[] = []
|
|
110
|
+
const added: { key: string; to: number }[] = []
|
|
111
|
+
const dropped: { key: string; from: number }[] = []
|
|
112
|
+
|
|
113
|
+
for (const [key, to] of Object.entries(result.counts)) {
|
|
114
|
+
const from = recorded[key]
|
|
115
|
+
if (from === undefined) {
|
|
116
|
+
added.push({ key, to })
|
|
117
|
+
} else if (from === to) {
|
|
118
|
+
steady.push(key)
|
|
119
|
+
} else {
|
|
120
|
+
moved.push({ key, from, to, delta: to - from })
|
|
121
|
+
}
|
|
122
|
+
}
|
|
123
|
+
|
|
124
|
+
for (const [key, from] of Object.entries(recorded)) {
|
|
125
|
+
if (!(key in result.counts)) dropped.push({ key, from })
|
|
126
|
+
}
|
|
127
|
+
|
|
128
|
+
return {
|
|
129
|
+
id: result.id,
|
|
130
|
+
kind: 'compared' as const,
|
|
131
|
+
moved,
|
|
132
|
+
steady,
|
|
133
|
+
added,
|
|
134
|
+
dropped,
|
|
135
|
+
}
|
|
136
|
+
})
|
|
137
|
+
}
|
|
138
|
+
|
|
139
|
+
function isBaseline(value: unknown): value is Baseline {
|
|
140
|
+
if (typeof value !== 'object' || value === null) return false
|
|
141
|
+
const record = value as Record<string, unknown>
|
|
142
|
+
return (
|
|
143
|
+
typeof record.recordedAt === 'string' &&
|
|
144
|
+
typeof record.commit === 'string' &&
|
|
145
|
+
typeof record.checks === 'object' &&
|
|
146
|
+
record.checks !== null &&
|
|
147
|
+
!Array.isArray(record.checks)
|
|
148
|
+
)
|
|
149
|
+
}
|
|
150
|
+
|
|
151
|
+
/**
|
|
152
|
+
* Reads the recorded floor, or `undefined` when none has been taken.
|
|
153
|
+
*
|
|
154
|
+
* An absent file and a broken one are different answers. Absent is a first run.
|
|
155
|
+
* Broken is a record someone hand-edited into a shape nothing can read, and
|
|
156
|
+
* reading that as absent would silently reset the floor the file exists to hold.
|
|
157
|
+
*/
|
|
158
|
+
export async function readBaseline(
|
|
159
|
+
root: string,
|
|
160
|
+
): Promise<Baseline | undefined> {
|
|
161
|
+
const path = join(root, BASELINE_REL)
|
|
162
|
+
|
|
163
|
+
let raw: string
|
|
164
|
+
try {
|
|
165
|
+
raw = await readFile(path, 'utf8')
|
|
166
|
+
} catch {
|
|
167
|
+
return undefined
|
|
168
|
+
}
|
|
169
|
+
|
|
170
|
+
let parsed: unknown
|
|
171
|
+
try {
|
|
172
|
+
parsed = JSON.parse(raw)
|
|
173
|
+
} catch {
|
|
174
|
+
throw new Error(`${BASELINE_REL} does not parse as JSON. Fix or delete it.`)
|
|
175
|
+
}
|
|
176
|
+
|
|
177
|
+
if (!isBaseline(parsed)) {
|
|
178
|
+
throw new Error(
|
|
179
|
+
`${BASELINE_REL} carries no recordedAt, commit, and checks. Fix or delete it.`,
|
|
180
|
+
)
|
|
181
|
+
}
|
|
182
|
+
|
|
183
|
+
return parsed
|
|
184
|
+
}
|
|
185
|
+
|
|
186
|
+
export async function writeBaseline(
|
|
187
|
+
root: string,
|
|
188
|
+
baseline: Baseline,
|
|
189
|
+
): Promise<string> {
|
|
190
|
+
const path = join(root, BASELINE_REL)
|
|
191
|
+
await mkdir(dirname(path), { recursive: true })
|
|
192
|
+
await writeFile(path, `${JSON.stringify(baseline, null, 2)}\n`, 'utf8')
|
|
193
|
+
return path
|
|
194
|
+
}
|