@erclx/aitk 2.0.0 → 2.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/claude/.claude-plugin/plugin.json +1 -1
- package/docs/agents/audits.md +14 -4
- package/docs/agents/commands.md +46 -42
- package/docs/agents/index.md +1 -0
- package/docs/agents/scripting.md +15 -0
- package/docs/agents/state-scoped-risk.md +105 -0
- package/package.json +1 -1
- package/scripts/lib/sandbox-git.sh +56 -4
- package/src/audits/baseline.ts +8 -1
- package/src/audits/catalog.ts +128 -9
- package/src/audits/run.ts +1 -0
- package/src/cli.ts +8 -0
- package/src/commands/audits.ts +11 -3
- package/src/commands/deps.ts +173 -0
- package/src/commands/secrets.ts +132 -0
- package/src/deps/audit.ts +153 -0
- package/src/secrets/marker.ts +44 -0
- package/src/secrets/patterns.ts +142 -0
- package/src/secrets/scan.ts +134 -0
- package/src/secrets/shipped.ts +111 -0
package/docs/agents/audits.md
CHANGED
|
@@ -22,15 +22,19 @@ aitk audits list --json
|
|
|
22
22
|
|
|
23
23
|
## What it runs
|
|
24
24
|
|
|
25
|
-
|
|
25
|
+
Fourteen verbs, listed by `aitk audits list`. Each runs once in its fullest form, and the aggregate reads that verb's own record rather than imposing a shared envelope on it. Every one of those records already has consumers naming its keys, so a common shape would be a breaking change bought for tidiness.
|
|
26
26
|
|
|
27
|
-
The verbs walk separate trees and share no state, so they run together. Measured on the authoring machine
|
|
27
|
+
The verbs walk separate trees and share no state, so they run together. Measured on the authoring machine at twelve verbs, a run finished in 0.8 seconds of wall clock against 4.4 seconds of processor, which is under every other stage in `bun run check`. `aitk deps audit` is the one that changes that reading, since it reaches a network rather than a tree and its latency is the index's rather than this machine's.
|
|
28
|
+
|
|
29
|
+
Twelve of the fourteen read a tree on this disk. The two added by `state-scoped-risk.md` read committed state rather than an arriving change, which is the gap every review surface here leaves by construction.
|
|
28
30
|
|
|
29
31
|
Each is invoked as the CLI the caller is running rather than as a global `aitk`. A globally installed binary resolves to the main checkout no matter which worktree is executing, so the aggregate would measure a tree the branch never touched and report a pass over it.
|
|
30
32
|
|
|
31
33
|
## What gates and what reports
|
|
32
34
|
|
|
33
|
-
|
|
35
|
+
Four findings fail the run: an unresolved context citation, a banned character, word, or spelling, a skill folder carrying no `REQUIREMENT.md`, and a credential-shaped value in the tree the package ships. Each is a fact with no false-positive class behind it.
|
|
36
|
+
|
|
37
|
+
Three of the four are the ones `scripts/core/verify.sh` already fails a push on. The secret scan is the one entry gating without a stage behind it, added on that same fact-or-judgment test rather than as a side effect of registering a measure, and the architecture record already ranks content leaving the repository above content that stays.
|
|
34
38
|
|
|
35
39
|
Everything else reports. A heavy bullet, a long entry, a board row nothing resolves, a degradation term in a comment, and an implementation reaching history ahead of its test are judgments a reader settles. A push failing on one of those teaches contributors to route around the stage, which is the split recorded across every audit here and the one this command inherits rather than moves.
|
|
36
40
|
|
|
@@ -53,7 +57,13 @@ The verdict is published as a field as well as an exit code. A reader branching
|
|
|
53
57
|
|
|
54
58
|
A per-machine folder that is not on this disk reports as `absent` and moves neither the verdict nor the exit code. Every one of those folders is gitignored, so a fresh clone and every CI run carry none of them, and counting six expected absences as failures pins the verdict at `incomplete` forever. A signal that never changes is one nobody reads after the second time they see it.
|
|
55
59
|
|
|
56
|
-
|
|
60
|
+
Which refusals count as an absence is declared per audit rather than inferred from the corpus alone. A per-machine corpus counts a missing folder. An upstream corpus counts every reason the advisory verb publishes, since each of the three means there is nothing resolved to measure. The secret scan overrides its tracked default for the three reasons that mean the project publishes nothing at all.
|
|
61
|
+
|
|
62
|
+
Everything else stays `unmeasured`. That covers a tree git cannot list, which is a broken checkout, and a package declaring no `files` field, where a publish would pack the whole tree and the scan read none of it. Both are a corpus that exists and went unread, so softening either would report a pass over a shipped tree nobody measured.
|
|
63
|
+
|
|
64
|
+
The override exists because the corpus answers the wrong question for one audit. What earns it is `private: true`, the one declaration that a package is never published, so a project carrying it reports an absent corpus rather than a broken one. Without that the aggregate would report `incomplete` in every such project on every run, which is the permanent signal this section opens by rejecting.
|
|
65
|
+
|
|
66
|
+
`upstream` is the third corpus value, carried by the advisory check alone. Its count moves when someone publishes rather than when someone edits here, so it is kept out of the retained baseline for the mirror image of the reason gitignored scratch is, and an offline run reports it absent rather than pinning the verdict at `incomplete` on every machine without a network.
|
|
57
67
|
|
|
58
68
|
Every run states how many corpora it measured against how many it skipped, including a run with no findings at all. A count of what passed reads as a verdict on the whole set unless the run also says what it never reached.
|
|
59
69
|
|
package/docs/agents/commands.md
CHANGED
|
@@ -9,48 +9,50 @@ Full help: `aitk <command> --help`. Behavior notes for the install and sync verb
|
|
|
9
9
|
|
|
10
10
|
## Project-level
|
|
11
11
|
|
|
12
|
-
| Command | Purpose
|
|
13
|
-
| -------------------------- |
|
|
14
|
-
| `aitk init [path]` | Bootstrap a project with selected toolkit domains
|
|
15
|
-
| `aitk sync [path]` | Sync all installed domains in a target project
|
|
16
|
-
| `aitk sync --check` | Report toolkit drift and the installed version against the newest published (`--json`, `--exit-code`)
|
|
17
|
-
| `aitk sandbox [cat:cmd]` | Run sandbox scenarios (interactive or routed), toolkit-only like the tree it reads
|
|
18
|
-
| `aitk sandbox reset` | Reset sandbox to baseline
|
|
19
|
-
| `aitk sandbox clean` | Wipe the sandbox
|
|
20
|
-
| `aitk sandbox check` | Score a provisioned sandbox against a scenario expectation (`--json` for the verdict)
|
|
21
|
-
| `aitk sandbox coverage` | Report which scenarios declare expectations (`--json`, `--strict`, `--skills`)
|
|
22
|
-
| `aitk indexes regen` | Regenerate `index.md` files from sibling frontmatter
|
|
23
|
-
| `aitk docs [topic]` | Emit toolkit reference docs (`list`, or a topic by name)
|
|
24
|
-
| `aitk design render` | Render `.claude/DESIGN.md` tokens to HTML and CSS
|
|
25
|
-
| `aitk slides render` | Render a `.claude/SLIDES.md` source into a PowerPoint deck
|
|
26
|
-
| `aitk slides list` | List the available slide layouts (`--json` for the catalog)
|
|
27
|
-
| `aitk feedback` | Write toolkit feedback from stdin to `.claude/review/`, or open a GitHub issue with `--github`
|
|
28
|
-
| `aitk transcripts <url>` | Fetch a YouTube transcript with metadata frontmatter (needs `yt-dlp`)
|
|
29
|
-
| `aitk tasks archive` | Move a shipped task off the board, clear its ordering row, and regenerate the index
|
|
30
|
-
| `aitk tasks pull-request` | Record a pull request number on the task a branch closes, by stem or `--plan` (`--json`)
|
|
31
|
-
| `aitk tasks outcome` | Mark outcomes `[x]` on a task by position, repeating `--close` (`--json`)
|
|
32
|
-
| `aitk tasks validate` | Report board rows whose plan, task file, group, file set, or blocker does not hold (`--json`)
|
|
33
|
-
| `aitk intake list` | Report intake folder counts, or one folder's items, keeping what is unread with `--unread` (`--json`)
|
|
34
|
-
| `aitk intake answer` | Write selections into one cluster's answer slots, repeating `--set <item>=<answer>` (`--json`)
|
|
35
|
-
| `aitk teach list` | Report learning workspaces and the ordinal a new one takes, or what one workspace holds (`--json`)
|
|
36
|
-
| `aitk teach open` | Open a workspace at the next ordinal with its mission, resources, and glossary files (`--json`)
|
|
37
|
-
| `aitk teach resource` | Record sources and leads in a workspace, repeating `--read` or `--lead` as `<title>=<url>` (`--json`)
|
|
38
|
-
| `aitk teach glossary` | Add terms to a workspace glossary alphabetically, repeating `--term <term>=<definition>` (`--json`)
|
|
39
|
-
| `aitk records validate` | Report a session record or a standard against the standard governing it, per kind (`--json`)
|
|
40
|
-
| `aitk records size` | Report what each record folder holds and how much of it is recent, heaviest first (`--json`)
|
|
41
|
-
| `aitk records push` | Commit the nine backed record folders and push them to a private records remote (`--json`)
|
|
42
|
-
| `aitk records pull` | Fetch the records remote and write it back, refusing rather than discarding unpushed records (`--json`)
|
|
43
|
-
| `aitk sessions list` | Resolve live sessions to the worktree and branch each holds, filtered by `--branch` (`--json`)
|
|
44
|
-
| `aitk comments scan` | Measure comment density by language and comment kind, with a trend recomputed from git
|
|
45
|
-
| `aitk context audit` | Report required sections, length, cited paths, reference form, catalog tables, provenance, superseded-decision narration, and index drift
|
|
46
|
-
| `aitk markdown audit` | Fail any markdown path on a banned character, word, or spelling, and report the structural checkpoints
|
|
47
|
-
| `aitk claude skills audit` | Report both skill corpora against the mechanical rules in `standards/skill.md`
|
|
48
|
-
| `aitk claude skills drift` | Name the shipped skill bodies rewritten between a given ref and `HEAD`, and the installed version against the newest published (`--json`)
|
|
49
|
-
| `aitk gov test-order` | Report where an implementation reached history ahead of the test covering it (`--json`)
|
|
50
|
-
| `aitk
|
|
51
|
-
| `aitk
|
|
52
|
-
| `aitk
|
|
53
|
-
| `aitk
|
|
12
|
+
| Command | Purpose |
|
|
13
|
+
| -------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
14
|
+
| `aitk init [path]` | Bootstrap a project with selected toolkit domains |
|
|
15
|
+
| `aitk sync [path]` | Sync all installed domains in a target project |
|
|
16
|
+
| `aitk sync --check` | Report toolkit drift and the installed version against the newest published (`--json`, `--exit-code`) |
|
|
17
|
+
| `aitk sandbox [cat:cmd]` | Run sandbox scenarios (interactive or routed), toolkit-only like the tree it reads |
|
|
18
|
+
| `aitk sandbox reset` | Reset sandbox to baseline |
|
|
19
|
+
| `aitk sandbox clean` | Wipe the sandbox |
|
|
20
|
+
| `aitk sandbox check` | Score a provisioned sandbox against a scenario expectation (`--json` for the verdict) |
|
|
21
|
+
| `aitk sandbox coverage` | Report which scenarios declare expectations (`--json`, `--strict`, `--skills`) |
|
|
22
|
+
| `aitk indexes regen` | Regenerate `index.md` files from sibling frontmatter |
|
|
23
|
+
| `aitk docs [topic]` | Emit toolkit reference docs (`list`, or a topic by name) |
|
|
24
|
+
| `aitk design render` | Render `.claude/DESIGN.md` tokens to HTML and CSS |
|
|
25
|
+
| `aitk slides render` | Render a `.claude/SLIDES.md` source into a PowerPoint deck |
|
|
26
|
+
| `aitk slides list` | List the available slide layouts (`--json` for the catalog) |
|
|
27
|
+
| `aitk feedback` | Write toolkit feedback from stdin to `.claude/review/`, or open a GitHub issue with `--github` |
|
|
28
|
+
| `aitk transcripts <url>` | Fetch a YouTube transcript with metadata frontmatter (needs `yt-dlp`) |
|
|
29
|
+
| `aitk tasks archive` | Move a shipped task off the board, clear its ordering row, and regenerate the index |
|
|
30
|
+
| `aitk tasks pull-request` | Record a pull request number on the task a branch closes, by stem or `--plan` (`--json`) |
|
|
31
|
+
| `aitk tasks outcome` | Mark outcomes `[x]` on a task by position, repeating `--close` (`--json`) |
|
|
32
|
+
| `aitk tasks validate` | Report board rows whose plan, task file, group, file set, or blocker does not hold (`--json`) |
|
|
33
|
+
| `aitk intake list` | Report intake folder counts, or one folder's items, keeping what is unread with `--unread` (`--json`) |
|
|
34
|
+
| `aitk intake answer` | Write selections into one cluster's answer slots, repeating `--set <item>=<answer>` (`--json`) |
|
|
35
|
+
| `aitk teach list` | Report learning workspaces and the ordinal a new one takes, or what one workspace holds (`--json`) |
|
|
36
|
+
| `aitk teach open` | Open a workspace at the next ordinal with its mission, resources, and glossary files (`--json`) |
|
|
37
|
+
| `aitk teach resource` | Record sources and leads in a workspace, repeating `--read` or `--lead` as `<title>=<url>` (`--json`) |
|
|
38
|
+
| `aitk teach glossary` | Add terms to a workspace glossary alphabetically, repeating `--term <term>=<definition>` (`--json`) |
|
|
39
|
+
| `aitk records validate` | Report a session record or a standard against the standard governing it, per kind (`--json`) |
|
|
40
|
+
| `aitk records size` | Report what each record folder holds and how much of it is recent, heaviest first (`--json`) |
|
|
41
|
+
| `aitk records push` | Commit the nine backed record folders and push them to a private records remote (`--json`) |
|
|
42
|
+
| `aitk records pull` | Fetch the records remote and write it back, refusing rather than discarding unpushed records (`--json`) |
|
|
43
|
+
| `aitk sessions list` | Resolve live sessions to the worktree and branch each holds, filtered by `--branch` (`--json`) |
|
|
44
|
+
| `aitk comments scan` | Measure comment density by language and comment kind, with a trend recomputed from git |
|
|
45
|
+
| `aitk context audit` | Report required sections, length, cited paths, reference form, catalog tables, provenance, superseded-decision narration, and index drift |
|
|
46
|
+
| `aitk markdown audit` | Fail any markdown path on a banned character, word, or spelling, and report the structural checkpoints |
|
|
47
|
+
| `aitk claude skills audit` | Report both skill corpora against the mechanical rules in `standards/skill.md` |
|
|
48
|
+
| `aitk claude skills drift` | Name the shipped skill bodies rewritten between a given ref and `HEAD`, and the installed version against the newest published (`--json`) |
|
|
49
|
+
| `aitk gov test-order` | Report where an implementation reached history ahead of the test covering it (`--json`) |
|
|
50
|
+
| `aitk secrets scan` | Report credential-shaped values in the tree the package ships, keyed on issued values rather than on words (`--json`) |
|
|
51
|
+
| `aitk deps audit` | Report published advisories against the resolved dependency set, refusing rather than reporting clean when the index is unreachable (`--json`) |
|
|
52
|
+
| `aitk audits run` | Run every audit as one set, report per check under one verdict, and compare each count to the recorded baseline (`--json`, `--record`) |
|
|
53
|
+
| `aitk audits list` | List every audit the set runs, with the corpus each reads and whether it gates (`--json`) |
|
|
54
|
+
| `aitk capture [source]` | Render HTML capture sources to PNG, toolkit-only and absent from an installed package |
|
|
55
|
+
| `aitk upgrade` | Reinstall the CLI globally with the package manager the install path names (`--json`) |
|
|
54
56
|
|
|
55
57
|
## Domain commands
|
|
56
58
|
|
|
@@ -72,6 +74,8 @@ Each domain exposes a consistent shape where applicable: `list`, `install`, `syn
|
|
|
72
74
|
| `comments` | `scan` |
|
|
73
75
|
| `context` | `audit` |
|
|
74
76
|
| `markdown` | `audit` |
|
|
77
|
+
| `secrets` | `scan` |
|
|
78
|
+
| `deps` | `audit` |
|
|
75
79
|
| `audits` | `run`, `list` |
|
|
76
80
|
|
|
77
81
|
Common patterns:
|
package/docs/agents/index.md
CHANGED
|
@@ -26,6 +26,7 @@ CLI catalog and invocation rules for agents, split by command domain. Start with
|
|
|
26
26
|
- [Scripting](scripting.md): The runtime catalogs that replace hardcoded names, what each carries, and a headless invocation per domain
|
|
27
27
|
- [Sessions](sessions.md): Resolving live peer sessions to the worktree and branch each holds, the liveness confidence field, the unresolved reasons, and what the read depends on
|
|
28
28
|
- [Skill audit](skills-audit.md): Measuring both skill corpora against standards/skill.md, the checks it reads, the requirement gate that is the only failing one, and the drift verb that names bodies rewritten since a ref
|
|
29
|
+
- [State-scoped risk](state-scoped-risk.md): Reading committed state rather than an arriving change, the shipped-tree corpus the secret scan reads, what it keys on and how a deliberate value is exempted, the advisory check and its network failure mode, and why one gates while the other reports
|
|
29
30
|
- [Tasks](tasks.md): Selecting a shipped task by stem or pull request, recording a number and closing an outcome, the refusal reasons, the board and backlog checks validate runs, and why the board root defaults to the main worktree
|
|
30
31
|
- [Teach](teach.md): Listing learning workspaces and the ordinal a new one takes, opening one with its required files, recording sources and glossary terms, resolving what the next lesson needs before it is written, the refusal reasons, and why every write here runs through a verb
|
|
31
32
|
- [Test order](test-order.md): Reading where an implementation reached history ahead of its test, how a pair is decided, the three verdicts, the coverage the pairing cannot reach, and why the check reports rather than gates
|
package/docs/agents/scripting.md
CHANGED
|
@@ -129,8 +129,23 @@ SANDBOX_SCENARIO=sync aitk sandbox infra:tooling
|
|
|
129
129
|
|
|
130
130
|
# Read every audit as one record. Exit 2 is a fact, 3 an audit that did not report
|
|
131
131
|
aitk audits run --json
|
|
132
|
+
|
|
133
|
+
# Read committed state rather than an arriving change. Exit 2 carries findings
|
|
134
|
+
aitk secrets scan --json
|
|
135
|
+
aitk deps audit --json
|
|
132
136
|
```
|
|
133
137
|
|
|
138
|
+
The two state-scoped verbs both exit 2 on findings and mean different things by
|
|
139
|
+
it. A credential in the shipped tree is a fact and fails `aitk audits run`, while
|
|
140
|
+
a published advisory is a judgment that reports and moves no verdict. Neither
|
|
141
|
+
reads a diff, so a consumer already running a review surface gets no overlap.
|
|
142
|
+
|
|
143
|
+
Both refuse rather than report clean when they have no corpus, and the refusal
|
|
144
|
+
reason is the field to branch on. A project publishing nothing refuses the secret
|
|
145
|
+
scan with `no-manifest`, and one whose dependencies are not installed refuses the
|
|
146
|
+
advisory check with `no-lockfile`. The aggregate reads those as an absent corpus
|
|
147
|
+
rather than a broken run, so neither pins its verdict at `incomplete`.
|
|
148
|
+
|
|
134
149
|
`aitk tooling sync` is the one verb above whose flag is mandatory headlessly. It
|
|
135
150
|
overwrites every golden config a stack ships, which reaches the CI workflow, the
|
|
136
151
|
git hooks, the end-to-end harness, and the shell scripts under `scripts/`, so a
|
|
@@ -0,0 +1,105 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: State-scoped risk
|
|
3
|
+
description: Reading committed state rather than an arriving change, the shipped-tree corpus the secret scan reads, what it keys on and how a deliberate value is exempted, the advisory check and its network failure mode, and why one gates while the other reports
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# State-scoped risk
|
|
7
|
+
|
|
8
|
+
Every review surface a session can reach is scoped to a change. `claude-review` reads the branch diff, `claude-pr-review` reads a pull request, and `code-review` takes a diff, a branch, or a path. A risk that arrived before the range under review is invisible to all three by construction, which is what these two commands answer.
|
|
9
|
+
|
|
10
|
+
| Question | Command |
|
|
11
|
+
| -------------------------------------------------------- | ------------------- |
|
|
12
|
+
| Does a credential sit in the tree this repository ships? | `aitk secrets scan` |
|
|
13
|
+
| Does a resolved dependency carry a published advisory? | `aitk deps audit` |
|
|
14
|
+
|
|
15
|
+
Neither reads a range. That boundary is deliberate: change-scoped correctness review already exists, and a state-scoped bug scan would be a different product competing with it. Both register in `aitk audits run`, so a reader who runs the aggregate gets them without knowing they exist.
|
|
16
|
+
|
|
17
|
+
## Secrets in the shipped tree
|
|
18
|
+
|
|
19
|
+
```bash
|
|
20
|
+
aitk secrets scan
|
|
21
|
+
aitk secrets scan --json
|
|
22
|
+
aitk secrets scan ../my-app
|
|
23
|
+
```
|
|
24
|
+
|
|
25
|
+
The corpus is the package's own `files` field rather than a list the check keeps. That field is the single statement of which trees leave this repository, so a second list beside it would answer the same question and drift. It also carries the negations the publish already makes, which is what puts the sandbox tree, the eval tree, and every test file out of scope by the rule that keeps them out of the tarball rather than by an exclusion this check invented.
|
|
26
|
+
|
|
27
|
+
The plugin reaches a target by a different route, loading live from `claude/` rather than from a tarball. That folder is a `files` entry too, and its `standards` and `snippets` symlinks resolve into trees the field lists in their own right, so both routes land inside the same corpus.
|
|
28
|
+
|
|
29
|
+
Three root files are read whether or not the field names them, being `package.json`, the readme, and the license, since npm packs those on every publish. A negation still removes one, because a field that excludes a file outranks the default that included it.
|
|
30
|
+
|
|
31
|
+
### What the corpus leaves out
|
|
32
|
+
|
|
33
|
+
The corpus answers what the package publishes, which is narrower than what the repository holds. Measured against this repository on 2026-08-21, the scan read 544 files and left 593 of the 1139 git lists unread. Those include everything under `.claude/`, `wiki/`, and `internal/`, the workflow definitions under `.github/`, and the trees the publish negations remove, being `scripts/sandbox/`, `scripts/eval/`, `src/capture/`, and every test file.
|
|
34
|
+
|
|
35
|
+
A public repository makes that gap readable by anyone, so a clean run means no credential in the published tree rather than none in the repository. The run states the number on every pass, including a clean one, so the bound travels with the verdict. Widening the corpus to every tracked file is a separate decision, since the row this implements puts the shipped tree first on the record's rule that content leaving the repository gates harder than content that stays.
|
|
36
|
+
|
|
37
|
+
### What it keys on
|
|
38
|
+
|
|
39
|
+
Every pattern matches an issued value and none of them matches a word. A scan keyed on `password`, `secret`, or `token` fires on the environment reads, the workflow inputs, and the prose that name those things, and this repository ships all three. Keying on values instead is what makes the exclusion set empty: measured across the shipped tree on 2026-08-21, 544 files produced zero findings with nothing exempted.
|
|
40
|
+
|
|
41
|
+
The set covers issuer-stamped shapes, being AWS access key ids, GitHub tokens in both forms, Google API keys, Slack tokens and webhooks, Stripe live keys, Anthropic and OpenAI keys, npm tokens, and private key block headers. What it does not reach is a credential no issuer stamps recognizably, which no exclusion policy would have helped with either.
|
|
42
|
+
|
|
43
|
+
A reported value is redacted to its two ends. Those are what a reader needs to find it in the file and to tell one match from another, and the middle is the part no report should carry.
|
|
44
|
+
|
|
45
|
+
### Exempting a deliberate value
|
|
46
|
+
|
|
47
|
+
A line carrying a credential-shaped value on purpose takes an inline marker, either on the line itself or on the line directly above it:
|
|
48
|
+
|
|
49
|
+
```bash
|
|
50
|
+
# aitk-allow-secret: documented sample from the vendor's own reference
|
|
51
|
+
AWS_KEY="<the sample value>"
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
Only a marker naming a reason counts. A bare token reads as a line that meant to say something and did not, which is the rule `stub: true` already applies to a seed field set to anything but `true`, and honoring it would let a typo mute a finding.
|
|
55
|
+
|
|
56
|
+
The exemption travels with the line rather than sitting in a path list away from it, so a reader meeting a muted match finds the reason on the spot. A path allow-list was weighed and declined: the noise it would target is word-keyed and spread past the fixture trees, so it would hide part of the noise and none of the risk.
|
|
57
|
+
|
|
58
|
+
### Exit codes and gating
|
|
59
|
+
|
|
60
|
+
Exit codes are `0` when the shipped tree carries no credential-shaped value, `1` for a refusal, and `2` for at least one value found.
|
|
61
|
+
|
|
62
|
+
This is the one entry in `aitk audits run` that gates without a `verify.sh` stage behind it. A credential in the published tree is a fact rather than a judgment, which is the test the catalog asks any gating addition to pass, and the architecture record already ranks content leaving the repository above content that stays.
|
|
63
|
+
|
|
64
|
+
A refusal is never a clean tree. Five reasons produce one, and each exits `1`, because zero findings over zero files reads in the report exactly like zero findings over the whole shipped tree.
|
|
65
|
+
|
|
66
|
+
| Reason | What it means | In the aggregate |
|
|
67
|
+
| ------------------ | ------------------------------------------------------------ | ---------------- |
|
|
68
|
+
| `no-manifest` | No `package.json`, so nothing is published from this tree | absent |
|
|
69
|
+
| `no-publish` | The manifest declares `private`, so it publishes nothing | absent |
|
|
70
|
+
| `no-shipped-files` | The `files` field matched nothing git lists | absent |
|
|
71
|
+
| `no-files-field` | A publish would pack the whole tree, and none of it was read | unmeasured |
|
|
72
|
+
| `no-git` | git could not list the tree, so the corpus is unknown | unmeasured |
|
|
73
|
+
|
|
74
|
+
The split turns on whether a corpus exists. The first three mean this project publishes nothing, which is where most targets installing this CLI sit, so reporting them as unmeasured would pin the aggregate at `incomplete` there forever.
|
|
75
|
+
|
|
76
|
+
The last two mean a corpus exists and went unread, so neither is softened. `private` is what separates them, since a manifest with no `files` field publishes everything rather than nothing and the field alone cannot tell those apart. Declaring `private: true` is the way a project that never publishes says so.
|
|
77
|
+
|
|
78
|
+
## Advisories against the resolved dependencies
|
|
79
|
+
|
|
80
|
+
```bash
|
|
81
|
+
aitk deps audit
|
|
82
|
+
aitk deps audit --json
|
|
83
|
+
```
|
|
84
|
+
|
|
85
|
+
The check shells the runtime's own advisory command rather than carrying an index. A vendored advisory database is a second corpus to keep current, and what this is worth is the report rather than the data.
|
|
86
|
+
|
|
87
|
+
It refuses for three reasons, and each is an absence rather than a break: no `package.json`, no lockfile beside it, and no record back from the lookup. The lockfile is checked here rather than left to the underlying command, because a project whose dependencies were never resolved needs an install and not a retry, and one message naming the network would send half the readers at the wrong cause.
|
|
88
|
+
|
|
89
|
+
That buys the one failure mode no other audit here carries. The command reaches a network, so an unreachable index has to be told from a tree with nothing against it. The exit code cannot separate them, since the underlying command exits non-zero both on advisories found and on a lookup that failed, so the record on stdout is what decides: output that parses is a measurement, and output that does not is a refusal under `no-record`.
|
|
90
|
+
|
|
91
|
+
Exit codes are `0` when nothing is published against the resolved set, `1` for a refusal, and `2` for at least one advisory.
|
|
92
|
+
|
|
93
|
+
### Why it reports rather than gates
|
|
94
|
+
|
|
95
|
+
A published advisory is a fact about the index and a judgment about this tree, since the upgrade closing it may not exist yet. A push failing on one would teach a contributor to route around the stage while nothing about the dependency has changed, which is the split every other reporting measure here rests on.
|
|
96
|
+
|
|
97
|
+
The corpus is recorded as `upstream`, a third value beside `tracked` and `per-machine`. Its count moves when someone publishes rather than when someone edits here, so a retained baseline would report growth against a tree nobody touched, and the aggregate keeps it out of the record for the mirror image of the reason it keeps gitignored scratch out.
|
|
98
|
+
|
|
99
|
+
The same value decides what an offline run reports. An index this machine could not reach is an ordinary absence rather than a broken checkout, so it lands as `absent` and moves no verdict. Reading it as unmeasured would pin the aggregate at `incomplete` on every machine without a network, which is a signal nobody reads after the second time they see it.
|
|
100
|
+
|
|
101
|
+
## Reading either from a skill
|
|
102
|
+
|
|
103
|
+
An exit code says nothing about a call made from a session, since a shell profile may wrap the binary in a function taking its status from a later command. Read the `--json` record instead.
|
|
104
|
+
|
|
105
|
+
The secret scan publishes `findings`, `files`, `skipped`, and `listed`, where `listed` is everything git reports so a consumer can state the bound alongside the verdict. The advisory check publishes `advisories` and a `severities` object. A refusal from either publishes `reason` and `message` and no measurement keys at all, which is what separates it from a clean run.
|
package/package.json
CHANGED
|
@@ -15,8 +15,13 @@ require_sandbox_anchor_config() {
|
|
|
15
15
|
fi
|
|
16
16
|
}
|
|
17
17
|
|
|
18
|
+
resolve_sandbox_anchor_repo() {
|
|
19
|
+
printf '%s\n' "${1:-${ANCHOR_REPO:-}}"
|
|
20
|
+
}
|
|
21
|
+
|
|
18
22
|
sandbox_anchor_url() {
|
|
19
|
-
local repo_name
|
|
23
|
+
local repo_name
|
|
24
|
+
repo_name="$(resolve_sandbox_anchor_repo "$@")"
|
|
20
25
|
|
|
21
26
|
if [ -z "$repo_name" ] || [ -z "${GITHUB_ORG:-}" ]; then
|
|
22
27
|
log_error "sandbox_anchor_url needs GITHUB_ORG and a repository name. Call require_sandbox_anchor_config first."
|
|
@@ -52,7 +57,7 @@ configure_sandbox_git_identity() {
|
|
|
52
57
|
|
|
53
58
|
# Every scenario that needs a remote points at the same throwaway repository, so
|
|
54
59
|
# the name lives here rather than in each one.
|
|
55
|
-
SANDBOX_ANCHOR_REPO="
|
|
60
|
+
SANDBOX_ANCHOR_REPO="aitk-sandbox"
|
|
56
61
|
|
|
57
62
|
# A scenario calls this from its own use_anchor hook rather than this file
|
|
58
63
|
# defining the hook. manage-sandbox.sh keys off `type -t use_anchor`, so
|
|
@@ -62,11 +67,58 @@ use_sandbox_anchor() {
|
|
|
62
67
|
export ANCHOR_REPO="${1:-$SANDBOX_ANCHOR_REPO}"
|
|
63
68
|
}
|
|
64
69
|
|
|
70
|
+
# gh answers an absent repository and an unreachable host with the same exit
|
|
71
|
+
# status, so the 404 is the only thing separating them. Anything else is a
|
|
72
|
+
# network or credential fault, where creating a repository would be the wrong
|
|
73
|
+
# answer and would fail for the same reason the read did.
|
|
74
|
+
ensure_sandbox_anchor_repo() {
|
|
75
|
+
local repo_name gh_error
|
|
76
|
+
|
|
77
|
+
repo_name="$(resolve_sandbox_anchor_repo "$@")"
|
|
78
|
+
|
|
79
|
+
if [ -z "$repo_name" ] || [ -z "${GITHUB_ORG:-}" ]; then
|
|
80
|
+
log_error "ensure_sandbox_anchor_repo needs GITHUB_ORG and a repository name. Call require_sandbox_anchor_config first."
|
|
81
|
+
fi
|
|
82
|
+
|
|
83
|
+
if gh_error="$(gh api "repos/${GITHUB_ORG}/${repo_name}" --silent 2>&1)"; then
|
|
84
|
+
return 0
|
|
85
|
+
fi
|
|
86
|
+
|
|
87
|
+
case "$gh_error" in
|
|
88
|
+
*"HTTP 404"*) ;;
|
|
89
|
+
*) log_error "Cannot reach ${GITHUB_ORG}/${repo_name}: ${gh_error}" ;;
|
|
90
|
+
esac
|
|
91
|
+
|
|
92
|
+
# An absent anchor is nearly always a wrong GITHUB_ORG or a rename nobody
|
|
93
|
+
# performed, and provisioning stages a fixture rather than cloning while every
|
|
94
|
+
# scenario force-pushes to main, so a repository created here would carry all
|
|
95
|
+
# nine to a pass against something that is not the anchor. Refusing is the safe
|
|
96
|
+
# default and creating is the opt-in, the shape `aitk tooling sync --write`
|
|
97
|
+
# already sets. `aitk records push` refuses outright for the reason
|
|
98
|
+
# `.claude/context/development/scratch.md` records, so the two still differ.
|
|
99
|
+
# The sibling SANDBOX_ flags are presence tests, so any non-empty value turns
|
|
100
|
+
# them on. This one allowlists instead, because a presence test would have
|
|
101
|
+
# SANDBOX_ANCHOR_CREATE=false provisioning a repository. Both spellings are
|
|
102
|
+
# accepted so the `true` those siblings are set to does not land on a refusal.
|
|
103
|
+
case "${SANDBOX_ANCHOR_CREATE:-}" in
|
|
104
|
+
1 | true) ;;
|
|
105
|
+
*) log_error "${GITHUB_ORG}/${repo_name} does not exist. Check GITHUB_ORG and whether a rename is pending, then create it with 'gh repo create ${GITHUB_ORG}/${repo_name} --private' or re-run with SANDBOX_ANCHOR_CREATE=true." ;;
|
|
106
|
+
esac
|
|
107
|
+
|
|
108
|
+
log_warn "${GITHUB_ORG}/${repo_name} does not exist and SANDBOX_ANCHOR_CREATE is set. Creating it as private."
|
|
109
|
+
if ! gh_error="$(gh repo create "${GITHUB_ORG}/${repo_name}" --private 2>&1)"; then
|
|
110
|
+
log_error "Could not create ${GITHUB_ORG}/${repo_name}: ${gh_error}"
|
|
111
|
+
fi
|
|
112
|
+
log_info "Created ${GITHUB_ORG}/${repo_name} as a private repository."
|
|
113
|
+
}
|
|
114
|
+
|
|
65
115
|
# A remote is useless without an author, so the scenarios that reach one always
|
|
66
116
|
# configure both. configure_sandbox_git_identity stays callable on its own for
|
|
67
|
-
# the scenarios that never push. The
|
|
68
|
-
#
|
|
117
|
+
# the scenarios that never push. The probe runs first so an absent remote is
|
|
118
|
+
# reported before the scenario stages anything. The remove keeps this idempotent
|
|
119
|
+
# against a sandbox tree that already carries an origin.
|
|
69
120
|
configure_sandbox_anchor_remote() {
|
|
121
|
+
ensure_sandbox_anchor_repo "$@"
|
|
70
122
|
configure_sandbox_git_identity
|
|
71
123
|
git remote remove origin 2>/dev/null || true
|
|
72
124
|
git remote add origin "$(sandbox_anchor_url "$@")"
|
package/src/audits/baseline.ts
CHANGED
|
@@ -53,6 +53,8 @@ export type Delta =
|
|
|
53
53
|
| { readonly id: string; readonly kind: 'unrecorded' }
|
|
54
54
|
/** Gitignored scratch, whose counts are one machine's and answer nobody else. */
|
|
55
55
|
| { readonly id: string; readonly kind: 'per-machine' }
|
|
56
|
+
/** A network index, whose count moves when someone publishes rather than edits. */
|
|
57
|
+
| { readonly id: string; readonly kind: 'upstream' }
|
|
56
58
|
/** The audit did not report, so there is nothing to compare. */
|
|
57
59
|
| { readonly id: string; readonly kind: 'unmeasured' }
|
|
58
60
|
|
|
@@ -62,7 +64,9 @@ export type Delta =
|
|
|
62
64
|
* Only a tracked corpus is retained. A gitignored record folder holds one
|
|
63
65
|
* machine's session scratch, so committing its counts writes a floor no other
|
|
64
66
|
* clone can reproduce, and every contributor would read a regression against a
|
|
65
|
-
* number that describes somebody else's disk.
|
|
67
|
+
* number that describes somebody else's disk. An upstream count is left out
|
|
68
|
+
* for the mirror-image reason: it moves when an advisory is published, so a
|
|
69
|
+
* recorded floor would report growth against a tree nobody touched.
|
|
66
70
|
*
|
|
67
71
|
* An audit that did not report is left out rather than written as zero. Zeros
|
|
68
72
|
* there record a clean corpus nobody measured, and the next run reads its real
|
|
@@ -95,6 +99,9 @@ export function compareBaseline(
|
|
|
95
99
|
results: readonly AuditResult[],
|
|
96
100
|
): Delta[] {
|
|
97
101
|
return results.map((result) => {
|
|
102
|
+
if (result.corpus === 'upstream') {
|
|
103
|
+
return { id: result.id, kind: 'upstream' as const }
|
|
104
|
+
}
|
|
98
105
|
if (!result.tracked) return { id: result.id, kind: 'per-machine' as const }
|
|
99
106
|
if (result.counts === undefined) {
|
|
100
107
|
return { id: result.id, kind: 'unmeasured' as const }
|