task-pipeline-skill 1.74.0 → 1.76.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +148 -0
- package/README.md +1 -0
- package/SKILL-CARD.md +1 -1
- package/package.json +5 -4
- package/plugins/task-pipeline/.claude-plugin/plugin.json +1 -1
- package/plugins/task-pipeline/commands/project-audit.md +23 -0
- package/plugins/task-pipeline/skills/project-audit/SKILL.md +202 -0
- package/plugins/task-pipeline/skills/project-audit/references/probes.md +199 -0
- package/plugins/task-pipeline/skills/project-audit/scripts/audit.py +1267 -0
- package/plugins/task-pipeline/skills/task-pipeline/references/artifacts.md +8 -0
- package/plugins/task-pipeline/skills/task-pipeline/references/audit.md +8 -0
- package/plugins/task-pipeline/skills/task-pipeline/references/documentation.md +17 -3
- package/plugins/task-pipeline/skills/task-pipeline/references/gates.md +92 -0
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,153 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## v1.76.0 — the audit that starts cold
|
|
4
|
+
|
|
5
|
+
**`project-audit` ships as this plugin's third skill.** `references/audit.md` has
|
|
6
|
+
been the family's audit *method* since v1.20.0 — the L0→L7 ladder, seams over
|
|
7
|
+
artifacts, axis rotation, the exit criterion. It assumes a brief with REQ rows
|
|
8
|
+
and a module map, which most repositories do not have; pointed at one that does
|
|
9
|
+
not, its true first finding is *"your spine is missing"*, and that is not an
|
|
10
|
+
audit. The new skill is the **procedure** around the method: discover what the
|
|
11
|
+
project is, probe it from that, read the production evidence a repository cannot
|
|
12
|
+
hold, hand phase 4 back to the ladder, and leave two artefacts. The method is
|
|
13
|
+
not copied — a second copy would be a second rule.
|
|
14
|
+
|
|
15
|
+
**Probes return three verdicts, and the third one is the design.** `clean`,
|
|
16
|
+
`finding`, `blind`. A probe whose need is unmet, whose command is missing or
|
|
17
|
+
whose output is empty returns `blind` **with the reason**, and the reason reaches
|
|
18
|
+
the page as a section of its own rather than an appendix. Without it, "no error
|
|
19
|
+
tracker is configured" and "no errors" render identically and a reader takes the
|
|
20
|
+
second meaning. This is `audit.md`'s *silence is not a reading* raised from a
|
|
21
|
+
command to a probe.
|
|
22
|
+
|
|
23
|
+
**The class a version check cannot see.** A registry serves the **tag**; a plugin
|
|
24
|
+
marketplace and a skills CLI serve the **branch tip**. Measured in this family on
|
|
25
|
+
2026-08-22: npm served one file at 4344 lines while the marketplace served it at
|
|
26
|
+
4575, and all three channels answered `1.15.0` — with the pin checker green
|
|
27
|
+
throughout, correctly, because it compared the two strings. `channel-divergence`
|
|
28
|
+
compares **trees**, and three of its traps are fixtures rather than prose:
|
|
29
|
+
|
|
30
|
+
- the first draft compared the tarball against the **tag** and reported clean.
|
|
31
|
+
Those agree by construction, because the registry publishes *from* the tag —
|
|
32
|
+
a tautology returning green, which is the *false success* shape `gates.md`
|
|
33
|
+
names;
|
|
34
|
+
- counting a path present in one channel and absent from the other produced
|
|
35
|
+
**22 findings** on a member where one file had moved. A tarball ships what its
|
|
36
|
+
`files` allowlist permits; that is packaging, not divergence;
|
|
37
|
+
- a branch already bumped past its tag makes no common claim, so it is `blind`.
|
|
38
|
+
|
|
39
|
+
Re-derived against a differently-shaped command: `git diff --name-only
|
|
40
|
+
v1.15.0..HEAD` returns the same eight paths the probe reports for `agent-sync`,
|
|
41
|
+
`sheleg-design` sits exactly on its tag and the probe is silent, and `make-skill`
|
|
42
|
+
is one commit past and the probe returns one.
|
|
43
|
+
|
|
44
|
+
**Two artefacts, and the second is what makes it a ratchet.**
|
|
45
|
+
`docs/audit/<date>-audit.html` for a person and `docs/audit/<date>-audit.json`
|
|
46
|
+
for the next run. A finding's id is derived from its probe and its place, so it
|
|
47
|
+
survives a rewording; the next audit prints what closed, what is new, and what
|
|
48
|
+
has now survived three runs — which is itself a finding, because a defect nobody
|
|
49
|
+
picks up is a decision nobody wrote down.
|
|
50
|
+
|
|
51
|
+
**Read-only, and the report is shareable by construction.** Findings leave as
|
|
52
|
+
proposed board rows priced with the project's own formula; nothing is written.
|
|
53
|
+
The page carries aggregates and pointers, never raw bodies — a secret is reported
|
|
54
|
+
by `file:line` and class with the value in neither artefact nor on stdout, and
|
|
55
|
+
the redaction is total rather than a prefix.
|
|
56
|
+
|
|
57
|
+
**Found by the fixtures rather than by reading**, and both are now checks:
|
|
58
|
+
|
|
59
|
+
- the audit read **its own output** as project state. Run 1 saw a clean tree; run
|
|
60
|
+
2 saw `docs/audit/` and reported the project dirty. Standing instruction #2 —
|
|
61
|
+
prove idempotence at the layer that repeats — is exactly this shape, and the
|
|
62
|
+
three-run fixture is what surfaced it;
|
|
63
|
+
- the obvious fix was worse than the bug. `git status --porcelain` collapses an
|
|
64
|
+
untracked directory to its shallowest path, so excluding by path silently
|
|
65
|
+
failed to match; widening the match to *either is a prefix of the other* would
|
|
66
|
+
have hidden every new file under `docs/`. `-uall` asks git for the full list
|
|
67
|
+
instead, and a fixture holds the sibling case.
|
|
68
|
+
|
|
69
|
+
**The refusal phrase is «без диагностики», not «без аудита».** `аудит` is this
|
|
70
|
+
plugin's own trigger — kept deliberately after re-derivation — and
|
|
71
|
+
`triggers_test.js` refuses any refusal phrase containing a trigger, because
|
|
72
|
+
saying it would fire the hook it exists to silence.
|
|
73
|
+
|
|
74
|
+
`npm test` now runs three suites: the validator, `graph.py`'s 149 cases and
|
|
75
|
+
`project_audit_test.py`'s 43. The validator refused the release until the new
|
|
76
|
+
suite joined both documented equations — a suite outside the full run is a suite
|
|
77
|
+
CI does not have.
|
|
78
|
+
|
|
79
|
+
Guards: 412 → **412**. Flat, and the flatness is the point: every check this
|
|
80
|
+
release adds guards the shipped collector, so it lives in `project_audit_test.py`
|
|
81
|
+
where the code it exercises is, not in the structural validator. Counting them
|
|
82
|
+
as validator guards would inflate a number whose whole job is to make a dormant
|
|
83
|
+
check visible.
|
|
84
|
+
|
|
85
|
+
## v1.75.0 — 2026-08-22 — the day's findings become doctrine
|
|
86
|
+
|
|
87
|
+
Nothing new ships behind a flag here; what ships is the doctrine the previous release
|
|
88
|
+
earned, written where the next run will read it instead of in a changelog nobody greps.
|
|
89
|
+
|
|
90
|
+
**`documentation.md` canon 2 gains its dual.** *Numbers are computed, never restated* had
|
|
91
|
+
a half nobody had written down: **an example that instantiates a number IS one.** A release
|
|
92
|
+
note explaining that a count was written in the wrong shape — with the digits in it — places
|
|
93
|
+
a second readable count in a section a gate reads, so a probe that removes the real one
|
|
94
|
+
leaves the narrative matching and the guard silent. Measured three times in one hour, each
|
|
95
|
+
time inside prose *about* this very failure. The umbrella already said it for commands:
|
|
96
|
+
name a dead command, never claim it. It holds for a number, a version and a shape.
|
|
97
|
+
|
|
98
|
+
**`gates.md` gains three sections.**
|
|
99
|
+
|
|
100
|
+
*A probe rots, and every way it rots reports green.* The three assertions prove a probe
|
|
101
|
+
works today; the thing it guards is the thing that moves it. An anchor pinned to a literal
|
|
102
|
+
dies on the release that changes the number it guards. A precondition inherited from the
|
|
103
|
+
tree evaporates **when the system works correctly** — an honest run stamp leaves nothing
|
|
104
|
+
"after the newest stamp", a release absorbs the `## Unreleased` section — and the probe
|
|
105
|
+
lands and proves nothing. The triage is one question: *what does this probe look for, and
|
|
106
|
+
who is allowed to change it?*
|
|
107
|
+
|
|
108
|
+
*A ratchet prices the rule, not the exception.* A coverage check that asserts once per
|
|
109
|
+
exception and falls through on the ordinary case makes the correct remediation lower the
|
|
110
|
+
count — so the gate goes red on the stricter answer and a legitimate lowering becomes
|
|
111
|
+
indistinguishable from the failure the floor exists to catch. One assertion per subject
|
|
112
|
+
examined, whatever its verdict. And measure the floor **after** the last edit: read, keep
|
|
113
|
+
editing, restate is how every floor ends up below its true count, silently, because a floor
|
|
114
|
+
is a minimum.
|
|
115
|
+
|
|
116
|
+
*Run the whole suite locally before you push the tag.* Also **R-010**, because it is a
|
|
117
|
+
run-level obligation rather than a note. One tag took five CI rounds where the last four
|
|
118
|
+
were a single twelve-minute local run apart — and the second half is what makes it a rule:
|
|
119
|
+
a branch push cannot see a tag that does not exist yet, so three checks have no earlier
|
|
120
|
+
chance to fire.
|
|
121
|
+
|
|
122
|
+
### And R-010 failed on its own release, which is the finding
|
|
123
|
+
|
|
124
|
+
The local suite was green and CI was not. The difference was one precondition asking
|
|
125
|
+
`os.path.isdir(".git")` — false in a **submodule** checkout, where `.git` is a file holding a
|
|
126
|
+
gitdir pointer. The whole release-gap check had been switching itself off in the only checkout
|
|
127
|
+
this family is developed in, since the day it was written, with no line of output. It ran in CI
|
|
128
|
+
alone, which clones standalone, so a class of defect reached four tag pushes before anyone asked
|
|
129
|
+
why the local run was green.
|
|
130
|
+
|
|
131
|
+
This repository had already recorded that class **twice**, in `docgate.sh` and in the retro log,
|
|
132
|
+
both naming `[ -d .git ]` as the wrong question. This instance was missed both times. Knowing a
|
|
133
|
+
class is not sweeping it — which is standing instruction R-003, also already in force.
|
|
134
|
+
|
|
135
|
+
Two fixes, and the second is the general one: ask `exists`, never `isdir`, of anything named
|
|
136
|
+
`.git`; and **a precondition that fails must disclose rather than skip**. A check guarded by a
|
|
137
|
+
bare `and` evaporates without output, and it evaporates most reliably in the environment its
|
|
138
|
+
authors work in. R-010 gains the half it learned by failing: *a green local suite is not
|
|
139
|
+
evidence until you know which checks looked* — tag locally first, run, then read the `unlooked`
|
|
140
|
+
line before believing the exit code.
|
|
141
|
+
|
|
142
|
+
Seven board citations into `gates.md` were re-pointed across two passes, which is the ordinary
|
|
143
|
+
cost of inserting text above them and the reason those citations carry a phrase as well as a
|
|
144
|
+
range.
|
|
145
|
+
|
|
146
|
+
Guards: 412 → **412**. Flat by construction: this release adds doctrine, not checks, and a
|
|
147
|
+
guard count that rose on a documentation change would be a number borrowed from a suite that
|
|
148
|
+
never ran. The shape of that line is the one canon 2's new half is about, so it is written
|
|
149
|
+
here in the form the guard reads and described nowhere else in this section.
|
|
150
|
+
|
|
3
151
|
## v1.74.0 — 2026-08-20 — a node is closed by three readings, not one
|
|
4
152
|
|
|
5
153
|
**A verifier reads the diff it was handed, and that is the definition of its context, not a
|
package/README.md
CHANGED
|
@@ -147,6 +147,7 @@ until it is installed.
|
|
|
147
147
|
| Stage | Built-in doctrine |
|
|
148
148
|
|---|---|
|
|
149
149
|
| **any claim read as true** | [`evidence-docs`](plugins/task-pipeline/skills/evidence-docs/SKILL.md) — **a second skill in this plugin**: the ten canons as an index, and where each is defined, enforced and seeded |
|
|
150
|
+
| **a whole project as the subject** | [`project-audit`](plugins/task-pipeline/skills/project-audit/SKILL.md) — **a third skill in this plugin**: a cold-start audit with no brief required. Discovery, a probe registry whose unmet needs read `blind` rather than clean, production evidence including one artefact served under two channels, and an HTML report beside a JSON sidecar that says what moved. Read-only. `/project-audit` |
|
|
150
151
|
| 0 Knowledge harvest | [`knowledge-sources.md`](plugins/task-pipeline/skills/task-pipeline/references/knowledge-sources.md) — source list, the wiki, the ledger, the stage-9 loop-back |
|
|
151
152
|
| 0 Intake grill | [`grill.md`](plugins/task-pipeline/skills/task-pipeline/references/grill.md) — interview loop, domain awareness, autonomy sweep |
|
|
152
153
|
| 2 Brainstorm | [`brainstorm.md`](plugins/task-pipeline/skills/task-pipeline/references/brainstorm.md) — approaches, YAGNI, the no-code-before-approval gate |
|
package/SKILL-CARD.md
CHANGED
|
@@ -12,7 +12,7 @@ harmless.
|
|
|
12
12
|
|---|---|
|
|
13
13
|
| **Purpose** | Runs a substantial task through ten gated delivery stages — intake grill, docs study, brainstorm, spec, plan, subagent build, tests, lint/deploy, post-deploy, docs+registers, acceptance — refusing to advance until each gate passes |
|
|
14
14
|
| **Owner** | ssheleg ([github.com/ssheleg/task-pipeline](https://github.com/ssheleg/task-pipeline)) |
|
|
15
|
-
| **Version** | 1.
|
|
15
|
+
| **Version** | 1.76.0 |
|
|
16
16
|
| **Surface** | Claude Code (filesystem skill + plugin) and the vercel `skills` CLI. **Not** uploaded to the Skills API; custom Skills do not sync across surfaces |
|
|
17
17
|
| **Dependencies** | None required. Optional: `context7` (MCP), `figma` (MCP), super-ux, agent-sync, graphify, obsidian-wiki, and **one of two browser channels** — `playwright` (CLI or MCP) or `chrome-devtools` (MCP); either satisfies the browser step and neither is required. Every stage's doctrine ships in-repo; the one conditional requirement is super-ux for the stage-3 UX track on a user-facing task |
|
|
18
18
|
| **Evaluation status** | Suite authored, 5 categories. One recorded run, **self-observed by the author**; **zero blind runs on zero of three models** — the split, and the numbers, live in [`evals/RESULTS.md`](evals/RESULTS.md) and are computed by `evals/run.py` |
|
package/package.json
CHANGED
|
@@ -1,20 +1,21 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "task-pipeline-skill",
|
|
3
|
-
"version": "1.
|
|
3
|
+
"version": "1.76.0",
|
|
4
4
|
"description": "Full-cycle delivery pipeline for coding agents: a mandatory built-in intake grill, then 10 gated stages (docs, brainstorm+decompose, spec, plan, build, tests, lint/deploy, post-deploy, docs/wiki, acceptance). Every stage's doctrine ships inside the skill — no companion plugin required. This package is the installer CLI.",
|
|
5
5
|
"bin": {
|
|
6
6
|
"task-pipeline": "bin/task-pipeline.js"
|
|
7
7
|
},
|
|
8
8
|
"scripts": {
|
|
9
|
-
"test": "python3 test/validate.py && python3 test/graph_test.py",
|
|
10
|
-
"test:all": "python3 test/validate.py && python3 test/graph_test.py && python3 test/negatives.py && npm run test:certify && npm run test:exposure && npm run test:probe && npm run test:hooks && npm run test:artifacts && npm run test:docs",
|
|
9
|
+
"test": "python3 test/validate.py && python3 test/graph_test.py && python3 test/project_audit_test.py",
|
|
10
|
+
"test:all": "python3 test/validate.py && python3 test/graph_test.py && python3 test/project_audit_test.py && python3 test/negatives.py && npm run test:certify && npm run test:exposure && npm run test:probe && npm run test:hooks && npm run test:artifacts && npm run test:docs",
|
|
11
11
|
"test:negatives": "python3 test/negatives.py",
|
|
12
12
|
"test:exposure": "python3 test/exposure_test.py",
|
|
13
13
|
"test:probe": "python3 test/probe.py --self-test",
|
|
14
14
|
"test:hooks": "python3 test/release_gate_test.py",
|
|
15
15
|
"test:artifacts": "python3 test/artifact_root_test.py && python3 test/migrate_artifacts_test.py",
|
|
16
16
|
"test:docs": "bash plugins/task-pipeline/skills/task-pipeline/templates/docgate.sh",
|
|
17
|
-
"test:certify": "python3 test/certify_mutations.py"
|
|
17
|
+
"test:certify": "python3 test/certify_mutations.py",
|
|
18
|
+
"test:audit": "python3 test/project_audit_test.py"
|
|
18
19
|
},
|
|
19
20
|
"files": [
|
|
20
21
|
"bin",
|
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
"name": "task-pipeline",
|
|
3
3
|
"displayName": "Task Pipeline",
|
|
4
4
|
"description": "Runs a substantial task through a mandatory built-in intake grill, then 10 gated stages (docs, brainstorm+decompose, spec, plan, subagent build, tests, lint/deploy, post-deploy, docs/wiki, acceptance). Every stage's doctrine is built into the skill — no companion plugin required — with typed auto/manual gates, a frozen requirement spine that closes with evidence, a work board and a verification ledger that outlive a run, an exposure line naming what shipped unconfirmed, a progress rail computed from the project's own config, a loop guard whose review ceiling measures rather than stops, and stage-3 tracks for what a product does, how it sounds and how it looks. Two modes need no task: `checkup` (what is unverified) and `setup` (audit existing docs). Retro insights can publish upstream as issues, opt-in and redacted.",
|
|
5
|
-
"version": "1.
|
|
5
|
+
"version": "1.76.0",
|
|
6
6
|
"author": {
|
|
7
7
|
"name": "ssheleg",
|
|
8
8
|
"url": "https://x.com/sshlg93"
|
|
@@ -0,0 +1,23 @@
|
|
|
1
|
+
---
|
|
2
|
+
description: Audit a whole project — discover what it is, probe it, read production evidence, and leave an HTML report plus a JSON sidecar. Read-only.
|
|
3
|
+
argument-hint: "[path] (default: the current project)"
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
Run the `project-audit` skill against $ARGUMENTS (default: this project).
|
|
7
|
+
|
|
8
|
+
Follow its six phases in order — discover, probe, prod, seams, report, propose —
|
|
9
|
+
and do not skip discovery: a fixed checklist returns the same findings on every
|
|
10
|
+
repository it is pointed at, which is a fact about the checklist.
|
|
11
|
+
|
|
12
|
+
Start with the mechanical half, then spend your reading where it could not look:
|
|
13
|
+
|
|
14
|
+
```bash
|
|
15
|
+
python3 "${CLAUDE_PLUGIN_ROOT}/skills/project-audit/scripts/audit.py" --root .
|
|
16
|
+
```
|
|
17
|
+
|
|
18
|
+
It writes `docs/audit/<date>-audit.html` and `docs/audit/<date>-audit.json` and
|
|
19
|
+
opens the page. Read the sidecar's `probes` array before saying anything is
|
|
20
|
+
clean — every `blind` entry is a question this run did not answer, and its
|
|
21
|
+
reason is why.
|
|
22
|
+
|
|
23
|
+
Close on the skill's exit criterion. **Propose board rows; write nothing.**
|
|
@@ -0,0 +1,202 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: project-audit
|
|
3
|
+
description: "Use when someone asks what is actually true of a whole project right now — what is finished, what is half-built, what is broken, and what nobody has looked at. Walks a cold start: discover what the project is, run a registry of probes chosen from that, read production evidence (published artefact against source, CI history, telemetry present or absent), then leave a self-contained HTML report and a JSON sidecar so the next audit can say what moved. Read-only: it proposes board rows and commits nothing. Triggers - 'project audit', 'audit the project', 'codebase audit', 'state of the project', 'what is unfinished', 'project health check', 'аудит проекта', 'проаудируй проект', 'состояние проекта', 'что не доделано', 'аудит кодовой базы'. Not for: auditing one deliverable inside a run (that is the pipeline's own ladder), reviewing a diff, or checking a skill's construction — say 'без диагностики' to opt out."
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Project audit — what is true of this project right now
|
|
7
|
+
|
|
8
|
+
**A gate asks whether an artifact is good. Acceptance asks whether anything on
|
|
9
|
+
the list was lost. Neither asks what the project actually is today**, and that
|
|
10
|
+
is the only question an operator has when they open a repository they have not
|
|
11
|
+
touched for a month.
|
|
12
|
+
|
|
13
|
+
This skill answers it from a cold start — no brief, no REQ table, no module map
|
|
14
|
+
required — and leaves two artefacts: a page a person reads, and a sidecar the
|
|
15
|
+
next audit reads.
|
|
16
|
+
|
|
17
|
+
## Contents
|
|
18
|
+
|
|
19
|
+
- What this is not
|
|
20
|
+
- The six phases
|
|
21
|
+
- Three verdicts, and why the third one exists
|
|
22
|
+
- The class that version checks cannot see
|
|
23
|
+
- The two artefacts
|
|
24
|
+
- What the script does and what you do
|
|
25
|
+
- Exit criterion
|
|
26
|
+
- Rationalizations
|
|
27
|
+
|
|
28
|
+
## What this is not
|
|
29
|
+
|
|
30
|
+
| | Owns | Reach for it when |
|
|
31
|
+
|---|---|---|
|
|
32
|
+
| [`references/audit.md`](../task-pipeline/references/audit.md) | the **method** — the L0→L7 ladder, seams, axis rotation | one deliverable is being walked inside a run |
|
|
33
|
+
| this skill | the **procedure** — cold start, probes, production, the report | a whole project is the subject |
|
|
34
|
+
| `/skill-audit` (make-skill) | a skill's construction against the standard | the thing audited is a skill or plugin |
|
|
35
|
+
| `/ux-audit` (super-ux) | code against documented scenarios | the question is user-facing behaviour |
|
|
36
|
+
|
|
37
|
+
**The method is not restated here.** Phase 4 below hands off to `audit.md` and
|
|
38
|
+
comes back; a second copy of the ladder would be a second rule, and the two
|
|
39
|
+
would disagree within a release.
|
|
40
|
+
|
|
41
|
+
## The six phases
|
|
42
|
+
|
|
43
|
+
Run them in order. Each writes into the same payload the report is rendered
|
|
44
|
+
from.
|
|
45
|
+
|
|
46
|
+
### 1. Discover — what this project IS, before anything is measured
|
|
47
|
+
|
|
48
|
+
`scripts/audit.py` answers this mechanically: languages, package managers,
|
|
49
|
+
monorepo shape, submodules, CI, deploy targets, declared telemetry, and the
|
|
50
|
+
version the project states about itself. It reads **committed** state — `git
|
|
51
|
+
ls-files`, never a directory walk — because a walk finds `node_modules` and
|
|
52
|
+
build residue, and reports a project no clone would produce.
|
|
53
|
+
|
|
54
|
+
The output is a profile, and the profile chooses the probes. Skipping this and
|
|
55
|
+
running a fixed checklist is how an audit produces the same nine findings on
|
|
56
|
+
every repository it is ever pointed at.
|
|
57
|
+
|
|
58
|
+
### 2. Probe — the registry, chosen from the profile
|
|
59
|
+
|
|
60
|
+
Every probe declares what it needs (`git`, `gh`, `npm`, `network`, a connected
|
|
61
|
+
MCP server). A need that is not met makes the probe **blind**, with the reason —
|
|
62
|
+
never absent, and never clean. The catalogue by stack, and what to add for a
|
|
63
|
+
project shape the script does not cover, is
|
|
64
|
+
[`references/probes.md`](references/probes.md).
|
|
65
|
+
|
|
66
|
+
### 3. Prod — the evidence that only exists outside the tree
|
|
67
|
+
|
|
68
|
+
This is the phase most audits skip, and the one that finds what the repository
|
|
69
|
+
cannot admit about itself:
|
|
70
|
+
|
|
71
|
+
- **the published artefact against the source** — see the next section;
|
|
72
|
+
- **CI history**: not "is it green" but what share of *release* runs failed, and
|
|
73
|
+
whether a failure was noticed. `gh run list --limit 60 --json conclusion,name`
|
|
74
|
+
is one command and it is the closest thing to a production log a package has;
|
|
75
|
+
- **telemetry, present or absent.** Absent is a finding only when something is
|
|
76
|
+
deployed; for a library it is a design. Either way the report says which,
|
|
77
|
+
because "no Sentry configured" and "no errors" must not render the same;
|
|
78
|
+
- **adoption**, where a registry serves it — downloads make "is this in prod"
|
|
79
|
+
a measurement rather than an opinion;
|
|
80
|
+
- **connected MCP servers** — error trackers, analytics, hosting, databases.
|
|
81
|
+
Ask what the project uses, use what is connected, and record the rest as
|
|
82
|
+
blind. Read aggregates and pointers, never raw bodies (see *The two
|
|
83
|
+
artefacts*).
|
|
84
|
+
|
|
85
|
+
### 4. Seams — hand off to the ladder
|
|
86
|
+
|
|
87
|
+
Now, and only now, walk `audit.md`'s ladder over the capabilities the discovery
|
|
88
|
+
found. Bottom-up, seam-ordered. Absences found here are findings like any other.
|
|
89
|
+
|
|
90
|
+
### 5. Report — two files, one command
|
|
91
|
+
|
|
92
|
+
`python3 scripts/audit.py --root <path>` writes both and opens the page. It is
|
|
93
|
+
the same script the probes live in, so what the report claims and what ran are
|
|
94
|
+
the same object.
|
|
95
|
+
|
|
96
|
+
### 6. Propose — rows, not edits
|
|
97
|
+
|
|
98
|
+
**This skill commits nothing.** Findings leave as board rows in the project's
|
|
99
|
+
own vocabulary, priced with the project's own formula —
|
|
100
|
+
`P = blast × (1 + age_runs) / effort` — and the operator accepts them. An audit
|
|
101
|
+
that edits while it reads cannot be re-run to check itself.
|
|
102
|
+
|
|
103
|
+
## Three verdicts, and why the third one exists
|
|
104
|
+
|
|
105
|
+
`clean` · `finding` · `blind`. The vocabulary is closed; a fourth value is
|
|
106
|
+
refused at construction.
|
|
107
|
+
|
|
108
|
+
**`blind` is the whole design.** Without it, a probe that could not look and a
|
|
109
|
+
probe that found nothing produce the same empty section, and a reader takes the
|
|
110
|
+
second meaning every time. This is `audit.md`'s *silence is not a reading*
|
|
111
|
+
raised from a command to a probe: a zero exit with no output has not answered.
|
|
112
|
+
|
|
113
|
+
The page renders the blind list as a section of its own, never an appendix.
|
|
114
|
+
**An audit's blind spots are part of its result.**
|
|
115
|
+
|
|
116
|
+
## The class that version checks cannot see
|
|
117
|
+
|
|
118
|
+
**One version string, more than one tree.** A package's channels do not all
|
|
119
|
+
serve the same thing: a registry serves the **tag**, while a plugin marketplace
|
|
120
|
+
and a skills CLI serve the **branch tip**. When the branch has moved past the
|
|
121
|
+
tag without a version bump, every channel answers the same number and ships
|
|
122
|
+
different code.
|
|
123
|
+
|
|
124
|
+
Measured in this family on 2026-08-22: npm served one file at 4344 lines while
|
|
125
|
+
the marketplace served it at 4575, and all three channels reported `1.15.0`. The
|
|
126
|
+
pin checker was green throughout — correctly, because it compared the two
|
|
127
|
+
strings.
|
|
128
|
+
|
|
129
|
+
**So compare trees, never labels**, and compare the pair that can actually
|
|
130
|
+
disagree. The first draft of this probe compared the registry tarball against
|
|
131
|
+
the tag and reported clean: those agree by construction, because the registry
|
|
132
|
+
publishes *from* the tag. A tautology returning green is the *false success*
|
|
133
|
+
shape — a mechanism trusted by its own reply.
|
|
134
|
+
|
|
135
|
+
Two more traps, both of which shipped in the first draft and are now fixtures:
|
|
136
|
+
|
|
137
|
+
- **A path in one channel and not the other is packaging, not divergence.** A
|
|
138
|
+
tarball ships what its `files` allowlist permits; counting `.github/` as a
|
|
139
|
+
disagreement produced 22 findings where one file had moved.
|
|
140
|
+
- **Compare only when both sides claim the same version.** A branch already
|
|
141
|
+
bumped past its tag makes no common claim, and is `blind`, not `clean`.
|
|
142
|
+
|
|
143
|
+
## The two artefacts
|
|
144
|
+
|
|
145
|
+
`docs/audit/<date>-audit.html` and `docs/audit/<date>-audit.json`.
|
|
146
|
+
|
|
147
|
+
**The page carries aggregates and pointers, never raw bodies.** Counts, top
|
|
148
|
+
classes, trends, and a link to the issue in its own system — never a stack
|
|
149
|
+
trace, a log line or a row of data. The report is a file people forward, and a
|
|
150
|
+
report that cannot be shared is one nobody writes twice.
|
|
151
|
+
|
|
152
|
+
**A secret is reported by place and class, never by value.** `file:line`, which
|
|
153
|
+
credential it is, and the remedy. The value appears in neither artefact nor on
|
|
154
|
+
stdout: an audit must not become the second place a credential leaks. Redaction
|
|
155
|
+
is total rather than a prefix — half a credential plus its context is often
|
|
156
|
+
enough to finish.
|
|
157
|
+
|
|
158
|
+
**The sidecar is what makes this a ratchet rather than a snapshot.** Each
|
|
159
|
+
finding carries an id derived from its probe and its place, so it survives a
|
|
160
|
+
rewording; the next run prints what closed, what is new, and what has now
|
|
161
|
+
survived three audits. That last number is itself a finding: a defect nobody
|
|
162
|
+
picks up is a decision nobody wrote down.
|
|
163
|
+
|
|
164
|
+
## What the script does and what you do
|
|
165
|
+
|
|
166
|
+
| The script | You |
|
|
167
|
+
|---|---|
|
|
168
|
+
| discovery, the registry, mechanical probes, both artefacts, the diff | the seam walk, MCP evidence, judgement about what a finding means |
|
|
169
|
+
| refuses a fourth verdict, redacts secrets, excludes its own output | deciding severity and effort, writing the remedy that fits this project |
|
|
170
|
+
|
|
171
|
+
The split is not tidiness. **A judgement encoded in a script becomes a gate
|
|
172
|
+
nobody agreed to**; a mechanic left in prose becomes a step nobody runs. Both
|
|
173
|
+
have shipped in this family and both are on its boards.
|
|
174
|
+
|
|
175
|
+
Run the script first, read its blind list, then spend your reading where it
|
|
176
|
+
could not look.
|
|
177
|
+
|
|
178
|
+
## Exit criterion
|
|
179
|
+
|
|
180
|
+
An audit is finished when:
|
|
181
|
+
|
|
182
|
+
1. every probe has a verdict, and every `blind` one names why;
|
|
183
|
+
2. the page and the sidecar are written and the page has been opened;
|
|
184
|
+
3. **every number in the report was produced by a command this run executed** —
|
|
185
|
+
a restated count is an assertion (`evidence-docs`);
|
|
186
|
+
4. at least one figure was **re-derived by a differently-shaped command** and
|
|
187
|
+
both were printed. Re-running the same command is a spell-check of the first
|
|
188
|
+
run;
|
|
189
|
+
5. the proposed rows are printed for the operator, with nothing written.
|
|
190
|
+
|
|
191
|
+
## Rationalizations
|
|
192
|
+
|
|
193
|
+
| Excuse | Reality |
|
|
194
|
+
|---|---|
|
|
195
|
+
| "The tests pass, so the project is healthy" | Tests compare what somebody thought to write down. This phase 3 exists because the sharpest defect in this family was invisible to a green suite in every repository it touched. |
|
|
196
|
+
| "There's no Sentry, so there are no errors to report" | Those are the same empty section and opposite facts. That is what `blind` is for. |
|
|
197
|
+
| "The versions match, so the channels agree" | Measured: three channels, one version string, 231 lines of difference. Compare trees. |
|
|
198
|
+
| "I'll note the raw log lines so we have context" | Then the report cannot be shared, and it will not be written again. Aggregate and point. |
|
|
199
|
+
| "I found a credential — I'll paste it so we can check it's live" | The report becomes the second leak. Place and class; rotation is the remedy. |
|
|
200
|
+
| "Findings should just be fixed while I'm in there" | An audit that edits while it reads cannot re-run to check itself, and the next run cannot tell a fix from a rewording. |
|
|
201
|
+
| "It's the same nine checks every time, so I can skip discovery" | Then the instrument returns the same answer for every input, which is a fact about the instrument and not about the project. |
|
|
202
|
+
| "No findings — clean bill" | Only against the probes that ran. Read the blind list before saying that sentence out loud. |
|
|
@@ -0,0 +1,199 @@
|
|
|
1
|
+
# The probe catalogue
|
|
2
|
+
|
|
3
|
+
What to run, chosen by what the discovery phase found. Every entry is a probe in
|
|
4
|
+
the sense `SKILL.md` defines: it declares what it needs, and a need that is not
|
|
5
|
+
met makes it **blind with a reason**, never absent and never clean.
|
|
6
|
+
|
|
7
|
+
`scripts/audit.py` ships the mechanical ones. This file is the larger catalogue
|
|
8
|
+
— what to add for a project shape the script does not cover, and what a person
|
|
9
|
+
runs by hand once the script has printed where it could not look.
|
|
10
|
+
|
|
11
|
+
## Contents
|
|
12
|
+
|
|
13
|
+
- How to read this file
|
|
14
|
+
- Universal — every project, whatever it is
|
|
15
|
+
- By language and package manager
|
|
16
|
+
- By shape: a published package
|
|
17
|
+
- By shape: a deployed service
|
|
18
|
+
- By shape: a monorepo or a submodule tree
|
|
19
|
+
- By shape: a public web surface
|
|
20
|
+
- Production evidence through connected MCP servers
|
|
21
|
+
- Adding a probe
|
|
22
|
+
- What deliberately has no probe
|
|
23
|
+
|
|
24
|
+
## How to read this file
|
|
25
|
+
|
|
26
|
+
Each table row is `probe · needs · what it answers · what its absence means`.
|
|
27
|
+
The fourth column is the one that matters: **a probe worth adding is one whose
|
|
28
|
+
silence would otherwise be read as health.**
|
|
29
|
+
|
|
30
|
+
Two rules bind every row, and both come from defects this family has shipped:
|
|
31
|
+
|
|
32
|
+
- **Read committed state.** `git ls-files`, `git show <ref>:<path>`. A probe
|
|
33
|
+
that walks the working tree reports build residue and local scratch as project
|
|
34
|
+
state, and its finding cannot be reproduced from a clone.
|
|
35
|
+
- **A command that printed nothing has not answered.** Check the exit code *and*
|
|
36
|
+
that the output is non-empty and shaped as expected. An empty result and a
|
|
37
|
+
broken invocation are the same characters.
|
|
38
|
+
|
|
39
|
+
## Universal — every project, whatever it is
|
|
40
|
+
|
|
41
|
+
| Probe | Needs | Answers | Silence would mean |
|
|
42
|
+
|---|---|---|---|
|
|
43
|
+
| `secrets-tree` | git | is a credential committed? | "no secrets" when nobody looked |
|
|
44
|
+
| `secrets-history` | git | is one still reachable in history? | a rotated-looking repo that is not |
|
|
45
|
+
| `worktree` | git | does the tree disagree with HEAD? | a finding no clone can reproduce |
|
|
46
|
+
| `gitignore-secrets` | git | is a credential-shaped file tracked? | `.env` in the repo, unnoticed |
|
|
47
|
+
| `docs-present` | — | is there an entry point for a reader? | the entry point is a person |
|
|
48
|
+
| `ci-present` | — | does anything run the checks? | "the tests pass" — on whose machine? |
|
|
49
|
+
| `test-command` | — | does the project's own test command exist and run? | a suite nobody can invoke |
|
|
50
|
+
| `dependency-age` | manager | how far behind are the pinned deps? | quiet rot |
|
|
51
|
+
| `licence` | git | is there one, and do the deps agree with it? | a distribution problem found by a lawyer |
|
|
52
|
+
|
|
53
|
+
**`test-command` is worth writing by hand for any project the script cannot
|
|
54
|
+
guess.** Run the project's *own* documented command, read its output, and quote
|
|
55
|
+
it. `npm test` exiting 0 while printing `FAIL: 2 guards did not fire` has
|
|
56
|
+
happened in this family — a wrapper's exit status is not the suite's verdict.
|
|
57
|
+
|
|
58
|
+
## By language and package manager
|
|
59
|
+
|
|
60
|
+
| Stack | Probes worth running | Note |
|
|
61
|
+
|---|---|---|
|
|
62
|
+
| node / npm | `npm audit --json`, `npm outdated --json`, `npm ls --all` for phantom deps, `npm pack --dry-run` to see what actually ships | `files` in `package.json` decides the tarball; compare it against what the README claims ships |
|
|
63
|
+
| python | `pip-audit` or `uv pip list --outdated`, import-time side effects, `pyproject` vs `requirements` drift | two dependency files that disagree is the *two copies* problem: one of them is what installs |
|
|
64
|
+
| go | `go vet ./...`, `govulncheck`, `go mod tidy -diff` | a dirty `go.sum` after tidy is a real finding |
|
|
65
|
+
| rust | `cargo audit`, `cargo tree --duplicates` | duplicate transitive versions bloat and diverge |
|
|
66
|
+
| php / composer | `composer audit`, `composer outdated --direct` | |
|
|
67
|
+
| java | `mvn versions:display-dependency-updates`, `gradle dependencies` | |
|
|
68
|
+
|
|
69
|
+
**Whatever the stack: run the analyser the ecosystem already ships before
|
|
70
|
+
writing one.** A hand-rolled check competing with `cargo audit` will be wrong in
|
|
71
|
+
a way nobody notices.
|
|
72
|
+
|
|
73
|
+
## By shape: a published package
|
|
74
|
+
|
|
75
|
+
This is where the sharpest findings live, because the repository cannot see
|
|
76
|
+
them.
|
|
77
|
+
|
|
78
|
+
| Probe | Needs | Answers |
|
|
79
|
+
|---|---|---|
|
|
80
|
+
| `published-version` | registry, network | does the registry serve what the manifest claims? |
|
|
81
|
+
| `channel-divergence` | git (+ registry) | do the channels that share a version share a tree? |
|
|
82
|
+
| `tarball-contents` | registry, network | does the published artefact contain what it should — and nothing it should not? |
|
|
83
|
+
| `install-smoke` | network | does a clean install from the registry actually run? |
|
|
84
|
+
| `adoption` | network | downloads: is this in production at all? |
|
|
85
|
+
| `release-reliability` | gh | what share of *release* runs failed, and was a failure noticed? |
|
|
86
|
+
|
|
87
|
+
**`channel-divergence` is the reason this section exists**, and its trap is in
|
|
88
|
+
`SKILL.md`: compare the pair that can disagree — the tag against the branch tip
|
|
89
|
+
— not the registry against the tag, which agree by construction.
|
|
90
|
+
|
|
91
|
+
**`tarball-contents` catches the other direction.** A `files` allowlist that is
|
|
92
|
+
too wide ships tests, fixtures and sometimes `.env.example`; one too narrow ships
|
|
93
|
+
a package that cannot run. `npm pack --dry-run` prints the list.
|
|
94
|
+
|
|
95
|
+
**`release-reliability` is the closest thing a package has to a production
|
|
96
|
+
log.** A library never phones home, so the only observable failure signal is the
|
|
97
|
+
release pipeline. Measure the *release* workflow specifically — an overall
|
|
98
|
+
success rate is dominated by cheap validation runs and hides it.
|
|
99
|
+
|
|
100
|
+
## By shape: a deployed service
|
|
101
|
+
|
|
102
|
+
| Probe | Needs | Answers |
|
|
103
|
+
|---|---|---|
|
|
104
|
+
| `telemetry-present` | — | can anyone see a failure that happens to a user? |
|
|
105
|
+
| `health-endpoint` | network | does the deployed thing answer, and with what? |
|
|
106
|
+
| `deployed-version` | network | does what is running match what is tagged? |
|
|
107
|
+
| `migration-drift` | db access | are there migrations the deployed schema does not have? |
|
|
108
|
+
| `config-surface` | — | which env vars are required, and which are documented? |
|
|
109
|
+
|
|
110
|
+
**`deployed-version` is `channel-divergence` for services**, and it fails the
|
|
111
|
+
same way: a health endpoint reporting a version string proves a string.
|
|
112
|
+
|
|
113
|
+
**`config-surface` is almost always a finding.** Collect every `env` reference
|
|
114
|
+
in the code, compare against `.env.example` and the README. The difference is
|
|
115
|
+
what a new deployment will discover at runtime.
|
|
116
|
+
|
|
117
|
+
## By shape: a monorepo or a submodule tree
|
|
118
|
+
|
|
119
|
+
| Probe | Needs | Answers |
|
|
120
|
+
|---|---|---|
|
|
121
|
+
| `pointer-freshness` | git | does the parent point at a commit the child still has? |
|
|
122
|
+
| `pin-vs-release` | git, network | is the pinned commit the released one, by **tree** and not by version string? |
|
|
123
|
+
| `cross-package-drift` | — | do two packages state the same fact differently? |
|
|
124
|
+
| `workspace-orphans` | manager | is a package in the tree and in no workspace list? |
|
|
125
|
+
|
|
126
|
+
**A parent records a submodule as a pointer to one commit, and moving the
|
|
127
|
+
submodule does not move the pointer.** Work can be committed, pushed and green
|
|
128
|
+
while a clone of the parent gets the commit before it — and neither repository
|
|
129
|
+
looks wrong alone. Require `git submodule status` with no line starting `+`.
|
|
130
|
+
|
|
131
|
+
## By shape: a public web surface
|
|
132
|
+
|
|
133
|
+
| Probe | Needs | Answers |
|
|
134
|
+
|---|---|---|
|
|
135
|
+
| `robots-and-sitemap` | network | can a crawler reach what the product wants read? |
|
|
136
|
+
| `render-without-js` | network | is the answer extractable without running scripts? |
|
|
137
|
+
| `heading-and-schema` | network | one question per page, one answer, marked up |
|
|
138
|
+
| `link-rot` | network | do the addresses the site publishes still resolve? |
|
|
139
|
+
|
|
140
|
+
Depth beyond this is `seo-aeo-audit`'s ground, and an audit that reproduces it
|
|
141
|
+
badly is worse than one that names the boundary and points.
|
|
142
|
+
|
|
143
|
+
## Production evidence through connected MCP servers
|
|
144
|
+
|
|
145
|
+
Where the project uses a service **and** its MCP server is connected in this
|
|
146
|
+
session, read it. Where it is not, the probe is `blind` with that as the reason
|
|
147
|
+
— and that sentence belongs in the report.
|
|
148
|
+
|
|
149
|
+
| Source | Read | Never read |
|
|
150
|
+
|---|---|---|
|
|
151
|
+
| error tracker | issue counts by class, first/last seen, release correlation, regression flags | stack traces, request bodies, user identifiers |
|
|
152
|
+
| analytics | funnel steps, retention curves, conversion rates as numbers | individual sessions or user paths |
|
|
153
|
+
| hosting / edge | error rates, deploy history, rollback events | request logs |
|
|
154
|
+
| database | row counts, table sizes, migration state, index health | rows |
|
|
155
|
+
| CI / VCS | run history, failure rates, open issues and their age | — |
|
|
156
|
+
|
|
157
|
+
**The right-hand column is the contract, not a suggestion.** The report carries
|
|
158
|
+
aggregates and pointers; a reader who needs the trace opens the tracker. A report
|
|
159
|
+
holding raw bodies is one nobody can forward, and a report nobody forwards is
|
|
160
|
+
written once.
|
|
161
|
+
|
|
162
|
+
**Ask before assuming a service is unused.** A project with no Sentry dependency
|
|
163
|
+
in its manifest may still run one at the platform layer. Absence of a dependency
|
|
164
|
+
is evidence about the manifest, not about production.
|
|
165
|
+
|
|
166
|
+
## Adding a probe
|
|
167
|
+
|
|
168
|
+
```python
|
|
169
|
+
@probe("my-probe", "prod", needs=("gh", "network"))
|
|
170
|
+
def _p_mine(ctx):
|
|
171
|
+
rc, out, err = ctx.sh("gh", "api", "repos/:owner/:repo/releases")
|
|
172
|
+
if classify_output(rc, out, err) == "blind":
|
|
173
|
+
return Result("blind", "gh returned nothing: %s" % (err or rc))
|
|
174
|
+
...
|
|
175
|
+
return Result("clean", "12 releases, newest 3 days old")
|
|
176
|
+
```
|
|
177
|
+
|
|
178
|
+
Four rules, each of which has a defect behind it:
|
|
179
|
+
|
|
180
|
+
1. **Declare every need.** A probe that shells out to find out cannot explain
|
|
181
|
+
why it was skipped, and the report loses the reason.
|
|
182
|
+
2. **Assert the input arrived.** A component that never received its input fails
|
|
183
|
+
*open*, and from outside it is indistinguishable from one that approved.
|
|
184
|
+
3. **Never let a probe take the run down.** It may raise; the registry converts
|
|
185
|
+
that to `blind`. It must not `sys.exit`.
|
|
186
|
+
4. **Exclude the audit's own output.** A probe reading `docs/audit/` measures
|
|
187
|
+
the instrument. Caught by the three-run fixture, not by reading.
|
|
188
|
+
|
|
189
|
+
## What deliberately has no probe
|
|
190
|
+
|
|
191
|
+
- **Whether the code is good.** Style, architecture and taste are readings, and
|
|
192
|
+
a script that scores them produces a number nobody can argue with.
|
|
193
|
+
- **Whether a finding matters.** Severity and effort are the operator's, priced
|
|
194
|
+
with the project's own formula.
|
|
195
|
+
- **Anything that writes.** Not a fix, not an issue, not a board row. The audit
|
|
196
|
+
proposes; a separate act accepts.
|
|
197
|
+
- **Anything requiring a credential the operator has not already connected.**
|
|
198
|
+
An audit that asks for a new secret to run is a supply-chain risk wearing a
|
|
199
|
+
clipboard.
|