@kirill.konshin/lint 0.6.4 → 0.6.6
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +6 -6
- package/dist/lintPrepare.d.ts +18 -23
- package/dist/lintPrepare.js +68 -82
- package/package.json +1 -3
- package/rules/agent.md +2 -7
- package/skills/lint-repo/SKILL.md +2 -2
- package/{commands/spec-author.md → skills/spec-author/SKILL.md} +4 -4
- package/skills/spec-steward/SKILL.md +14 -9
- package/skills/spec-steward/references/apply.md +14 -14
- package/skills/spec-steward/references/audit.md +15 -15
- package/skills/spec-steward/references/contract.md +6 -6
- package/skills/spec-steward/references/model.md +13 -1
- package/skills/spec-steward/references/review-format.md +70 -44
- package/skills/spec-steward/references/worker.md +51 -53
- package/skills/spec-tools/SKILL.md +14 -8
- package/skills/spec-tools/references/contract.md +19 -12
- package/skills/spec-tools/references/gitlab.md +22 -24
- package/skills/spec-tools/references/rules/common.md +46 -0
- package/skills/spec-tools/references/rules/spec-steward.md +71 -0
- package/skills/spec-tools/references/rules/spec-verify-judge.md +17 -0
- package/skills/spec-tools/references/rules/spec-verify.md +50 -0
- package/skills/spec-tools/scripts/cli.js +8255 -1811
- package/skills/spec-verify/SKILL.md +78 -128
- package/skills/spec-steward/references/criteria.md +0 -59
|
@@ -1,32 +1,32 @@
|
|
|
1
1
|
# Audit procedure
|
|
2
2
|
|
|
3
|
-
An audit asks
|
|
3
|
+
An audit asks if the corpus holds the correct rules with the correct evidence. Its judging rules are [`common.md`](../../spec-tools/references/rules/common.md) and [`spec-steward.md`](../../spec-tools/references/rules/spec-steward.md). It writes a review file ([review-format.md](review-format.md)). Nothing in the specifications or the code changes during an audit. If the code agrees with the specifications is the `spec-verify` audit's question.
|
|
4
4
|
|
|
5
5
|
## 1. Run the engine
|
|
6
6
|
|
|
7
|
-
Both audits run on one engine, `spec-tools` (the `spec-tools` skill), one repository per run
|
|
7
|
+
Both audits run on one engine, `spec-tools` (the `spec-tools` skill), one repository per run. Wording drift between repositories is [align](../SKILL.md#align). The full sequence, from a detached tree of `HEAD` (`--here` for uncommitted edits):
|
|
8
8
|
|
|
9
9
|
```bash
|
|
10
10
|
spec-tools audit --audit spec-steward --context <decisions file>
|
|
11
11
|
```
|
|
12
12
|
|
|
13
|
-
It runs locally, on the owner's request
|
|
13
|
+
It runs locally, on the owner's request. No pipeline runs it. `--context` passes a file of established facts and earlier owner decisions (earlier review answers, the decision log) into each worker's and judge's brief, so that no finding reverses a decision without saying so.
|
|
14
14
|
|
|
15
|
-
It runs these passes
|
|
15
|
+
It runs these passes. Each is also a command of its own:
|
|
16
16
|
|
|
17
|
-
1. `spec-tools scope --audit spec-steward
|
|
18
|
-
2. `spec-tools workers --audit spec-steward
|
|
19
|
-
3. `spec-tools report --audit spec-steward
|
|
20
|
-
4. `spec-tools workers --complete --audit spec-steward
|
|
21
|
-
5. `spec-tools workers --verify --audit spec-steward
|
|
17
|
+
1. `spec-tools scope --audit spec-steward`: the evidence, cut into parts that one reader holds, plus one unit per [sweep](../sweeps.json) (tooling and CI, tests citing the corpus, comments and citations, missing invariants, placement). It writes `.spec-audit/steward/scope.yaml`.
|
|
18
|
+
2. `spec-tools workers --audit spec-steward`: one headless worker per unit, under [worker.md](worker.md). Each writes `.spec-audit/steward/findings/part-<n>-<reader>.yaml`. The tool checks each file and sends its errors back to the worker.
|
|
19
|
+
3. `spec-tools report --audit spec-steward`: the merge. It looks up each quote at its line and drops a finding whose quote is not there. It lists the requirements that no worker judged as short.
|
|
20
|
+
4. `spec-tools workers --complete --audit spec-steward`: the short requirements, judged again. Then the merge again.
|
|
21
|
+
5. `spec-tools workers --verify --audit spec-steward`: one judge per finding, under the steward judge rules. It keeps, revises or drops the finding and gives `judgeConfidence`. Then the last merge joins duplicates (one rule, one problem), offers the findings that share a `themeKey` as a theme, and renders `.spec-audit/spec-review.yaml`.
|
|
22
22
|
|
|
23
|
-
The run's data
|
|
23
|
+
The run's data is `.spec-audit/steward/review-data.yaml`: the themes and findings, the dropped findings with the reason, the requirements judged sound, the short list and the workers' notes.
|
|
24
24
|
|
|
25
25
|
## 2. Consolidate and hand back
|
|
26
26
|
|
|
27
|
-
- **Read
|
|
28
|
-
- **Order the
|
|
29
|
-
- **
|
|
30
|
-
- **Check** with `steward review lint` and `steward review verify --root NAME=path …`.
|
|
27
|
+
- **Read each item yourself.** Make the wording clear and in ASD-STE100, correct what the readers got wrong, and decide each theme: keep it as one item when its members share one solution, split it when they do not.
|
|
28
|
+
- **Order the items** so that a decision that others depend on comes first.
|
|
29
|
+
- **List the requirements judged sound** at the end, in a YAML comment, so that the coverage is visible.
|
|
30
|
+
- **Check** with `steward review lint .spec-audit/spec-review.yaml` and `steward review verify .spec-audit/spec-review.yaml --root NAME=path …`.
|
|
31
31
|
|
|
32
|
-
|
|
32
|
+
Give the file to the owner and stop.
|
|
@@ -55,7 +55,7 @@ There are exactly two markers. Each stands on a line of its own: the Advisory li
|
|
|
55
55
|
|
|
56
56
|
| Marker | Meaning |
|
|
57
57
|
| --- | --- |
|
|
58
|
-
| `**⚠️ Advisory:** <why review is its evidence>` | The requirement is ADVISORY. It keeps its RFC 2119 keywords and is exempt from the scenario ratchet and from `absolute-unproven`. Coverage reports it as `advisory`, and the evidence
|
|
58
|
+
| `**⚠️ Advisory:** <why review is its evidence>` | The requirement is ADVISORY. It keeps its RFC 2119 keywords and is exempt from the scenario ratchet and from `absolute-unproven`. Coverage reports it as `advisory`, and the evidence model classes it `advisory`. Marking an existing requirement Advisory is reported as `weakened` ("newly marked advisory"). It qualifies the whole requirement, so a requirement that mixes a contract with guidance is split. |
|
|
59
59
|
| `**⚠️ Known gap (<tracker>):** <what is missing>` | Behaviour the requirement describes is knowingly absent or incomplete, or a REQUIRED rule lacks the evidence that would catch its failure. `<tracker>` is required text: a ticket key (`PROJ-123`), a tickets-page id, or an open `openspec/changes/<name>` folder. Steward checks only that it is present: any non-empty text inside the parentheses counts and is not validated, so a placeholder counts until it is replaced. In the statement, the line records the gap and exempts nothing, unless it names scenarios of its requirement by title after `exempts`, each in quotes (`exempts scenario 'A'`, `exempts scenarios 'A' and 'B'`; double quotes, curly quotes, `_…_` and `*…*` are read too). Exactly the named scenarios are then exempt from the scenario ratchet. Coverage lists every gap with its tracker. |
|
|
60
60
|
|
|
61
61
|
Retired spellings are `**⚠️ Unenforced:**` and `**⚠️ Known gap:**` without a tracker. Each is reported as `marker-hygiene` and grants no exemption. So is a marker inside a line rather than on a line of its own, and an Advisory or Known gap line inside a scenario. A misspelt marker (wrong case, missing emoji or bold) is reported as `marker-hygiene`, and `--fix` normalises it.
|
|
@@ -132,8 +132,8 @@ An explicit `--base <ref>` compares with the merge base of `<ref>` and `HEAD`. W
|
|
|
132
132
|
| Command | Purpose | Output |
|
|
133
133
|
| --- | --- | --- |
|
|
134
134
|
| `spec-tools steward check [--base auto\|<ref>] [--binds <glob>]… [--file <f>] [--fix] [--strict] [--json]` | The corpus gate, run locally, in CI and by the hook | Findings ([below](#findings)) |
|
|
135
|
-
| `spec-tools steward coverage [--out <file>] [--binds <glob>]…` | The [coverage report](#coverage-report) |
|
|
136
|
-
| `spec-tools steward evidence --json [--binds <glob>]…` | The audits' [evidence model](#evidence-
|
|
135
|
+
| `spec-tools steward coverage [--out <file>] [--binds <glob>]…` | The [coverage report](#coverage-report) | `.spec-audit/coverage.yaml`, and the Markdown at `--out` (default `.spec-audit/coverage.md`) |
|
|
136
|
+
| `spec-tools steward evidence --json [--binds <glob>]…` | The audits' [evidence model](#evidence-model), which `spec-tools scope` writes to `.spec-audit/evidence.yaml` and cuts into each audit's evidence files | JSON on stdout |
|
|
137
137
|
| `spec-tools steward review`, `align`, `wire`, `hook` | Review files, cross-repository drift, wiring, the edit hook | As each command documents |
|
|
138
138
|
|
|
139
139
|
`spec-tools steward` runs through the package's one `bin`, `spec-tools`. A repository calls it from its root scripts and never imports steward's modules: the package's `exports` map does not expose them, so the CLI and its JSON are the interface.
|
|
@@ -155,7 +155,7 @@ Exit 2 on an empty listing keeps a broken checkout from passing a scan that read
|
|
|
155
155
|
|
|
156
156
|
## Coverage report
|
|
157
157
|
|
|
158
|
-
`spec-tools steward coverage` writes a Markdown report that is MDX-safe: `<`, `>`, `{`, `}` and `|` are escaped. It gates nothing: it exits 0, or 2 on an environment error. Statuses come from [bindings](#binding) and [markers](#markers) only. A pointer never changes a status.
|
|
158
|
+
`spec-tools steward coverage` writes the coverage as data, `.spec-audit/coverage.yaml` (totals, each requirement and scenario with its status, bindings and pointers, the known gaps, the retired markers), and as a Markdown report for people that is MDX-safe: `<`, `>`, `{`, `}` and `|` are escaped. It gates nothing: it exits 0, or 2 on an environment error. Statuses come from [bindings](#binding) and [markers](#markers) only. A pointer never changes a status.
|
|
159
159
|
|
|
160
160
|
- **Headline.** Requirements and capabilities; Advisory requirements; Known gaps; REQUIRED requirements no test binds; REQUIRED scenarios no test binds; retired markers.
|
|
161
161
|
- **Per capability.** One table: `| Requirement / Scenario | Spec | Bound by | Pointers | Status |`.
|
|
@@ -164,9 +164,9 @@ Exit 2 on an empty listing keeps a broken checkout from passing a scan that read
|
|
|
164
164
|
- **Pointers.** The code and document citations of the requirement and its scenarios.
|
|
165
165
|
- **Closing sections.** "Known gaps" (tracker, requirement, text) and "Retired markers".
|
|
166
166
|
|
|
167
|
-
## Evidence
|
|
167
|
+
## Evidence model
|
|
168
168
|
|
|
169
|
-
`spec-tools steward evidence --json`
|
|
169
|
+
`spec-tools steward evidence --json` prints the evidence an audit reads, one entry per requirement. `spec-tools scope` writes the same model as `.spec-audit/evidence.yaml`:
|
|
170
170
|
|
|
171
171
|
```
|
|
172
172
|
{ version: 1, root, requirements: [{ id, capability, file, line, end, block, class: 'required'|'advisory',
|
|
@@ -21,7 +21,9 @@ If you cannot name a failure, the rule is not a REQUIRED contract.
|
|
|
21
21
|
| Kind | Meaning | Evidence |
|
|
22
22
|
| --- | --- | --- |
|
|
23
23
|
| **REQUIRED** | An invariant whose violation is a real defect — behavioural, API, compatibility, security, data. | The cheapest rung that catches the failure (ladder below). The requirement names that failure and that rung; a test is one rung, not the only one. |
|
|
24
|
-
| **⚠️ ADVISORY** |
|
|
24
|
+
| **⚠️ ADVISORY** | Guidance whose violation has no production effect, or that cannot reasonably be proven mechanically: design direction, implementation detail kept in the specification, development-only and local-only behaviour, demo agents, internal CI and developer tools, prompt wording. | Review judgement. Never a gate, never a reason to write a test. |
|
|
25
|
+
|
|
26
|
+
A REQUIRED requirement is one whose violation has a production effect, as the [common judging rules](../../spec-tools/references/rules/common.md#production-effect) define it: a broken flow, a contract or invariant violation, a sensitive-data leak, or a performance problem.
|
|
25
27
|
|
|
26
28
|
A REQUIRED requirement stating an absolute names its exceptions, or has none; one with neither is listed for the owner's review.
|
|
27
29
|
|
|
@@ -64,6 +66,16 @@ A scenario adds precision — one concrete case, preferably Given/When/Then —
|
|
|
64
66
|
|
|
65
67
|
New and edited text uses EARS for requirement statements — When <trigger>, the <system> SHALL <response>; While <state>, the <system> SHALL …; If <condition>, then the <system> SHALL …; or The <system> SHALL … — and Given/When/Then for scenarios, with Given only where a precondition matters. The RFC 2119 keyword stays in capitals: OpenSpec's strict validation fails a requirement whose body has no SHALL or MUST. Untouched text is not rewritten for syntax alone.
|
|
66
68
|
|
|
69
|
+
New and edited text is also written in a light subset of ASD-STE100:
|
|
70
|
+
|
|
71
|
+
- One obligation per sentence.
|
|
72
|
+
- At most 25 words per sentence.
|
|
73
|
+
- One term for one concept, and the same term each time.
|
|
74
|
+
- Active voice: the system or the component is the subject.
|
|
75
|
+
- No vague words in a REQUIRED statement (thin, safe, lean, at parity, needed): name the measurable condition.
|
|
76
|
+
|
|
77
|
+
Untouched text is not rewritten for this subset alone.
|
|
78
|
+
|
|
67
79
|
## Changing a rule
|
|
68
80
|
|
|
69
81
|
- Never weaken, narrow, delete or reclassify a rule to make code pass. A demonstrated divergence is decided by the owner: the code is wrong, or the rule is — and a wrong rule is changed openly, through the repository's change process.
|
|
@@ -1,61 +1,87 @@
|
|
|
1
1
|
# Review file format
|
|
2
2
|
|
|
3
|
-
An audit
|
|
3
|
+
An audit gives its findings to the owner as one YAML review file, `.spec-audit/spec-review.yaml`. The owner answers each item in place, and the file stays as the record across rounds. `steward review render` writes the items. `steward review lint` checks the shape. `steward review verify --root NAME=path` checks that each quote is at its location. `steward review status` counts the answers.
|
|
4
4
|
|
|
5
5
|
## Items
|
|
6
6
|
|
|
7
|
-
Each item
|
|
8
|
-
|
|
9
|
-
```
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
7
|
+
Each item is one top-level block, keyed by its id, `R-NNN`. An item about one rule has `rule`:
|
|
8
|
+
|
|
9
|
+
```yaml
|
|
10
|
+
R-001:
|
|
11
|
+
title: 'Implementation detail in a REQUIRED rule'
|
|
12
|
+
area: 'system-overview/logging'
|
|
13
|
+
layer: 'SPEC-ADVISORY'
|
|
14
|
+
criteria:
|
|
15
|
+
- 8
|
|
16
|
+
confidence: 85
|
|
17
|
+
rule:
|
|
18
|
+
text: |-
|
|
19
|
+
The server SHALL promote accountId, rcAccountId, rcRequestId and rcUserId…
|
|
20
|
+
location: 'SHOP openspec/specs/system-overview/logging/spec.md:47'
|
|
21
|
+
wrong: |
|
|
22
|
+
The rule names the exact log fields. A field change breaks the rule with no effect in production.
|
|
23
|
+
proposal: |
|
|
24
|
+
Reclassify → OpenSpec ⚠️ ADVISORY — mark the rule Advisory, or move the field list to the logger's scoped rule.
|
|
25
|
+
evidence:
|
|
26
|
+
- 'src/logger.ts:12'
|
|
27
|
+
decision: ''
|
|
17
28
|
```
|
|
18
29
|
|
|
19
|
-
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
-
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
|
|
30
|
+
An item about several rules with one problem and one solution is a theme. It has `theme` and a `rules` list:
|
|
31
|
+
|
|
32
|
+
```yaml
|
|
33
|
+
R-002:
|
|
34
|
+
theme: 'Development-only behaviour held as REQUIRED'
|
|
35
|
+
layer: 'SPEC-ADVISORY'
|
|
36
|
+
confidence: 75
|
|
37
|
+
rules:
|
|
38
|
+
- text: |-
|
|
39
|
+
WHEN the dev server starts THEN it seeds the demo account
|
|
40
|
+
location: 'SHOP openspec/specs/apps/runtime/spec.md:88'
|
|
41
|
+
- text: |-
|
|
42
|
+
The local database SHALL reset on each start
|
|
43
|
+
location: 'SHOP openspec/specs/apps/db/spec.md:12'
|
|
44
|
+
note: 'the reset is also in the README'
|
|
45
|
+
wrong: |
|
|
46
|
+
Both rules apply only on a developer machine. A violation has no effect in production.
|
|
47
|
+
proposal: |
|
|
48
|
+
Reclassify → OpenSpec ⚠️ ADVISORY — mark both rules Advisory.
|
|
49
|
+
decision: ''
|
|
50
|
+
```
|
|
27
51
|
|
|
28
|
-
|
|
29
|
-
### R-012 · DELETE · Theme: Scenarios that restate their requirement · 19, 20
|
|
52
|
+
The keys:
|
|
30
53
|
|
|
31
|
-
-
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
54
|
+
- `title` or `theme`: one line.
|
|
55
|
+
- `area`: the capability or the sweep that found the item.
|
|
56
|
+
- `layer`: the owning layer that the item proposes ([worker.md](worker.md)).
|
|
57
|
+
- `criteria`: the item numbers of the [steward rules](../../spec-tools/references/rules/spec-steward.md#items-to-report).
|
|
58
|
+
- `confidence`: an integer from 0 to 100, the judge's confidence, else the reader's. A theme takes the lowest confidence of its rules. Items at 70 or lower come after the comment `# Low confidence — 70% or lower`.
|
|
59
|
+
- `needsHumanIntent: true`: the intent is not clear, and the proposal gives options.
|
|
60
|
+
- `rule` / `rules[]`: `text` is the original text, so that the owner knows the rule without opening the file; `…` leaves text out. `location` is `NAME path:line`, where NAME is the repository and path is relative to it. `note` is the audit's remark about one member of a theme. `decision` on a member is the owner's answer for that member only.
|
|
61
|
+
- `wrong`: the problem, in ASD-STE100.
|
|
62
|
+
- `proposal`: starts with the target notation of the [steward rules](../../spec-tools/references/rules/spec-steward.md) (`Keep →`, `Reclassify →`, `Move →`, `Replace →`, `Merge →`, `Rewrite →`, `Delete →`).
|
|
63
|
+
- `evidence`, `crossRefs`, `reversesPastDecision`: present only when they have content.
|
|
64
|
+
- `decision`: the owner's answer. It starts empty.
|
|
39
65
|
|
|
40
|
-
|
|
66
|
+
A rule is in one item only. A second problem with the same rule is its own item, with a `crossRefs` entry to the first.
|
|
41
67
|
|
|
42
|
-
##
|
|
68
|
+
## Decisions
|
|
43
69
|
|
|
44
|
-
| The owner writes | Meaning
|
|
45
|
-
|
|
|
46
|
-
|
|
|
47
|
-
|
|
|
48
|
-
|
|
|
49
|
-
|
|
|
50
|
-
|
|
|
51
|
-
|
|
|
70
|
+
| The owner writes in `decision` | Meaning |
|
|
71
|
+
| ------------------------------ | -------------------------------------------------------------------------- |
|
|
72
|
+
| `""` or `"⬜"` | open: not answered |
|
|
73
|
+
| `"✅"` | accept the proposal |
|
|
74
|
+
| `"✅ <comment>"` | accept, with the instruction in the comment |
|
|
75
|
+
| `"<comment>"` | not accepted and not rejected: read the comment and examine the item again |
|
|
76
|
+
| `"❌"` | reject: keep the rule as it is |
|
|
77
|
+
| `"❌ <comment>"` | reject, and do what the comment tells |
|
|
52
78
|
|
|
53
|
-
|
|
79
|
+
The owner can answer one member of a theme with a `decision` on that member, in the same form. No answer is never approval. An empty `decision` is open.
|
|
54
80
|
|
|
55
81
|
## After a round
|
|
56
82
|
|
|
57
|
-
Processing a round ([apply.md](apply.md)) adds to the file and never
|
|
83
|
+
Processing a round ([apply.md](apply.md)) adds to the file and never removes from it:
|
|
58
84
|
|
|
59
|
-
- **An applied item** gets `
|
|
60
|
-
- **A comment-only item** is
|
|
61
|
-
- **New findings**
|
|
85
|
+
- **An applied item** gets `applied: "R<round> — <what changed, with paths>"`.
|
|
86
|
+
- **A comment-only item** is examined again. The comment moves into `previousDecision`, the proposal changes in place, and `decision` becomes `""` again.
|
|
87
|
+
- **New findings** come after the comment `# Round <n> — new`, numbered after the last item, so that the resolved items stay as the record.
|
|
@@ -1,67 +1,65 @@
|
|
|
1
1
|
# Audit worker
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
This file tells how one worker of the corpus-quality audit runs. `spec-tools workers --audit spec-steward` starts one worker per unit of the scope. Its brief names the unit, the files to read and the findings file to write. The procedure around the workers is [audit.md](audit.md).
|
|
4
4
|
|
|
5
|
-
|
|
5
|
+
What to judge is in the judging rules: [`common.md`](../../spec-tools/references/rules/common.md), then [`spec-steward.md`](../../spec-tools/references/rules/spec-steward.md). How a specification is written is [model.md](model.md). Where guidance belongs is the placement guide, `rules/agent.md` of this package. Read them before you judge.
|
|
6
6
|
|
|
7
|
-
|
|
8
|
-
|
|
9
|
-
- Specification quality first. An ordinary implementation bug is out of scope unless it reveals the rule is unrealistic, unclear, outdated, ambiguous, or contradicted by what the tests actually assert. Whether code conforms is the `spec-verify` audit's.
|
|
10
|
-
- Comments, descriptions, prompt text and citations are never evidence that code complies, and never a violation by themselves. Judge executable behaviour and structure.
|
|
11
|
-
- For every rule you would keep REQUIRED, ask what meaningful failure its verification catches. Never propose a test merely because a rule lacks one, and never a test of incidental text or structure.
|
|
12
|
-
- Internal component and API contracts are legitimate specification. The line is contract versus implementation or process, not product versus engineering.
|
|
13
|
-
- Do not decide intent silently: when the meaning is ambiguous, describe the readings, propose options and set `needsHumanIntent`.
|
|
14
|
-
- No style preferences: every finding states a concrete problem.
|
|
15
|
-
- A proposal that reverses an earlier owner decision says so, in `reversesPastDecision`.
|
|
16
|
-
- Quote the original text verbatim, from the line you name; open the file to confirm before writing it.
|
|
17
|
-
|
|
18
|
-
The audit is read-only: you write your findings file and nothing else. Your tools are Read, Glob, Grep, Write and the read-only git commands; anything else is denied and only costs a turn.
|
|
7
|
+
The audit is read-only. You write your findings file and nothing else. Your tools are Read, Glob, Grep, Write and the read-only git commands. Other commands are denied and only cost a turn.
|
|
19
8
|
|
|
20
9
|
## Your unit
|
|
21
10
|
|
|
22
|
-
- **A part** is a set of capabilities
|
|
23
|
-
- **A sweep** looks across the corpus from one angle
|
|
11
|
+
- **A part** is a set of capabilities. Each capability has its evidence file under `.spec-audit/parts/`: each requirement with its line, the tests bound to it, where its terms occur, and its related requirements. Read the evidence files and the specifications they name. Read code and tests where a verdict depends on them. Each requirement of the part is in a finding (on it or on one of its scenarios) or in `sound`. Each requirement you judged is in `judged`.
|
|
12
|
+
- **A sweep** looks across the corpus from one angle. Its focus names the items to report. Read the files the brief lists. `judged` and `sound` can stay empty.
|
|
24
13
|
|
|
25
14
|
## The findings file
|
|
26
15
|
|
|
27
|
-
|
|
16
|
+
Write one finding for each rule and problem. A rule can be a requirement, a scenario, a sentence, or, outside the specifications, a section or line of `AGENTS.md`, a skill, a configuration, a script or a CI job. Write the file with the Write tool, at the path the brief names:
|
|
28
17
|
|
|
29
|
-
```
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
"billing/refunds#requirement-a-refund-names-its-order"
|
|
56
|
-
],
|
|
57
|
-
"notes": []
|
|
58
|
-
}
|
|
18
|
+
```yaml
|
|
19
|
+
findings:
|
|
20
|
+
- file: 'openspec/specs/billing/refunds/spec.md'
|
|
21
|
+
line: 12
|
|
22
|
+
quote: |-
|
|
23
|
+
The service SHALL apply a refund at most once per request id
|
|
24
|
+
ruleName: 'Refunds are idempotent'
|
|
25
|
+
criteria:
|
|
26
|
+
- 6
|
|
27
|
+
- 7
|
|
28
|
+
layer: 'SPEC-ADVISORY'
|
|
29
|
+
whatsWrong: |
|
|
30
|
+
The rule names no failure that a test can catch. Its tests assert only the text of the log line.
|
|
31
|
+
proposed: |
|
|
32
|
+
Reclassify → OpenSpec ⚠️ ADVISORY — mark the requirement Advisory and remove the log-text test.
|
|
33
|
+
evidence:
|
|
34
|
+
- 'src/refunds.ts:40'
|
|
35
|
+
themeKey: 'unverifiable-absolute'
|
|
36
|
+
needsHumanIntent: false
|
|
37
|
+
readerConfidence: 75
|
|
38
|
+
sound:
|
|
39
|
+
- 'billing/refunds#requirement-a-refund-names-its-order'
|
|
40
|
+
judged:
|
|
41
|
+
- 'billing/refunds#requirement-refunds-are-idempotent'
|
|
42
|
+
- 'billing/refunds#requirement-a-refund-names-its-order'
|
|
43
|
+
notes: []
|
|
59
44
|
```
|
|
60
45
|
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
- `
|
|
64
|
-
- `
|
|
65
|
-
- `
|
|
46
|
+
How to write each key:
|
|
47
|
+
|
|
48
|
+
- `file`, `ruleName`, `layer`, `themeKey`, `reversesPastDecision`: double-quoted strings. `file` is relative to the repository.
|
|
49
|
+
- `line`, `criteria[]`, `readerConfidence`: plain integers. `line` is where the quoted text starts. `readerConfidence` is from 0 to 100 ([common rules](../../spec-tools/references/rules/common.md#confidence)).
|
|
50
|
+
- `quote`: a stripped literal block (`|-`), copied from that line, at most 300 characters. Use `…` where you leave text out. The merge looks for the quote at its line and drops a finding whose quote is not there.
|
|
51
|
+
- `whatsWrong`, `proposed`: literal blocks (`|`), in ASD-STE100.
|
|
52
|
+
- `evidence`, `crossRefs`, `sound`, `judged`, `notes`: block lists of double-quoted strings. Address a requirement as `capability#requirement-slug`.
|
|
53
|
+
- `needsHumanIntent`: `true` or `false`.
|
|
54
|
+
|
|
55
|
+
What each key holds:
|
|
56
|
+
|
|
57
|
+
- `criteria`: the item numbers of the [steward rules](../../spec-tools/references/rules/spec-steward.md#items-to-report).
|
|
58
|
+
- `proposed`: starts with the target notation (`Keep →`, `Reclassify →`, `Move →`, `Replace →`, `Merge →`, `Rewrite →`, `Delete →`) and tells exactly what to do. A rewrite gives the new text. A move names the target file.
|
|
59
|
+
- `layer`: the layer you propose as the owner: `SPEC-REQUIRED`, `SPEC-ADVISORY`, `DESIGN`, `AGENTS`, `RULE`, `SKILL`, `COMMAND`, `OPSX-CONFIG`, `README`, `TOOLING`, `TEST`, `CI`, `MERGE`, `REWRITE` or `DELETE`.
|
|
60
|
+
- `themeKey`: a kebab-case name for the problem under the finding. The merge groups the findings that share a key into a theme. Use a key that another finding uses for the same problem, for example `impl-detail-in-spec`, `advisory-not-required`, `restating-scenario`, `tooling-should-enforce`, `duplicate-rule` or `stale-rule`. When no key fits, make a new one.
|
|
61
|
+
- `reversesPastDecision`: the owner decision that the proposal reverses. Leave it out when there is none.
|
|
62
|
+
- `sound`: the requirements that you judged correct, in the correct place, and with sufficient evidence.
|
|
63
|
+
- `judged`: each requirement that you judged, with or without a finding. A requirement of your part that is not in `judged` goes to a completion worker.
|
|
66
64
|
|
|
67
|
-
|
|
65
|
+
When you stop, the tool checks the file. If the file has errors, the tool sends them back to you, and you correct the file. Then return one line: the path and your count of findings.
|
|
@@ -5,10 +5,11 @@ description: The machinery around both audits — set up and run the `spec-tools
|
|
|
5
5
|
|
|
6
6
|
# spec-tools
|
|
7
7
|
|
|
8
|
-
One CLI, `spec-tools`, installed with `@kirill.konshin/lint` beside `spec-steward`. What each command does, reads and writes, and its exit codes are [references/contract.md](references/contract.md); a GitLab pipeline is [references/gitlab.md](references/gitlab.md).
|
|
8
|
+
One CLI, `spec-tools`, installed with `@kirill.konshin/lint` beside `spec-steward`. What each command does, reads and writes, and its exit codes are [references/contract.md](references/contract.md); a GitLab pipeline is [references/gitlab.md](references/gitlab.md). Four things live elsewhere:
|
|
9
9
|
|
|
10
10
|
- the corpus gate — citations, binding, markers, the scenario ratchet — is the `spec-steward` skill;
|
|
11
|
-
-
|
|
11
|
+
- what an audit judges — the production effect, the tiers, the confidence, the checks — is the judging rules in [references/rules/](references/rules): `common.md`, then one file per audit role (`spec-verify.md`, `spec-verify-judge.md`, `spec-steward.md`);
|
|
12
|
+
- how an audit worker runs and writes its findings file is the `spec-verify` skill, which `spec-tools workers` hands every worker;
|
|
12
13
|
- how the repository makes a change is its own `AGENTS.md`.
|
|
13
14
|
|
|
14
15
|
## Commands
|
|
@@ -20,10 +21,10 @@ One CLI, `spec-tools`, installed with `@kirill.konshin/lint` beside `spec-stewar
|
|
|
20
21
|
| What changed in the specifications, by requirement | `spec-tools diff [<base>]` |
|
|
21
22
|
| Check a spec edit before handing it back | `spec-tools changed`, then `/spec-verify changed` |
|
|
22
23
|
| The full code-conformance audit, locally | `spec-tools audit` (`--here` for uncommitted edits, `--dry` to stop after the scope) |
|
|
23
|
-
| spec-steward's corpus-quality audit, on the same engine | `spec-tools audit --audit spec-steward`, then consolidate
|
|
24
|
+
| spec-steward's corpus-quality audit, on the same engine | `spec-tools audit --audit spec-steward`, then consolidate `.spec-audit/spec-review.yaml` as the `spec-steward` skill says |
|
|
24
25
|
| One pass of a CI audit | `spec-tools tier`, `scope`, `workers [--complete\|--verify]`, `report [--gate]` |
|
|
25
|
-
| A headless skill review in CI | `spec-tools run <skill> --verdict
|
|
26
|
-
| Gate a review's report | `spec-tools verdict
|
|
26
|
+
| A headless skill review in CI | `spec-tools run <skill> --verdict .spec-audit/<report>.yaml` |
|
|
27
|
+
| Gate a review's report | `spec-tools verdict .spec-audit/spec-verify.yaml` or `.spec-audit/<report>.yaml` |
|
|
27
28
|
| The merge-request comment and HTML reports | `spec-tools comment <name>=<report.md>...`, `spec-tools html <file.md>...` |
|
|
28
29
|
|
|
29
30
|
In CI: `gates`, `evidence`, `diff`, `tier`, `scope`, `workers`, `report`, `verdict`, `comment`, `html`, for spec-verify. Locally: any of them, plus `audit --audit spec-steward`, which never runs in CI.
|
|
@@ -37,12 +38,17 @@ Run them through the package manager (`yarn spec-tools gates`), the bin (`node_m
|
|
|
37
38
|
1. Pin `@kirill.konshin/lint`; `lint-prepare` links the `spec-steward`, `spec-tools` and `spec-verify` skills.
|
|
38
39
|
2. Declare what binds beyond spec-steward's defaults once, in the root `package.json`: `"spec-steward": { "binds": ["scripts/checks/**", "**/*.Dockerfile", ".gitlab-ci.yml"] }`.
|
|
39
40
|
3. List the gates in the repository's `AGENTS.md` checks: `openspec validate --strict --all`, `spec-tools gates`, and after a requirement changes `spec-tools changed` with `/spec-verify changed` and `spec-tools diff`.
|
|
40
|
-
4. Ignore the generated files
|
|
41
|
+
4. Ignore the generated files with one `.gitignore` line, `.spec-audit/`: every file the tools and the audits write is under it — YAML data, the Markdown and HTML reports, `claude.jsonl`, the job logs. In CI, publish `.spec-audit/` as each audit job's artifacts path.
|
|
41
42
|
5. Wire the pipeline: [GitLab](references/gitlab.md). Other CI providers are not supported yet — `spec-tools` reads GitLab's pipeline variables.
|
|
42
43
|
|
|
43
44
|
## Reading a failure
|
|
44
45
|
|
|
45
46
|
- `gates` — spec-steward's findings come first (the `spec-steward` skill says what each kind asks), then one line per change gate with what violates it and a hint.
|
|
46
47
|
- `evidence` — an open change carries deltas, or is synced but not archived: archive it before the branch merges.
|
|
47
|
-
- `report --gate` / `verdict` — exit 1 is
|
|
48
|
-
- `workers` exit 1 — a worker changed a file outside
|
|
48
|
+
- `report --gate` / `verdict` — exit 1 is `FAIL`, an ERROR above 80% confidence on a gating run; 77 is `ADVISORY` (ERRORs at 71–80%) or any `FAIL` on an advisory run; 3 is `INCOMPLETE`, a part with no valid findings file or requirements nobody judged (the report names them). An ERROR stands only when its judge confirmed it above 70% confidence. A reader's own coverage claims are notes and change no verdict. Summarize `.spec-audit/spec-verify.md` as the `spec-verify` skill's _Interactive use_ says.
|
|
49
|
+
- `workers` exit 1 — a worker changed a file outside `.spec-audit/`; the paths are listed.
|
|
50
|
+
- `workers` log line `still invalid: …` — a worker's YAML file failed its schema after two corrections; the part counts as not read, and the completion pass reads it again.
|
|
51
|
+
|
|
52
|
+
## Language
|
|
53
|
+
|
|
54
|
+
Write finding details, verdict reasons, review items and messages to the reviewer in ASD-STE100 ([common rules](references/rules/common.md#language)).
|
|
@@ -2,6 +2,10 @@
|
|
|
2
2
|
|
|
3
3
|
What `spec-tools` does in every repository that runs it. It ships with `@kirill.konshin/lint`, so a repository is held to the version it pins. The tests beside the source (`spec-tools/src/*.test.ts`) prove the cases below. The corpus gate's own contract — citations, binding, markers, size lines, the scenario ratchet — is spec-steward's [`contract.md`](../../spec-steward/references/contract.md); how a pipeline wires the commands is [`gitlab.md`](gitlab.md).
|
|
4
4
|
|
|
5
|
+
## Files
|
|
6
|
+
|
|
7
|
+
Every file the tools and the audits write is under one folder at the repository root, `.spec-audit/`: one `.gitignore` line, one CI artifacts path. Every intermediate file is YAML — the evidence, the scope, the findings, the verdicts, the merges, the coverage data, the steward review. Markdown is rendered only at the end, for people: `spec-verify.md`, a skill review's `<report>.md`, `coverage.md`, `spec-diff.md` and `mr-comment.md`, each with its `.html` from `spec-tools html`.
|
|
8
|
+
|
|
5
9
|
## Bases
|
|
6
10
|
|
|
7
11
|
Every pipeline variable is read in one module (`ci.ts`); GitLab CI's today. The **gate base** is spec-steward's: a merge request's diff base, none in any other pipeline, locally the merge base with `origin/HEAD` (or `origin/main`). The **diff base** of the diff, the focused check and the audit's affected set is a merge request's diff base, else the merge base with its target branch, else a push's tip before it (`HEAD^` on a branch's first push), else `HEAD`.
|
|
@@ -31,7 +35,7 @@ Cases:
|
|
|
31
35
|
|
|
32
36
|
## Diff — `spec-tools diff [<base>]`
|
|
33
37
|
|
|
34
|
-
The specification change against the diff base (or the given ref), by `### Requirement:` heading, each block in full: a heading only the head has is ADDED, only the base has is REMOVED, a block that differs beyond whitespace is MODIFIED, a removed and an added heading with the same body in one file are one RENAMED; a capability whose file moved verbatim is MOVED and counts as no change. Written to
|
|
38
|
+
The specification change against the diff base (or the given ref), by `### Requirement:` heading, each block in full: a heading only the head has is ADDED, only the base has is REMOVED, a block that differs beyond whitespace is MODIFIED, a removed and an added heading with the same body in one file are one RENAMED; a capability whose file moved verbatim is MOVED and counts as no change. Written to `.spec-audit/spec-diff.md` and printed; it always exits 0.
|
|
35
39
|
|
|
36
40
|
Cases:
|
|
37
41
|
|
|
@@ -41,7 +45,7 @@ Cases:
|
|
|
41
45
|
|
|
42
46
|
## Changed — `spec-tools changed`
|
|
43
47
|
|
|
44
|
-
The author's pre-hand-back check: every requirement a hunk of the diff overlaps (a deletion at its line included), once, plus the requirements an unsynced touched delta names, what the changed files bind or cite, and their related requirements — each with its bound tests, where its own terms occur in the comment-stripped sources, and its related requirements, quoted in full. Written to
|
|
48
|
+
The author's pre-hand-back check: every requirement a hunk of the diff overlaps (a deletion at its line included), once, plus the requirements an unsynced touched delta names, what the changed files bind or cite, and their related requirements — each with its bound tests, where its own terms occur in the comment-stripped sources, and its related requirements, quoted in full. Written to `.spec-audit/parts/changed.yaml` from spec-steward's evidence model (`.spec-audit/evidence.yaml`, written first).
|
|
45
49
|
|
|
46
50
|
## The audit
|
|
47
51
|
|
|
@@ -61,7 +65,7 @@ Every run reads on the cheap model. `AUDIT_MODEL_CHEAP`, `AUDIT_EFFORT_CHEAP`, `
|
|
|
61
65
|
|
|
62
66
|
### Scope — `spec-tools scope`
|
|
63
67
|
|
|
64
|
-
The corpus on the nightly, or wherever there is no merge-request target; otherwise the affected set — what `changed` names — judged with cross-capability checks against the corpus its evidence quotes. The evidence is assembled mechanically first: per capability, every requirement verbatim, its bound tests at their real lines, where its terms occur, its related requirements; citations are not evidence and are left out. The scope is cut into `AUDIT_JOBS` × `AUDIT_SLOTS` parts — one round of workers across the pipeline's reading jobs (`AUDIT_JOBS`, its own count, default 1) — balanced by load, none larger than one cheap reader holds, a capability larger than that cut between its requirements; only a scope larger than a round holds opens more parts. The cut is a function of the evidence and those two numbers. Written to
|
|
68
|
+
The corpus on the nightly, or wherever there is no merge-request target; otherwise the affected set — what `changed` names — judged with cross-capability checks against the corpus its evidence quotes. The evidence is assembled mechanically first: per capability, every requirement verbatim, its bound tests at their real lines, where its terms occur, its related requirements; citations are not evidence and are left out. The scope is cut into `AUDIT_JOBS` × `AUDIT_SLOTS` parts — one round of workers across the pipeline's reading jobs (`AUDIT_JOBS`, its own count, default 1) — balanced by load, none larger than one cheap reader holds, a capability larger than that cut between its requirements; only a scope larger than a round holds opens more parts. The cut is a function of the evidence and those two numbers. Written to `.spec-audit/scope.yaml`, `.spec-audit/evidence.yaml` and `.spec-audit/parts/` (one YAML evidence file per capability or slice, and `units.yaml`).
|
|
65
69
|
|
|
66
70
|
Cases:
|
|
67
71
|
|
|
@@ -70,18 +74,20 @@ Cases:
|
|
|
70
74
|
|
|
71
75
|
### Workers — `spec-tools workers [--complete|--verify]`
|
|
72
76
|
|
|
73
|
-
One headless `claude --print` worker per part loads the `spec-verify` skill and writes its own findings file. In a parallel job (`CI_NODE_INDEX` of `CI_NODE_TOTAL`) it reads only its share: part p goes to job ((p − 1) mod total) + 1. `--complete` judges, as each part's next reader, exactly what the merge listed short; `--verify` puts every standing ERROR to one
|
|
77
|
+
One headless `claude --print` worker per part loads the `spec-verify` skill, reads the judging rules (`references/rules/`), and writes its own YAML findings file, `.spec-audit/parts/findings/part-<n>-<reader>.yaml`. When a worker stops, its file is parsed and checked against its schema; on errors the worker's own session is resumed (`claude --resume <session>`) with the error list, at most twice, and a file still invalid counts as not written — the reader as not run, the verdict as missing. In a parallel job (`CI_NODE_INDEX` of `CI_NODE_TOTAL`) it reads only its share: part p goes to job ((p − 1) mod total) + 1. `--complete` judges, as each part's next reader, exactly what the merge listed short; `--verify` puts every standing ERROR to one judge, which loads no skill: its brief inlines the common and judge rules, and it answers `confirmed` or `warn` with a reason, a production scenario and `judgeConfidence` (0–100). A judge never raises a tier. How many run at once follows the container's memory (300 MB a worker), never more than `AUDIT_WORKERS` (default 2), and each pass logs the job's memory peak against its limit; a worker past `AUDIT_WORKER_TIMEOUT` (20m) is killed and its part judged by the completion pass. A pass that changed any file outside the audit's outputs exits 1, naming the paths; a missing input exits 2. A worker that exits non-zero within its first minute and wrote nothing failed to start (e.g. "Not logged in" when many sessions start at once); it is retried once, at once, rather than left to the completion pass.
|
|
74
78
|
|
|
75
79
|
### Merge — `spec-tools report [--gate] [--worktree]`
|
|
76
80
|
|
|
77
|
-
Nothing is judged twice. Every ERROR's quotes are looked up at their `file:line` in the audited commit (`HEAD`, or the working tree with `--worktree`); a quote not found
|
|
81
|
+
Nothing is judged twice. Every ERROR's quotes are looked up at their `file:line` in the audited commit (`HEAD`, or the working tree with `--worktree`); a quote not found makes it WARN with the reason. Quotes from `.spec-audit/` and from installed dependency code (`node_modules/`) are not the repository's own and count for nothing — an ERROR needs both sides from the repository's specifications and sources, so a run in CI and a run on the working tree grade it alike. An ERROR stays ERROR only when its judge answered `confirmed` with `judgeConfidence` above 70 (`SURE`); any other answer makes it WARN with the reason. An ERROR no judge reached stands only when its `readerConfidence` is above 70, and says so. Kinds are tiered by a fixed list: `code-mismatch` and `undeclared-gap` can be ERROR, `conflict` is WARN at most, `spec-dup` and `untested` are INFO, an unknown kind is INFO; a finding against an ⚠️ Advisory requirement is at most WARN. A requirement of a part that no reader's judged list names was not judged, whatever coverage the reader claimed. Written to `.spec-audit/spec-verify.yaml` (the data, its `verdict` field the verdict) and `.spec-audit/spec-verify.md` (for people, its first line the verdict); each finding shows both confidences and the judge's production scenario.
|
|
78
82
|
|
|
79
|
-
The verdict: `
|
|
83
|
+
The verdict: `FAIL` when an ERROR stands at a confidence above 80 (`BLOCKING`: the judge's, else the reader's); `ADVISORY` when ERRORs stand and none is above 80; `INCOMPLETE` when none stands but a part fell short; `PASS` otherwise. A part falls short only by mechanical signals: no valid findings file (none written, still invalid after corrections, a killed worker), or a requirement id missing from every reader's judged list. What a reader says of its own coverage (`sampled: …`, `not-run: …`) is a claim no tool can prove — a path it names may not exist — so the report lists it under "Readers' coverage notes" and it changes no verdict. The merge never raises a tier: a reader's WARN or INFO stands. `--gate` exits 0 (`PASS`), 1 (`FAIL`), 77 (`ADVISORY`) or 3 (`INCOMPLETE`).
|
|
80
84
|
|
|
81
85
|
Cases:
|
|
82
86
|
|
|
83
87
|
- **WHEN** an ERROR's quote is not at the `file:line` it names **THEN** the report carries it as WARN, naming the quote
|
|
84
|
-
- **WHEN** a
|
|
88
|
+
- **WHEN** a part has no findings file and no ERROR stands **THEN** the verdict is `INCOMPLETE` and `--gate` exits 3
|
|
89
|
+
- **WHEN** a reader claims it sampled a check **THEN** the claim is a note and the verdict does not change
|
|
90
|
+
- **WHEN** the only ERRORs stand at 80% or lower **THEN** the verdict is `ADVISORY` and `--gate` exits 77
|
|
85
91
|
- **WHEN** every check ran in full and no ERROR stands **THEN** the verdict is `PASS`
|
|
86
92
|
|
|
87
93
|
### Local — `spec-tools audit [--here] [--dry]`
|
|
@@ -90,7 +96,7 @@ The same sequence — scope, reading, merge, completion, merge, verification, ga
|
|
|
90
96
|
|
|
91
97
|
### spec-steward's corpus-quality audit — `--audit spec-steward`
|
|
92
98
|
|
|
93
|
-
`scope`, `workers`, `report` and `audit` take `--audit spec-steward` to run spec-steward's audit on the same engine, one repository per run, locally — no pipeline runs it. `workers` and `audit` take `--context <file>`: its text — facts established and the owner's earlier decisions — goes into every worker's and verifier's brief, and a finding or verdict that would reverse one says so. Its scope is the same parts plus one unit per sweep of `skills/spec-steward/sweeps.json`, each reading the files its globs match or the evidence's tests or citations; its workers follow `skills/spec-steward/references/worker.md
|
|
99
|
+
`scope`, `workers`, `report` and `audit` take `--audit spec-steward` to run spec-steward's audit on the same engine, one repository per run, locally — no pipeline runs it. `workers` and `audit` take `--context <file>`: its text — facts established and the owner's earlier decisions — goes into every worker's and verifier's brief, and a finding or verdict that would reverse one says so. Its scope is the same parts plus one unit per sweep of `skills/spec-steward/sweeps.json`, each reading the files its globs match or the evidence's tests or citations; its workers follow `skills/spec-steward/references/worker.md` and the `common` and `spec-steward` judging rules; its verification puts every finding, not only ERRORs, to a judge that keeps, revises or drops it with a `judgeConfidence`. Its merge drops a finding whose quote is not at its line, folds duplicates (one file, line and theme key), offers three or more findings sharing a theme key as a theme, lists the requirements no worker judged as short, writes `.spec-audit/steward/review-data.yaml` and renders the YAML review `.spec-audit/spec-review.yaml` with `spec-tools steward review render`, items at 70% confidence or lower last, under a comment saying so. It never gates: the review file is the owner's to answer.
|
|
94
100
|
|
|
95
101
|
Cases:
|
|
96
102
|
|
|
@@ -100,12 +106,13 @@ Cases:
|
|
|
100
106
|
|
|
101
107
|
## Review glue
|
|
102
108
|
|
|
103
|
-
- `spec-tools run <skill> [--verdict <
|
|
104
|
-
- `spec-tools verdict <file> [--
|
|
105
|
-
- `spec-tools comment <name>=<report.md>...` — the merge-request comment: one row per review from its verdict line, links into the job's artifacts, the coverage report and the diff beneath; written to
|
|
109
|
+
- `spec-tools run <skill> [--verdict <report.yaml>] [--advisory]` — one skill-driven review, headless, on `AUDIT_MODEL`; the stream kept as `.spec-audit/claude.jsonl`, the final result as the job log (`.spec-audit/job-log-<job>.md`). The skill writes its report as YAML (`findings[tier, where, detail, readerConfidence, fields]`, `coverage{complete, notes}`); the report is checked and, on errors, sent back to the same session at most twice. Exits with Claude's status, then gates `--verdict` if given.
|
|
110
|
+
- `spec-tools verdict <file> [--advisory]` — the verdict as an exit code: 0 `PASS`, 1 `FAIL`, 77 `ADVISORY` or `FAIL` on an advisory run (`--advisory` or `AUDIT_GATING=advisory`), 3 `INCOMPLETE`, 1 no verdict or no report. For the spec-verify data it reads `verdict`; for a skill's YAML report it computes the verdict itself, never taking it from the model — an ERROR counts only when its `readerConfidence` is above 70, it is `FAIL` above 80 and `ADVISORY` otherwise, and `coverage.complete: false` is `INCOMPLETE` — and renders the Markdown report beside it (`<report>.md`, its first line the verdict).
|
|
111
|
+
- `spec-tools comment <name>=<report.md>...` — the merge-request comment: one row per review from its verdict line, links into the job's artifacts, the coverage report and the diff beneath; written to `.spec-audit/mr-comment.md`.
|
|
106
112
|
- `spec-tools html <file.md>...` — each report as self-contained HTML beside it.
|
|
107
113
|
|
|
108
114
|
Cases:
|
|
109
115
|
|
|
110
116
|
- **WHEN** a review's report says `FAIL` on an advisory run **THEN** `verdict` exits 77
|
|
111
|
-
- **WHEN**
|
|
117
|
+
- **WHEN** a skill's report holds one ERROR at 70% confidence and nothing else **THEN** `verdict` computes `PASS` and exits 0
|
|
118
|
+
- **WHEN** a review wrote no report **THEN** `verdict` exits 1
|