@kirill.konshin/lint 0.6.3 → 0.6.5

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -5,10 +5,11 @@ description: The machinery around both audits — set up and run the `spec-tools
5
5
 
6
6
  # spec-tools
7
7
 
8
- One CLI, `spec-tools`, installed with `@kirill.konshin/lint` beside `spec-steward`. What each command does, reads and writes, and its exit codes are [references/contract.md](references/contract.md); a GitLab pipeline is [references/gitlab.md](references/gitlab.md). Three things live elsewhere:
8
+ One CLI, `spec-tools`, installed with `@kirill.konshin/lint` beside `spec-steward`. What each command does, reads and writes, and its exit codes are [references/contract.md](references/contract.md); a GitLab pipeline is [references/gitlab.md](references/gitlab.md). Four things live elsewhere:
9
9
 
10
10
  - the corpus gate — citations, binding, markers, the scenario ratchet — is the `spec-steward` skill;
11
- - how an audit worker judges — the five checks, the tiers, the finding kinds, the findings file — is the `spec-verify` skill, which `spec-tools workers` hands every worker;
11
+ - what an audit judges — the production effect, the tiers, the confidence, the checks — is the judging rules in [references/rules/](references/rules): `common.md`, then one file per audit role (`spec-verify.md`, `spec-verify-judge.md`, `spec-steward.md`);
12
+ - how an audit worker runs and writes its findings file is the `spec-verify` skill, which `spec-tools workers` hands every worker;
12
13
  - how the repository makes a change is its own `AGENTS.md`.
13
14
 
14
15
  ## Commands
@@ -20,10 +21,10 @@ One CLI, `spec-tools`, installed with `@kirill.konshin/lint` beside `spec-stewar
20
21
  | What changed in the specifications, by requirement | `spec-tools diff [<base>]` |
21
22
  | Check a spec edit before handing it back | `spec-tools changed`, then `/spec-verify changed` |
22
23
  | The full code-conformance audit, locally | `spec-tools audit` (`--here` for uncommitted edits, `--dry` to stop after the scope) |
23
- | spec-steward's corpus-quality audit, on the same engine | `spec-tools audit --audit spec-steward`, then consolidate `spec-review.md` as the `spec-steward` skill says |
24
+ | spec-steward's corpus-quality audit, on the same engine | `spec-tools audit --audit spec-steward`, then consolidate `.spec-audit/spec-review.yaml` as the `spec-steward` skill says |
24
25
  | One pass of a CI audit | `spec-tools tier`, `scope`, `workers [--complete\|--verify]`, `report [--gate]` |
25
- | A headless skill review in CI | `spec-tools run <skill> --verdict <report>` |
26
- | Gate a review's report | `spec-tools verdict <report>` |
26
+ | A headless skill review in CI | `spec-tools run <skill> --verdict .spec-audit/<report>.yaml` |
27
+ | Gate a review's report | `spec-tools verdict .spec-audit/spec-verify.yaml` or `.spec-audit/<report>.yaml` |
27
28
  | The merge-request comment and HTML reports | `spec-tools comment <name>=<report.md>...`, `spec-tools html <file.md>...` |
28
29
 
29
30
  In CI: `gates`, `evidence`, `diff`, `tier`, `scope`, `workers`, `report`, `verdict`, `comment`, `html`, for spec-verify. Locally: any of them, plus `audit --audit spec-steward`, which never runs in CI.
@@ -37,12 +38,17 @@ Run them through the package manager (`yarn spec-tools gates`), the bin (`node_m
37
38
  1. Pin `@kirill.konshin/lint`; `lint-prepare` links the `spec-steward`, `spec-tools` and `spec-verify` skills.
38
39
  2. Declare what binds beyond spec-steward's defaults once, in the root `package.json`: `"spec-steward": { "binds": ["scripts/checks/**", "**/*.Dockerfile", ".gitlab-ci.yml"] }`.
39
40
  3. List the gates in the repository's `AGENTS.md` checks: `openspec validate --strict --all`, `spec-tools gates`, and after a requirement changes `spec-tools changed` with `/spec-verify changed` and `spec-tools diff`.
40
- 4. Ignore the generated files: `spec-evidence.json`, `spec-coverage.md`, `spec-diff.md`, `audit-scope.json`, `audit-parts/`, `audit-tier.env`, `spec-verify.*`, `mr-comment.md`, `claude.jsonl`, `job-log-*.md` and the rendered `*.html`.
41
+ 4. Ignore the generated files with one `.gitignore` line, `.spec-audit/`: every file the tools and the audits write is under it — YAML data, the Markdown and HTML reports, `claude.jsonl`, the job logs. In CI, publish `.spec-audit/` as each audit job's artifacts path.
41
42
  5. Wire the pipeline: [GitLab](references/gitlab.md). Other CI providers are not supported yet — `spec-tools` reads GitLab's pipeline variables.
42
43
 
43
44
  ## Reading a failure
44
45
 
45
46
  - `gates` — spec-steward's findings come first (the `spec-steward` skill says what each kind asks), then one line per change gate with what violates it and a hint.
46
47
  - `evidence` — an open change carries deltas, or is synced but not archived: archive it before the branch merges.
47
- - `report --gate` / `verdict` — exit 1 is a confirmed ERROR on a gating run, 77 the same on an advisory run, 3 `INCOMPLETE` (the report names the check that fell short). Summarize `spec-verify.md` as the `spec-verify` skill's _Interactive use_ says.
48
- - `workers` exit 1 — a worker changed a file outside the audit's outputs; the paths are listed.
48
+ - `report --gate` / `verdict` — exit 1 is `FAIL`, an ERROR above 80% confidence on a gating run; 77 is `ADVISORY` (ERRORs at 71–80%) or any `FAIL` on an advisory run; 3 is `INCOMPLETE`, a part with no valid findings file or requirements nobody judged (the report names them). An ERROR stands only when its judge confirmed it above 70% confidence. A reader's own coverage claims are notes and change no verdict. Summarize `.spec-audit/spec-verify.md` as the `spec-verify` skill's _Interactive use_ says.
49
+ - `workers` exit 1 — a worker changed a file outside `.spec-audit/`; the paths are listed.
50
+ - `workers` log line `still invalid: …` — a worker's YAML file failed its schema after two corrections; the part counts as not read, and the completion pass reads it again.
51
+
52
+ ## Language
53
+
54
+ Write finding details, verdict reasons, review items and messages to the reviewer in ASD-STE100 ([common rules](references/rules/common.md#language)).
@@ -2,6 +2,10 @@
2
2
 
3
3
  What `spec-tools` does in every repository that runs it. It ships with `@kirill.konshin/lint`, so a repository is held to the version it pins. The tests beside the source (`spec-tools/src/*.test.ts`) prove the cases below. The corpus gate's own contract — citations, binding, markers, size lines, the scenario ratchet — is spec-steward's [`contract.md`](../../spec-steward/references/contract.md); how a pipeline wires the commands is [`gitlab.md`](gitlab.md).
4
4
 
5
+ ## Files
6
+
7
+ Every file the tools and the audits write is under one folder at the repository root, `.spec-audit/`: one `.gitignore` line, one CI artifacts path. Every intermediate file is YAML — the evidence, the scope, the findings, the verdicts, the merges, the coverage data, the steward review. Markdown is rendered only at the end, for people: `spec-verify.md`, a skill review's `<report>.md`, `coverage.md`, `spec-diff.md` and `mr-comment.md`, each with its `.html` from `spec-tools html`.
8
+
5
9
  ## Bases
6
10
 
7
11
  Every pipeline variable is read in one module (`ci.ts`); GitLab CI's today. The **gate base** is spec-steward's: a merge request's diff base, none in any other pipeline, locally the merge base with `origin/HEAD` (or `origin/main`). The **diff base** of the diff, the focused check and the audit's affected set is a merge request's diff base, else the merge base with its target branch, else a push's tip before it (`HEAD^` on a branch's first push), else `HEAD`.
@@ -31,7 +35,7 @@ Cases:
31
35
 
32
36
  ## Diff — `spec-tools diff [<base>]`
33
37
 
34
- The specification change against the diff base (or the given ref), by `### Requirement:` heading, each block in full: a heading only the head has is ADDED, only the base has is REMOVED, a block that differs beyond whitespace is MODIFIED, a removed and an added heading with the same body in one file are one RENAMED; a capability whose file moved verbatim is MOVED and counts as no change. Written to `spec-diff.md` and printed; it always exits 0.
38
+ The specification change against the diff base (or the given ref), by `### Requirement:` heading, each block in full: a heading only the head has is ADDED, only the base has is REMOVED, a block that differs beyond whitespace is MODIFIED, a removed and an added heading with the same body in one file are one RENAMED; a capability whose file moved verbatim is MOVED and counts as no change. Written to `.spec-audit/spec-diff.md` and printed; it always exits 0.
35
39
 
36
40
  Cases:
37
41
 
@@ -41,7 +45,7 @@ Cases:
41
45
 
42
46
  ## Changed — `spec-tools changed`
43
47
 
44
- The author's pre-hand-back check: every requirement a hunk of the diff overlaps (a deletion at its line included), once, plus the requirements an unsynced touched delta names, what the changed files bind or cite, and their related requirements — each with its bound tests, where its own terms occur in the comment-stripped sources, and its related requirements, quoted in full. Written to `audit-parts/changed.md` from spec-steward's evidence model (`spec-evidence.json`, written first).
48
+ The author's pre-hand-back check: every requirement a hunk of the diff overlaps (a deletion at its line included), once, plus the requirements an unsynced touched delta names, what the changed files bind or cite, and their related requirements — each with its bound tests, where its own terms occur in the comment-stripped sources, and its related requirements, quoted in full. Written to `.spec-audit/parts/changed.yaml` from spec-steward's evidence model (`.spec-audit/evidence.yaml`, written first).
45
49
 
46
50
  ## The audit
47
51
 
@@ -61,7 +65,7 @@ Every run reads on the cheap model. `AUDIT_MODEL_CHEAP`, `AUDIT_EFFORT_CHEAP`, `
61
65
 
62
66
  ### Scope — `spec-tools scope`
63
67
 
64
- The corpus on the nightly, or wherever there is no merge-request target; otherwise the affected set — what `changed` names — judged with cross-capability checks against the corpus its evidence quotes. The evidence is assembled mechanically first: per capability, every requirement verbatim, its bound tests at their real lines, where its terms occur, its related requirements; citations are not evidence and are left out. The scope is cut into `AUDIT_JOBS` × `AUDIT_SLOTS` parts — one round of workers across the pipeline's reading jobs (`AUDIT_JOBS`, its own count, default 1) — balanced by load, none larger than one cheap reader holds, a capability larger than that cut between its requirements; only a scope larger than a round holds opens more parts. The cut is a function of the evidence and those two numbers. Written to `audit-scope.json` and `audit-parts/`.
68
+ The corpus on the nightly, or wherever there is no merge-request target; otherwise the affected set — what `changed` names — judged with cross-capability checks against the corpus its evidence quotes. The evidence is assembled mechanically first: per capability, every requirement verbatim, its bound tests at their real lines, where its terms occur, its related requirements; citations are not evidence and are left out. The scope is cut into `AUDIT_JOBS` × `AUDIT_SLOTS` parts — one round of workers across the pipeline's reading jobs (`AUDIT_JOBS`, its own count, default 1) — balanced by load, none larger than one cheap reader holds, a capability larger than that cut between its requirements; only a scope larger than a round holds opens more parts. The cut is a function of the evidence and those two numbers. Written to `.spec-audit/scope.yaml`, `.spec-audit/evidence.yaml` and `.spec-audit/parts/` (one YAML evidence file per capability or slice, and `units.yaml`).
65
69
 
66
70
  Cases:
67
71
 
@@ -70,18 +74,20 @@ Cases:
70
74
 
71
75
  ### Workers — `spec-tools workers [--complete|--verify]`
72
76
 
73
- One headless `claude --print` worker per part loads the `spec-verify` skill and writes its own findings file. In a parallel job (`CI_NODE_INDEX` of `CI_NODE_TOTAL`) it reads only its share: part p goes to job ((p − 1) mod total) + 1. `--complete` judges, as each part's next reader, exactly what the merge listed short; `--verify` puts every standing ERROR to one verifier that confirms it only when the divergence is real and critical. How many run at once follows the container's memory (300 MB a worker), never more than `AUDIT_WORKERS` (default 2), and each pass logs the job's memory peak against its limit; a worker past `AUDIT_WORKER_TIMEOUT` (20m) is killed and its part judged by the completion pass. A pass that changed any file outside the audit's outputs exits 1, naming the paths; a missing input exits 2.
77
+ One headless `claude --print` worker per part loads the `spec-verify` skill, reads the judging rules (`references/rules/`), and writes its own YAML findings file, `.spec-audit/parts/findings/part-<n>-<reader>.yaml`. When a worker stops, its file is parsed and checked against its schema; on errors the worker's own session is resumed (`claude --resume <session>`) with the error list, at most twice, and a file still invalid counts as not written — the reader as not run, the verdict as missing. In a parallel job (`CI_NODE_INDEX` of `CI_NODE_TOTAL`) it reads only its share: part p goes to job ((p − 1) mod total) + 1. `--complete` judges, as each part's next reader, exactly what the merge listed short; `--verify` puts every standing ERROR to one judge, which loads no skill: its brief inlines the common and judge rules, and it answers `confirmed` or `warn` with a reason, a production scenario and `judgeConfidence` (0–100). A judge never raises a tier. How many run at once follows the container's memory (300 MB a worker), never more than `AUDIT_WORKERS` (default 2), and each pass logs the job's memory peak against its limit; a worker past `AUDIT_WORKER_TIMEOUT` (20m) is killed and its part judged by the completion pass. A pass that changed any file outside the audit's outputs exits 1, naming the paths; a missing input exits 2. A worker that exits non-zero within its first minute and wrote nothing failed to start (e.g. "Not logged in" when many sessions start at once); it is retried once, at once, rather than left to the completion pass.
74
78
 
75
79
  ### Merge — `spec-tools report [--gate] [--worktree]`
76
80
 
77
- Nothing is judged twice. Every ERROR's quotes are looked up at their `file:line` in the audited commit (`HEAD`, or the working tree with `--worktree`); a quote not found, or a verifier's `warn`, makes it WARN with the reason; one the verifier never reached stands on its quotes and says so. Kinds are tiered by the fixed list in the `spec-verify` skill, an unknown kind is INFO, and a finding against an ⚠️ Advisory requirement is at most WARN. A requirement of a part that no reader's judged list names was not judged, whatever coverage the reader claimed. Written to `spec-verify.md` (its first line the verdict) and `spec-verify.json`.
81
+ Nothing is judged twice. Every ERROR's quotes are looked up at their `file:line` in the audited commit (`HEAD`, or the working tree with `--worktree`); a quote not found makes it WARN with the reason. Quotes from `.spec-audit/` and from installed dependency code (`node_modules/`) are not the repository's own and count for nothing — an ERROR needs both sides from the repository's specifications and sources, so a run in CI and a run on the working tree grade it alike. An ERROR stays ERROR only when its judge answered `confirmed` with `judgeConfidence` above 70 (`SURE`); any other answer makes it WARN with the reason. An ERROR no judge reached stands only when its `readerConfidence` is above 70, and says so. Kinds are tiered by a fixed list: `code-mismatch` and `undeclared-gap` can be ERROR, `conflict` is WARN at most, `spec-dup` and `untested` are INFO, an unknown kind is INFO; a finding against an ⚠️ Advisory requirement is at most WARN. A requirement of a part that no reader's judged list names was not judged, whatever coverage the reader claimed. Written to `.spec-audit/spec-verify.yaml` (the data, its `verdict` field the verdict) and `.spec-audit/spec-verify.md` (for people, its first line the verdict); each finding shows both confidences and the judge's production scenario.
78
82
 
79
- The verdict: `PASS` when every requirement in scope was judged, every check ran in full and no ERROR stands; `FAIL` when an ERROR stands; `INCOMPLETE` when none stands but a check was sampled or skipped — the report names it. `--gate` exits 0, 1 or 3.
83
+ The verdict: `FAIL` when an ERROR stands at a confidence above 80 (`BLOCKING`: the judge's, else the reader's); `ADVISORY` when ERRORs stand and none is above 80; `INCOMPLETE` when none stands but a part fell short; `PASS` otherwise. A part falls short only by mechanical signals: no valid findings file (none written, still invalid after corrections, a killed worker), or a requirement id missing from every reader's judged list. What a reader says of its own coverage (`sampled: …`, `not-run: …`) is a claim no tool can prove — a path it names may not exist — so the report lists it under "Readers' coverage notes" and it changes no verdict. The merge never raises a tier: a reader's WARN or INFO stands. `--gate` exits 0 (`PASS`), 1 (`FAIL`), 77 (`ADVISORY`) or 3 (`INCOMPLETE`).
80
84
 
81
85
  Cases:
82
86
 
83
87
  - **WHEN** an ERROR's quote is not at the `file:line` it names **THEN** the report carries it as WARN, naming the quote
84
- - **WHEN** a check was sampled and no ERROR stands **THEN** the verdict is `INCOMPLETE` and `--gate` exits 3
88
+ - **WHEN** a part has no findings file and no ERROR stands **THEN** the verdict is `INCOMPLETE` and `--gate` exits 3
89
+ - **WHEN** a reader claims it sampled a check **THEN** the claim is a note and the verdict does not change
90
+ - **WHEN** the only ERRORs stand at 80% or lower **THEN** the verdict is `ADVISORY` and `--gate` exits 77
85
91
  - **WHEN** every check ran in full and no ERROR stands **THEN** the verdict is `PASS`
86
92
 
87
93
  ### Local — `spec-tools audit [--here] [--dry]`
@@ -90,7 +96,7 @@ The same sequence — scope, reading, merge, completion, merge, verification, ga
90
96
 
91
97
  ### spec-steward's corpus-quality audit — `--audit spec-steward`
92
98
 
93
- `scope`, `workers`, `report` and `audit` take `--audit spec-steward` to run spec-steward's audit on the same engine, one repository per run, locally — no pipeline runs it. `workers` and `audit` take `--context <file>`: its text — facts established and the owner's earlier decisions — goes into every worker's and verifier's brief, and a finding or verdict that would reverse one says so. Its scope is the same parts plus one unit per sweep of `skills/spec-steward/sweeps.json`, each reading the files its globs match or the evidence's tests or citations; its workers follow `skills/spec-steward/references/worker.md`; its verification puts every finding, not only ERRORs, to a skeptic that keeps, revises or drops it. Its merge drops a finding whose quote is not at its line, folds duplicates (one file, line and theme key), offers three or more findings sharing a theme key as a theme, lists the requirements no worker judged as short, writes `audit-parts/steward/review.json` and renders `spec-review.md` with `spec-tools steward review render`. It never gates: the review file is the owner's to answer.
99
+ `scope`, `workers`, `report` and `audit` take `--audit spec-steward` to run spec-steward's audit on the same engine, one repository per run, locally — no pipeline runs it. `workers` and `audit` take `--context <file>`: its text — facts established and the owner's earlier decisions — goes into every worker's and verifier's brief, and a finding or verdict that would reverse one says so. Its scope is the same parts plus one unit per sweep of `skills/spec-steward/sweeps.json`, each reading the files its globs match or the evidence's tests or citations; its workers follow `skills/spec-steward/references/worker.md` and the `common` and `spec-steward` judging rules; its verification puts every finding, not only ERRORs, to a judge that keeps, revises or drops it with a `judgeConfidence`. Its merge drops a finding whose quote is not at its line, folds duplicates (one file, line and theme key), offers three or more findings sharing a theme key as a theme, lists the requirements no worker judged as short, writes `.spec-audit/steward/review-data.yaml` and renders the YAML review `.spec-audit/spec-review.yaml` with `spec-tools steward review render`, items at 70% confidence or lower last, under a comment saying so. It never gates: the review file is the owner's to answer.
94
100
 
95
101
  Cases:
96
102
 
@@ -100,12 +106,13 @@ Cases:
100
106
 
101
107
  ## Review glue
102
108
 
103
- - `spec-tools run <skill> [--verdict <file>] [--advisory]` — one skill-driven review, headless, on `AUDIT_MODEL`; the stream kept as `claude.jsonl`, the final result as the job log (`job-log-<job>.md`); exits with Claude's status, then gates `--verdict` if given.
104
- - `spec-tools verdict <file> [--log <file>] [--advisory]` — the report's verdict (a Markdown first line, or `.verdict` of a JSON report) as an exit code: 0 `PASS`, 1 `FAIL`, 77 `FAIL` on an advisory run (`--advisory` or `AUDIT_GATING=advisory`), 3 `INCOMPLETE`, 1 no verdict. A report the skill printed but did not write is recovered from the job log when it carries a verdict line.
105
- - `spec-tools comment <name>=<report.md>...` — the merge-request comment: one row per review from its verdict line, links into the job's artifacts, the coverage report and the diff beneath; written to `mr-comment.md`.
109
+ - `spec-tools run <skill> [--verdict <report.yaml>] [--advisory]` — one skill-driven review, headless, on `AUDIT_MODEL`; the stream kept as `.spec-audit/claude.jsonl`, the final result as the job log (`.spec-audit/job-log-<job>.md`). The skill writes its report as YAML (`findings[tier, where, detail, readerConfidence, fields]`, `coverage{complete, notes}`); the report is checked and, on errors, sent back to the same session at most twice. Exits with Claude's status, then gates `--verdict` if given.
110
+ - `spec-tools verdict <file> [--advisory]` — the verdict as an exit code: 0 `PASS`, 1 `FAIL`, 77 `ADVISORY` or `FAIL` on an advisory run (`--advisory` or `AUDIT_GATING=advisory`), 3 `INCOMPLETE`, 1 no verdict or no report. For the spec-verify data it reads `verdict`; for a skill's YAML report it computes the verdict itself, never taking it from the model — an ERROR counts only when its `readerConfidence` is above 70, it is `FAIL` above 80 and `ADVISORY` otherwise, and `coverage.complete: false` is `INCOMPLETE` — and renders the Markdown report beside it (`<report>.md`, its first line the verdict).
111
+ - `spec-tools comment <name>=<report.md>...` — the merge-request comment: one row per review from its verdict line, links into the job's artifacts, the coverage report and the diff beneath; written to `.spec-audit/mr-comment.md`.
106
112
  - `spec-tools html <file.md>...` — each report as self-contained HTML beside it.
107
113
 
108
114
  Cases:
109
115
 
110
116
  - **WHEN** a review's report says `FAIL` on an advisory run **THEN** `verdict` exits 77
111
- - **WHEN** the report is missing and the job log carries `PASS (0 errors, 2 warnings)` **THEN** the report is written from the log and `verdict` exits 0
117
+ - **WHEN** a skill's report holds one ERROR at 70% confidence and nothing else **THEN** `verdict` computes `PASS` and exits 0
118
+ - **WHEN** a review wrote no report **THEN** `verdict` exits 1
@@ -7,7 +7,7 @@ The jobs a GitLab repository runs around `spec-steward` and `spec-tools`. Names
7
7
  - `openspec validate` and `spec-tools gates` are gates over repository state, not unit tests: each runs as a job of its own, never inside a unit suite. An error fails its job; a warning never does.
8
8
  - The diff, the coverage report and the merge-request comment are reports: they never fail a pipeline.
9
9
  - `workflow-evidence` runs on merge requests to the default branch, and on default-branch pushes, where a red holds the release.
10
- - The audit jobs accept exit 3 (`INCOMPLETE`) and 77 (an advisory `FAIL`) as allowed failures; a gating `FAIL` (exit 1) blocks.
10
+ - The audit jobs accept exit 3 (`INCOMPLETE`) and 77 (`ADVISORY`, ERRORs at 80% confidence or lower, or a `FAIL` on an advisory run) as allowed failures; a gating `FAIL` (exit 1, an ERROR above 80%) blocks.
11
11
  - Report jobs depend on the audits with `dependencies:` and `when: always`, never `needs:` — a failed audit is exactly when its report matters.
12
12
  - The default-branch pipeline publishes the reports on Pages, a failing one included; a report a run did not produce is omitted.
13
13
  - The scheduled nightly sets `SPEC_AUDIT_NIGHTLY=true`, so `spec-tools tier` audits the corpus there.
@@ -40,6 +40,8 @@ Cache `.claude-code/` keyed on `CLAUDE_CODE_VERSION`, and gitignore it. The job
40
40
 
41
41
  ## Jobs
42
42
 
43
+ Every job writes under `.spec-audit/` and publishes that one folder; the repository ignores it with one `.gitignore` line, `.spec-audit/`. Jobs that hand files on through `dependencies:` or `needs:` merge their `.spec-audit/` folders.
44
+
43
45
  ```yaml
44
46
  openspec:
45
47
  stage: review
@@ -60,31 +62,25 @@ spec-diff:
60
62
  stage: review
61
63
  rules: [if: $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == $CI_DEFAULT_BRANCH]
62
64
  script: [yarn spec-tools diff]
63
- artifacts: { when: always, paths: [spec-diff.md] }
65
+ artifacts: { when: always, paths: [.spec-audit/] }
64
66
 
65
67
  # The run class, the coverage report and the audit's inputs, handed on as artifacts — with the package's skills/,
66
68
  # which hold the CLI the AI jobs run.
67
69
  spec-coverage:
68
70
  stage: review
69
71
  script:
70
- - yarn spec-tools tier | tee audit-tier.env
71
- - set -a && . ./audit-tier.env && set +a
72
- - yarn spec-tools steward coverage --out spec-coverage.md
72
+ - mkdir -p .spec-audit
73
+ - yarn spec-tools tier | tee .spec-audit/tier.env
74
+ - set -a && . ./.spec-audit/tier.env && set +a
75
+ - yarn spec-tools steward coverage
73
76
  - yarn spec-tools scope
74
77
  artifacts:
75
78
  when: always
76
- reports: { dotenv: audit-tier.env }
77
- paths:
78
- [
79
- audit-tier.env,
80
- spec-coverage.md,
81
- audit-scope.json,
82
- audit-parts/,
83
- node_modules/@kirill.konshin/lint/skills/,
84
- ]
79
+ reports: { dotenv: .spec-audit/tier.env }
80
+ paths: [.spec-audit/, node_modules/@kirill.konshin/lint/skills/]
85
81
 
86
82
  # The audit, in one job: read (a failed reading leaves its parts to the completion pass), merge, complete what the
87
- # reading left short, verify every ERROR, gate.
83
+ # reading left short, judge every ERROR, gate.
88
84
  spec-verify:
89
85
  stage: review
90
86
  image: <an image with node>
@@ -99,10 +95,10 @@ spec-verify:
99
95
  - node node_modules/@kirill.konshin/lint/skills/spec-tools/scripts/cli.js report
100
96
  - node node_modules/@kirill.konshin/lint/skills/spec-tools/scripts/cli.js workers --verify
101
97
  - node node_modules/@kirill.konshin/lint/skills/spec-tools/scripts/cli.js report
102
- - node node_modules/@kirill.konshin/lint/skills/spec-tools/scripts/cli.js verdict spec-verify.json
103
- artifacts: { when: always, paths: [spec-verify.md, spec-verify.json, audit-parts/findings/] }
98
+ - node node_modules/@kirill.konshin/lint/skills/spec-tools/scripts/cli.js verdict .spec-audit/spec-verify.yaml
99
+ artifacts: { when: always, paths: [.spec-audit/] }
104
100
 
105
- # Any other skill-driven review: headless, then its verdict.
101
+ # Any other skill-driven review: headless, its YAML report checked, then its verdict computed and its Markdown rendered.
106
102
  some-review:
107
103
  stage: review
108
104
  image: <an image with node>
@@ -110,8 +106,8 @@ some-review:
110
106
  allow_failure: { exit_codes: [3, 77] }
111
107
  script:
112
108
  - *claude-code
113
- - node node_modules/@kirill.konshin/lint/skills/spec-tools/scripts/cli.js run some-review --verdict some-review.md --advisory
114
- artifacts: { when: always, paths: [some-review.md, claude.jsonl, 'job-log-*.md'] }
109
+ - node node_modules/@kirill.konshin/lint/skills/spec-tools/scripts/cli.js run some-review --verdict .spec-audit/some-review.yaml --advisory
110
+ artifacts: { when: always, paths: [.spec-audit/] }
115
111
 
116
112
  render-reviews:
117
113
  stage: deploy
@@ -122,11 +118,13 @@ render-reviews:
122
118
  - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
123
119
  when: always
124
120
  script:
125
- - yarn spec-tools html spec-verify.md some-review.md spec-coverage.md spec-diff.md
126
- - yarn spec-tools comment spec-verify=spec-verify.md some-review=some-review.md
121
+ - yarn spec-tools html .spec-audit/spec-verify.md .spec-audit/some-review.md .spec-audit/coverage.md .spec-audit/spec-diff.md
122
+ - yarn spec-tools comment spec-verify=.spec-audit/spec-verify.md some-review=.spec-audit/some-review.md
127
123
  # Posting needs a token that may write notes; CI_JOB_TOKEN cannot.
128
- - '[ -z "$CI_MERGE_REQUEST_IID" ] || glab api --method POST "projects/$CI_PROJECT_ID/merge_requests/$CI_MERGE_REQUEST_IID/notes" -f "body=$(cat mr-comment.md)"'
129
- artifacts: { when: always, paths: ['*.md', '*.html'] }
124
+ - '[ -z "$CI_MERGE_REQUEST_IID" ] || glab api --method POST "projects/$CI_PROJECT_ID/merge_requests/$CI_MERGE_REQUEST_IID/notes" -f "body=$(cat .spec-audit/mr-comment.md)"'
125
+ artifacts: { when: always, paths: [.spec-audit/] }
130
126
  ```
131
127
 
132
128
  A release job that must wait for the audit `needs:` the judge (`spec-verify`), so a gating `FAIL` on the default branch holds it.
129
+
130
+ A skill-driven review writes `.spec-audit/<review>.yaml` in the shape `spec-tools run` checks (`findings[tier, where, detail, readerConfidence, fields]`, `coverage{complete, notes}`). `verdict` computes its state from that file, never from the model's words, and writes `.spec-audit/<review>.md` beside it for the comment and the HTML.
@@ -0,0 +1,46 @@
1
+ # Judging rules: common
2
+
3
+ These rules tell what a finding is and which tier it gets. They apply to every audit that spec-tools runs: spec-verify, its judge, spec-steward and check-entity-hacks. The skill and the tool tell how a run works.
4
+
5
+ ## Evidence
6
+
7
+ - Judge what the repository shows: the specification as written, the code, the tests and the configuration.
8
+ - A comment, a citation, a description, a marker (`SANCTIONED`, `FIXME`, `TODO`) or prompt text is not evidence. It is also not a defect.
9
+ - Behaviour that comes from a dependency (a command-line flag, a library default) is not demonstrated when the repository does not show it.
10
+ - Installed dependency code (`node_modules/`) is not the repository. Do not judge it, and do not use it as evidence. A defect in a dependency is for that dependency's own tests, not for this audit.
11
+ - A finding is demonstrated when you read both sides and quote each side at `file:line`. Other findings are suspected.
12
+ - A finding names the difference. The owner decides which side to change.
13
+
14
+ ## Production effect
15
+
16
+ - A real defect has a production effect when it does one of these in production:
17
+ - It breaks a flow. An operation fails, stops, does not stop, or does a completely wrong thing (strong disconnect between requirement and effect: save deletes, update goes nowhere, etc).
18
+ - It violates a contract or an invariant: an interface, a data rule, an order, an identity, or a write to the wrong record.
19
+ - It leaks sensitive data: credentials, tokens or personal data go to logs, to users or to another system.
20
+ - It causes a performance problem: work without a limit, a loop that does not stop, or a resource that is not released.
21
+ - Code that does not run in production has no production effect. This includes tests, local-only and development-only modes, demo agents, and internal CI or developer scripts. The exception: such code lets a production defect through now, and you show an instance.
22
+ - A difference in words, names, structure or style has no production effect when the behaviour stays the same.
23
+
24
+ ## Tiers
25
+
26
+ - ERROR: a demonstrated defect with a production effect. Write the production scenario in one sentence: what occurs, to whom, and what breaks, leaks or becomes slow.
27
+ - WARN: a suspected defect, or a demonstrated defect with no production effect.
28
+ - INFO: a state or a risk for a reviewer. It is never a failure.
29
+ - A finding against a requirement marked `**⚠️ Advisory:**` is WARN or lower.
30
+ - Use the effect to find the tier. The size of the difference does not decide the tier.
31
+
32
+ ## Confidence
33
+
34
+ - Give each finding `readerConfidence`, an integer from 0 to 100. It tells how sure you are that the finding is an ERROR as ERROR rule in Tiers section tells. It includes the defect and its production effect.
35
+ - Use a value below 50 when you could not read a side, or when the effect needs a condition that you could not confirm.
36
+ - Assess if defect is on critical path or is an edge case. Issues not on critical path or low severity edge cases are a WARN. Edge cases that may lead to real production defect may be ERROR. Reflect this in `readerConfidence`.
37
+
38
+ ## Scope
39
+
40
+ - Report each defect one time, at its final tier.
41
+ - Do not report a thing that a deterministic tool reports: coverage, citations, gates, lint.
42
+ - Do not report style preferences.
43
+
44
+ ## Language
45
+
46
+ - Write finding details, verdict reasons, review items and messages to the reviewer in ASD-STE100: short sentences, active voice, one statement in each sentence, one word for one meaning. Technical names (files, functions, fields) stay as they are.
@@ -0,0 +1,71 @@
1
+ # Judging rules: spec-steward
2
+
3
+ Use the common rules first: [common.md](common.md). How to write a specification is in the spec-steward [model](../../../spec-steward/references/model.md).
4
+
5
+ This audit judges if the specification holds the correct rules with the correct evidence. It reports problems in the specification. Defects in the code are for spec-verify.
6
+
7
+ ## Items to report
8
+
9
+ 1. A REQUIRED rule that must be ⚠️ Advisory (see Severity).
10
+ 2. A rule that belongs in `AGENTS.md`, a scoped rule, a skill, a command, the OpenSpec configuration or a README. The placement guide is `rules/agent.md`.
11
+ 3. A rule that a deterministic tool can enforce better than an AI audit.
12
+ 4. A rule that causes tests of low value or artificial tests.
13
+ 5. A rule whose tests satisfy the wording while failing to protect the intent.
14
+ 6. A rule that you cannot verify in a reasonable way.
15
+ 7. A rule whose words permit different meanings. This includes a vague word in a REQUIRED statement (thin, safe, lean, at parity, needed).
16
+ 8. A rule that describes implementation detail, not an invariant that is important outside the code.
17
+ 9. A rule redundant with, or substantially overlapping, another.
18
+ 10. Contradictory rules.
19
+ 11. A rule that no longer matches the implementation or architecture.
20
+ 12. An implementation that shows that the specification is wrong or old.
21
+ 13. A missing invariant of high value that the architecture uses.
22
+ 14. A rule with an unclear scope.
23
+ 15. A rule that needs reasoning about the full repository when one subsystem is sufficient.
24
+ 16. A rule that causes agents to change a specification only to make the code pass.
25
+ 17. A rule whose verification uses comments or text matching.
26
+ 18. A rule whose cost is too high for the failure that it prevents.
27
+ 19. Requirements or scenarios that repeat each other.
28
+ 20. A scenario structure that adds words but not precision.
29
+ 21. One concern that two or more mechanisms hold. Name the owner and the copies to remove.
30
+ 22. A rule that more than one repository shares, with different words.
31
+ 23. A CI or audit configuration that costs more than the signal it gives.
32
+ 24. Citation machinery that costs more to keep than it gives.
33
+
34
+ Items 5, 9, 10 and 11 have the same names as spec-verify's checks 5, 3, 2 and 4. spec-verify judges them against the code. This audit judges if the rule itself must change.
35
+
36
+ ## How to judge
37
+
38
+ 25. Find the failure that the verification of the rule catches. If that failure is trivial, artificial or only textual, do not propose that verification.
39
+ 26. Do not propose a test only because a rule has no test.
40
+ 27. An ordinary code defect is out of scope. The exception: it shows that the rule is not realistic, not clear, old, or contradicted.
41
+ 28. When the intent is not clear, say it, give options, and set `needsHumanIntent`.
42
+ 29. An internal contract that other code uses is specification. Do not move it out because it is not visible to users.
43
+ 30. A proposal that reverses an owner decision names that decision.
44
+
45
+ ## Severity: REQUIRED or ⚠️ Advisory
46
+
47
+ 31. REQUIRED: a rule whose violation has a production effect (common rules, Production effect). This includes a contract, an invariant, security, data and performance.
48
+ 32. ⚠️ Advisory: implementation detail that stays in the specification, development-only and local-only behaviour, demo agents, internal CI and developer tools, prompt wording, and design direction.
49
+ 33. Split a rule that mixes a contract and guidance. The contract stays REQUIRED.
50
+ 34. An existing requirement becomes Advisory only by the owner's decision. Propose it. Do not apply it.
51
+
52
+ ## Proposals
53
+
54
+ 35. Each proposal names its target:
55
+ - `Keep → OpenSpec REQUIRED invariant`, with a rewrite or a `**⚠️ Known gap (<tracker>):**` line when its behaviour or evidence is missing.
56
+ - `Reclassify → OpenSpec ⚠️ ADVISORY`.
57
+ - `Move → <target>`: the design narrative, `<dir>/AGENTS.md`, `scoped rule (<glob>)`, `skill <name>`, `command <name>`, `openspec/config.yaml#<key>`, `README.md`.
58
+ - `Replace → <tool>`: an ESLint rule, a dependency rule, a type assertion, a schema, a script.
59
+ - `Merge → <other rule>`, `Rewrite → "<new text>"`, `Delete → <reason>`.
60
+
61
+ ## The steward judge
62
+
63
+ 36. Keep a finding that is specific, has evidence, and has a correct proposal. Change or remove a finding when:
64
+ - it decides an unclear intent without the owner (change it: set `needsHumanIntent`);
65
+ - it is a style preference, an ordinary defect or a guess (remove it);
66
+ - its proposed verification catches no real failure (change or remove it);
67
+ - it moves an internal contract out only because users cannot see it (change it);
68
+ - it contradicts the placement guide (change it);
69
+ - it reverses an owner decision and does not say so (change it);
70
+ - it uses a false statement about code, tests, tools or CI (change or remove it).
71
+ 37. Give `judgeConfidence`, an integer from 0 to 100. The review file puts items at 70 or lower under "Low confidence".
@@ -0,0 +1,17 @@
1
+ # Judging rules: spec-verify judge
2
+
3
+ Use the common rules first: [common.md](common.md).
4
+
5
+ Be aware of the reader rules: [spec-verify.md](spec-verify.md), but focus on the rules below in THIS file, you MUST follow them strictly.
6
+
7
+ You judge one finding. The reader gave it the tier ERROR and a `readerConfidence`. You can confirm it or make it WARN. You CANNOT make a finding more severe (WARN/INFO -> ERROR or INFO->ERROR etc).
8
+
9
+ - Read the requirement, the two quoted sources, and the code near them.
10
+ - Answer `confirmed` only when all three statements are true:
11
+ - The quotes exist, and they say what the finding tells.
12
+ - The code does not satisfy the intent of the requirement. The difference is not in words only. Other code does not resolve it. It does not come from a dependency that the repository does not show.
13
+ - The defect has a production effect. Write the production scenario. If you cannot write it, answer `warn`.
14
+ - In all other conditions, answer `warn`. Name the statement that is not true.
15
+ - Give `judgeConfidence`, an integer from 0 to 100. It tells how sure you are that the finding is an ERROR.
16
+ - Think about `readerConfidence`. When it is low, you need a very strong evidence to confirm. Do not copy it, make your own judgement.
17
+ - The tool keeps the finding as an ERROR only when you answer `confirmed` and `judgeConfidence` is more than 70.
@@ -0,0 +1,50 @@
1
+ # Judging rules: spec-verify reader
2
+
3
+ Use the common rules first: [common.md](common.md).
4
+
5
+ This audit compares the specification with the code. It does not decide if a rule is a good rule. The spec-steward audit does that.
6
+
7
+ | Check | Compares | Kind |
8
+ | ----------- | --------------------------------------------------- | ---------------- |
9
+ | 1 Gap | a requirement with the code that must implement it | `undeclared-gap` |
10
+ | 2 Conflict | a requirement with a related requirement | `conflict` |
11
+ | 3 Duplicate | a requirement with a related requirement | `spec-dup` |
12
+ | 4 Mismatch | the intent of a requirement with what the code does | `code-mismatch` |
13
+ | 5 Test | a scenario with the test that cites it | `untested` |
14
+
15
+ Do not compare code with code.
16
+
17
+ ## Gap honesty
18
+
19
+ - For a requirement that no test binds, find if the behaviour exists. Start where its terms occur.
20
+ - When no code implements the behaviour and no `**⚠️ Known gap (<tracker>):**` line records it, report `undeclared-gap`.
21
+ - A process rule or a documentation rule has nothing to implement. It is not a gap.
22
+ - A missing citation is not a reason to suspect a gap.
23
+
24
+ ## Contradictory rules
25
+
26
+ - Two requirements conflict when one system cannot obey both.
27
+ - A conflict is a defect in the specification. It is WARN or lower. The owner decides.
28
+
29
+ ## A rule redundant with, or substantially overlapping, another
30
+
31
+ - Two requirements that state the same rule are `spec-dup`. This is always INFO.
32
+
33
+ ## A rule that no longer matches the implementation or architecture
34
+
35
+ - Find the intent of the requirement: the behaviour it requires and the failure it prevents.
36
+ - Report `code-mismatch` when the code does not satisfy that intent: it does what the requirement prevents, or it does not do what the requirement requires.
37
+ - Do not compare words. A different word, a different order of words, or a narrower or wider phrase is not a finding when the code satisfies the intent.
38
+ - When the requirement names a specific class, pattern, value or field as its contract, the code must use it.
39
+ - Code that satisfies the requirement is not a defect. A citation is not necessary.
40
+
41
+ ## A rule whose tests satisfy the wording while failing to protect the intent
42
+
43
+ - For each test that a scenario binds, make sure that the test asserts the outcome of the scenario. The name of the test is not evidence.
44
+ - A test that asserts less than its scenario is `untested`. This is always INFO.
45
+
46
+ ## Coverage
47
+
48
+ - A check is full when you judged each item that the part lists for it, and each requirement id is in your judged list.
49
+ - A check is sampled only when an assigned source was not available after you tried to read it. Name that source.
50
+ - An uncertain conclusion is a finding with a low `readerConfidence`. It is not sampled coverage.